There is the possibility that the original portion was built around 1810 by member of the Wells family and the Loyd family updated it around 1830. During the Civil War the house escaped damage although its owner, James D. Loyd was hanged by the federals in 1864 as a Confederate spy. There are many mysteries about Loyd Hall, including the two bullet holes in the wall on the front hall and arrowheads in the dining room door. A number of unmarked graves are on the grounds and there in one in the cellar which might be that of Inez Loyd, James' sister, who is said to have committed suicide when she was rejected by a lover. Other tales say that the grave is that of a Union soldier shot by Mrs. Loyd.
Legend has it that a ghost walks the house each night and plays a violin.
During the Civil War period a man named Joseph Sharrit, a former Englishman and Union Sympathizer is said to have lived at the plantation doing business for the syndicate. Through Sharrit's affiliation, Union troops stayed at Loyd's Hall. Diaries of the Loyd Family indicate that James Loyd posed as a Union friend, telling soldiers where to buy horses and supplies and then sending them into traps. The English syndicate acquired the house in 1870 in payment of debts, and at the death of Mrs. Loyd that year there were no Loyds left to re-buy the mansion. Sharrit went back to Ohio in 1876 and the house passed through various owners.