Named Bonnie Castle by original owner Samuel Gallatin Smith, the Greek Revival cottage was built in 1859 on a parcel of in Tyler. Smith joined the Confederate States Army in 1861 and sold Bonnie Castle to local Tyler teacher F.N. Gary, who never resided in the house. After the 1862 Capture of New Orleans, Gary made the house available to numerous families of refugees fleeing the Union Army.