In 1799, the two story structure with four rooms was built by Judge John Overton (1766–1833). Overton originally named the property Golgotha after the large number of prehistoric skulls that were unearthed while digging the cellar of the house. Archaeologists now know that these remains were part of a large Mississippian village site. In 1996 a modern barn structure was built and 13 stone coffins were discovered, also dating back to the Mississippians. Overton changed the name of the plantation to Travellers Rest in the early 19th century to reflect the recreational effect his home had on him after the long rides on horseback that he had to undertake as a circuit judge.