The oldest portion of the house, its right three bays, was built in 1791. The left three bays were added sometime after 1800, and the front porch was also probably added at a later date. Although the property was never owned by him, it was from 1791 to his death in 1804 the home of George Walton, a signer of the United States Declaration of Independence. Trained as a lawyer, he served in the Continental Congress (1776–81) and in the Georgia militia, in whose service he was captured by the British during the 1778 Capture of Savannah. Exchanged and released, he was soon afterward elected Governor of Georgia, an office he held from November 1779 to January 1780. He also served as the state's Chief Justice, and in a second term as Governor 1789-80. He died at College Hill, his country house, in 1804. The house passed out of the Walton family in 1812, and is now owned and operated as a museum by the local chapter of the Daughters of the American Revolution.