The library was built as a private residence in 1835 for Daniel Gay, who owned a textile mill on the White River, which Route 107 parallels. Daniel and his brother Jeremiah moved to Stockbridge in the 1820s, and built the mill in what became known as Gaysville in 1832. The mill continued to be operated by Gay family members until its destruction (along with most of the village) in Vermont's devastating 1927 floods. In 1937 the house was given by Lucia Gay for use by the Belcher Library Association.