The house is assumed to have been built about 1718, when Benjamin Flagg II purchased a large tract of land in the area; however, it is not known how much of the building is his original construction, and how much may have been later alteration. According to Caleb Wall's "Reminiscences," Flagg's son, also named Benjamin, "commanded the company of militia, who marched with Col. Timothy Bigelow's company of minute men for Lexington, on the alarm, April 18, 1775, afterwards Colonel in the revolutionary service, filled important town offices, and died in Worcester, October 8, 1818, aged 95... Col. Flagg's location was on Plantation Street, on a farm of 150 acres..." The house was sold out of the Flagg family in 1850.