Although the house was built in 1759 by Nathaniel Hempstead, the English grandson of Joshua Hempstead (whose 1678 home stands adjacent), its form and building material are unusual for southern New England in that period, leading to local lore attributing its construction to French Huguenot immigrants. It is possible that laborers who built the house were Acadians resettled to New London after the Expulsion of the Acadians from Nova Scotia in the 1750s. The same workers may have worked on another major stone building in the city, the Shaw Mansion.