The oldest portion of the house was built about 1670 by Joseph Clay. Clay's grandson inherited the property in the 18th century, but vacated the house for a more modern one in 1726. The house has its name because a family of Acadians from Grand Pré were settled in the house following their 1755 deportation from Nova Scotia.