Until the 17th century, the castle was the residence of the Avaugour family, descendants of François Ier d'Avaugour, illegitimate son of François II. He modified and transformed the castle to suit the tastes and fashions of the day. During the War in the Vendée, the town and its castle were burned by the Infernal columns of Jean-Baptiste Kléber. In 1807, the estate was bought by the sculptor François-Frédéric Lemot with the goal of conservation. During the 19th century, the ruined castle attracted Romantic painters and sculptors.