It is unknown how old the castle is and what it originally looked like. From 1494 there was a walled stone house on the De Boeverie estate near the two farms, one small and one large, already located there. The small farm stood at the intersection of Duivelsbruglaan and Bouvignelaan but went up in flames during the siege of Breda by Spinola in 1624. The large farm had been owned by the Princes of Orange since 1614 and by the state since 1881. In 1941 the farm was destroyed by a V1 rocket but later rebuilt.