The first proprietor of Eynsford Castle was William de Eynsford I, a knight and sheriff of Kent. The palace was later bought by the Hart Dyke family, who were already the owners of Lullingstone Castle.