The house was built in 1760 on the site of a former Medieval castle. For centuries, the house and surrounding lands remained in the same noble family, the viscounts de Butler. In the late nineteenth century it came into the possession of Senator Jules Elby, connected by marriage to the Leroy and du Merle families. A member of the Elby family was involved in the design of the Paris Métro, and a white-tiled underground passageway still exists between the château and its former stables. During the First World War, the château was used by the British Army as an operations centre and Remaisnil lies close to the Somme battlefields. Following the German invasion of France in 1940 during the Second World War, the house was occupied by the Germans as officers' quarters, and a V-1 flying bomb launch site was constructed in the parkland, of which the foundation of part of the launch apparatus is still visible. Following the war the then-dilapidated house was retained by the Elby family until the late 1950s.