The house which forms the architectural core of the palace was built for the first Duke of Buckingham and Normanby in 1703 to the design of William Winde.