Broughton Castle Broughton Castle is a medieval manor house located in the village of Broughton which is about two miles south - west of Banbury , Oxfordshire , England on the B4035 road ( ) . It is the home of the Fiennes family , Barons Saye and Sele . The castle sits in parkland and is surrounded by a wide moat . History . The castle was built as a manor house by Sir John de Broughton in 1300 at a location where the confluence of three streams created a natural site for a moat ed manor . The house was sold in 1377 to William of Wykeham , Bishop of Winchester , and has remained in the same family since that time . The original house was crenellated by Sir Thomas Wykeham in 1406 . In 1451 it passed by inheritance to the Fiennes family , Barons Saye and Sele . Beginning in 1550 Richard Fiennes transformed the medieval manor into a house in the Tudor style , significantly expanding the building . In the 17th century William Fiennes , 1st Viscount Saye and Sele was one of the leading activists against Charles I . As a result the castle was used by Parliamentarian sympathisers , such as John Pym and John Hampden , as a meeting place in the decade leading up to the Civil War . The 1st Viscount raised troops to fight against the king at the inconclusive Battle of Edgehill in 1642 . In the following days , Royalist troops besieged the castle , quickly overcoming the defenders and occupying the castle for a time . Following the end of hostilities , reconstruction work needed to be undertaken to repair damage inflicted by Royalist canon . James I stayed at Broughton Castle on several occasions . Broughton fell into decay in the 19th century , but was eventually rescued by Frederick Fiennes , 16th Lord Saye and Sele , who brought in the prominent Victorian architect Sir George Gilbert Scott . At the turn of the century the house was rented by the Edwardian socialite Lady Algernon Gordon - Lennox , who transformed the gardens and hosted Edward VII at the castle . It is still the home of the Fiennes family and currently lived in by the 21st Baron and his wife . It opens to the public over the summer . Filming . Parts of the films " Oxford Blues " ( 1984 ) , " The Scarlet Pimpernel " ( 1982 ) , " Three Men and a Little Lady " ( 1990 ) , " The Madness of King George " ( 1994 ) and " Shakespeare in Love " ( 1998 ) were shot in the castle . TV filming for parts of " Elizabeth The Virgin Queen " , " Friends and Crocodiles " , " 1975 Morecambe and Wise Christmas Show " and the titles of " Noel ' s House Party " also took place there . The castle has been used as the location for several other films and TV programmes including an adaptation of Jane Eyre . Concerts . In August 1981 , the electric folk band Fairport Convention held their annual reunion concert at Broughton Castle , rather than the usual Cropredy location . The concert was recorded , and released on the album " Moat on the Ledge " ( 1982 ) . Fiction . Published in April 2009 , " The Music Room " is a novel by William Fiennes . This fictionalized memoir of his childhood and his epileptic brother is set in ( the never identified ) Broughton Castle . It has been described as " a beautiful poem of a tribute to his family , his parents , the magical , moated castle that was his home " .