The FitzWalter family was a wealthy and long-established family in the north-Essex area. Descended from the conquest-era Lords of Clare, the family held estates concentrated around the lordship of Dunmow. They also held estates as distant as Woodham to the south east of the county, Chigwell to the south west, Diss in Norfolk, and Castle Baynard in London. John FitzWalter was the son (probably the only son) of Sir Robert FitzWalter and Joan, daughter of Thomas, Lord Moulton. The family has been described as "warlike as well as rich" even before FitzWalter was born: his ancestor, also named Robert, had been a leading rebel against King John in the early 13th century.