Hubert began to upgrade his new castles, starting with Grosmont, but was captured while fighting in France. During Hubert's captivity, King John took back the Three Castles and gave them to William de Braose, a rival of Hubert's. King John subsequently fell out with William and dispossessed him of his lands in 1207, but de Braose's son, also called William, took the opportunity presented by the First Barons' War to retake the castles.