The fate of the original fortifications is unclear. Henry II ordered that all royal castles be returned to the Crown. He had a policy of destroying adulterine castles, built without royal permission, during Stephen's chaotic reign. Initially, Aumale resisted the call to hand over Scarborough, which he had built on a royal manor, until Henry's forces arrived at York. The wooden castle vanished – William of Newburgh, writing near the time, claimed that the structure had decayed through age and the elements, battered beyond repair on the windswept headland. Later interpretations view this as implausible and argue that Henry wanted to stamp his mark on Scarborough, by demolishing William's fort and creating a much stronger stone complex.