the city started to grow up along with the castle