Some of the castle fell into the ocean while people were still living in it