In 1704, following the Capture of Gibraltar by an Anglo-Dutch fleet on behalf of the Archduke Charles, claimant to the Spanish throne, almost all the population left the town. Some of the refugees built their houses by the chapel; in theory they were provisional buildings, since they were entrusting that Gibraltar would be recovered soon. However, with the passage of the time, the provisional settlement became the seed of a new town, beginning in this way the modern history of Algeciras after his destruction in 1379. The hermitage gave shelter to the statue of Our Lady of Europe. The statue of the Virgin and the Child was venerated in Gibraltar, in the Shrine of Our Lady of Europe. At the time of the capture, the shrine was looted and the statue of the Virgin and Child was broken and its remains thrown out to sea. As the statue was carved from wood, the pieces were recovered by a fisherman who found them floating in the Bay of Gibraltar and later delivered them to Juan Romero de Figueroa, the priest in charge at the Church of St. Mary the Crowned and St. Bernard in Gibraltar. Romero de Figueroa remained in the town after the capture and eventually took the pieces of the statue to Algeciras for safekeeping in the former Chapel of St. Bernard, which was dedicated then to Our Lady of Europe and named Chapel of Our Lady of Europe (). The original statue was returned to Gibraltar in 1864.