During the 1798 insurrection in Ireland his palace in Ferns was ransacked and Cleaver was obliged to take refuge in Beaumaris, Anglesey which was in his brother, William Cleaver's diocese of Bangor, Gwynedd, where he lived at what is now the Bishopsgate Hotel.