Gabay was born in Jerusalem, Israel to a family of Sephardi descent. Following the Spanish expulsion the family was expelled from Kingdom of Castile and resettled in Gaza. After the 1799 retreat of the Napoleonic troupes, Gaza Jews were condemned for breach of the limitations of their tolerated inferior status as dhimmi by associating with the French. According to Islamic jurisprudence, jihad led to a massacre of the Gaza Jews. The surviving members of the Castel family fled and resettled in Hebron. In the 1929 Hebron massacre the family was saved by a Muslim friend.