Vermes's parents immigrated to the United States from Hungary as refugees during the 1956 revolution. His father, Michael, was a professional footballer for Budapest Honvéd FC and operated an indoor soccer training center in New Jersey after immigrating. Peter has two brothers and one sister.