That invasion's failure forced the members of those regiments into exile, and they settled mostly in upstate New York