Mayer was born into a middle-class family in Luxembourg on 19 June 1926. His father was a wholesaler who had studied at the University of Heidelberg and had strong social democratic and Zionist beliefs. Mayer described his family as "fully emancipated and largely acculturated Luxembourgian Jews". The family fled into France amid the German invasion on 10 May 1940 and reached the France–Spain border by Autumn 1940 but were turned back by Spanish border guards and were in the Vichy-controlled "Free Zone" after the Fall of France. The family succeeded in boarding a ship to Oran in French Algeria on 18 October 1940 but were prevented from entering Morocco because they lacked a visa and were house arrested for several weeks in Oudja. They secured visas for the United States on 22 November 1940 and arrived in New York during January 1941. His maternal grandparents who had refused to leave Luxembourg were deported to the Theresienstadt Ghetto where his grandfather died in December 1943.