Peter Selz was born in Munich of Jewish parents. In 1936, aged 17, he fled Nazi Germany because his parents wanted to send him to study in the United States. His family managed to escape Germany just before the Night of Broken Glass, with the help of some nuns, whom his optometrist father had treated for free. He spent one year at Columbia University and discovered that he was distantly related to Alfred Stieglitz, who became his mentor. After serving in World War II he received an A.M. from the University of Chicago on the GI Bill in 1949. He received several Fulbright scholarships in the following years to study at the University of Paris and École du Louvre as well as the Musées Royaux d'Art et d'Histoire; at the same time, Selz was teaching at the University of Chicago and also chaired the education department at the Institute of Design at the Illinois Institute of Technology in Chicago. In 1955 he moved to Pomona College to chair the art department and serve as director of the school's art gallery.