For the 1964 Olympics in Austria, the "dashing" and "handsome" Zimmermann was heavily favored to win. However, the course at Patscherkofel was quite difficult (nicknamed the "Course of Fear"), but he still managed to win by 0.74 seconds. (Franz Klammer famously won on the same course a dozen years later in 1976.) He did not enter the slalom and did not finish the giant slalom. Despite not sweeping the alpine events as did his compatriot Toni Sailer in 1956, Zimmermann appeared on the February 10, 1964 cover of "Sports Illustrated" magazine in the United States.