Attending the Olympic trials at the 1964 New York World's Fair the following year, Carmody won a shock victory over the favored Melvin Miller to secure a place on the 1964 Olympic team. At the training camp, Carmody formed a close friendship with Joe Frazier, at one point reportedly talking Frazier out of quitting the sport following an unexpected defeat. At the Olympics Carmody suffered a bruised hand, but still beat the Nepalese Thapa Namsing and the German Otto Babiasch before losing in the semi-finals to the eventual winner Fernando Atzori, thus claiming a bronze medal.