Moyer was born on March 2, 1919, in Portland, Oregon. He grew up in the Sellwood area of Portland and went to St. Agatha's Catholic School until eighth grade and then dropped out of school as a freshman to start an amateur boxing career. Known as "Tommy," he was a lightweight boxer with 145 victories of 156 amateur fights. Moyer is recognized as one of the greatest boxers ever produced by the state of Oregon, and was inducted into the Oregon Sports Hall of Fame in 1999. He was undefeated as a welterweight in 22 professional fights. He was Northwest Champion from 1935 to 1941, a five-time Pacific Coast Champion, a National AAU State Champion, a National Diamond Belt Champion, and an International Diamond Belt Champion. He fought Sugar Ray Robinson in a five-round amateur bout in Rochester, New York; the winner would go to the 1940 Olympics. Robinson was the winner in a close decision; Moyer was later named to the team after Robinson turned professional. This was Moyer's last defeat in his boxing career. Moyer trained as a member of the U.S. Olympic Boxing team for the Helsinki, Finland Olympics of 1940 but the games were cancelled because of the breakout of World War II.