Potter was born on October 21, 1931, in Humansville, Missouri. He was raised in Buffalo, Missouri, where he was high school senior class president and also president of the student council. He attended Central Missouri State College on a basketball scholarship before enlisting in the United States Air Force. Potter served 13 months in Korea War between 1952 and 1956; he rose to the rank of a staff sergeant. He earned a bachelor of science degree in industrial relations and personnel management from the University of Missouri in 1958.