Patterson graduated from the Mississippi State College School of Agriculture in 1943. At 17 he hitchhiked from Clarksdale to Starkville to try out with the Bulldogs, "hoping to earn a scholarship to play football and study farming." He failed in the tryouts as a center, but when placed as an end he succeeded so emphatically that he was awarded a four-year scholarship. He was on the 1940 Orange Bowl championship team, the only undefeated team in the school's history. In 1942 he was made captain. That year he played in the Blue-Gray College All Star game and was selected as an All Southeastern Conference end. He was named to the MSU Sports Hall of Fame in 1995.