Cole was born in Illinois and grew up in Swanton, Ohio, where he was a standout athlete in high school. He attended Toledo on an athletic scholarship and was a mainstay on the football team between 1947 and 1949, setting several school records. The Browns selected him late in the 1950 NFL draft, and groomed him as the replacement for Marion Motley. Cole ran for more than 100 yards in 1950, when the Browns won the NFL championship, and saw his playing time increase the following year after Motely was injured. He was cut, however, in 1952 after coming into conflict with head coach Paul Brown.