While Cole appeared to be Motley's chosen successor at fullback in an offense that featured quarterback Otto Graham and ends Mac Speedie and Dante Lavelli, his relationship with Brown was strained. Cole felt that Brown did not like that he challenged his authority and refused to submit to his authoritarian coaching style. Cole saw his carries increase in 1951, when Motley was injured. The Browns again made the championship game that year but lost to the Rams. Brown cut Cole during the following season, and he spent the remainder of the year with the Chicago Bears before leaving football. Cleveland halfback Dub Jones later criticized Brown's handling of Cole, calling it the biggest waste of talent he had ever seen.