Collier was fired after the 1961 season and Brown rehired him as an assistant. Art Modell, the owner of the Browns, then fired Brown in 1963 and promoted Collier to head coach. Under Collier, the Browns reached the NFL championship game four times and won once, in 1964. Struggling with hearing loss, Collier retired after the 1970 season, although he remained a scout and quarterbacks coach for several more years. He was diagnosed with prostate cancer in 1976, and retired to Texas, where he died in 1983.