 Pollard was born in Glendale, California, on September 7, 1928. His mother was Phyllis Pollard. He starred as a halfback at Loyola High School, where he excelled at executing T formations. In 1946, his senior year, he scored 23 touchdowns and gained a total of 1,772 yards from scrimmage. These achievements earned him his second consecutive "Player of the Year" award for Southern California high school football players by the Helms Athletic Foundation.
