Gifford was unable to gain an athletic scholarship to the University of Southern California (USC) because of his low grade point average in high school, so he played a season of football for Bakersfield Junior College. While at Bakersfield, he made the Junior College All-America team and earned the grades needed to enroll at USC. At USC, Gifford was named an All-American after rushing for 841 yards on 195 carries during his final season. He graduated from USC in 1952.