The youngest of five children, Robinson was raised in relative poverty by a single mother. He attended John Muir High School in Pasadena, California, and Pasadena Junior College, where he was an excellent athlete and played four sports: football, basketball, track and baseball.