Allen was raised by his birth parents in Norfolk, where he attended a school for African-American children for two years. The school was closed down due as part of the reaction to Nat Turner's slave rebellion of 1831. There was no school for colored children in Fort Monroe, but he received some informal education from Federal soldiers, including some French and German. He had access to libraries and was to some degree self-educated, but took advantage of the few educational opportunities available to a black boy. A teacher recommended him to Gerrit Smith, a wealthy abolitionist and philanthropist, whose help, along with that of Lewis Tappan, made it possible for Allen to attend the Oneida Institute. Oneida was the first college in the country that accepted African Americans as a matter of policy; it was a hotbed of abolitionism.