Richardson was born in Bengal, India. His father was Robert Richardson, a civil servant of the East India Company who ran a silk factory. Richardson received his education at the Company's Military Seminary in Addiscombe, Surrey, England. Afterwards, he was in the Bengal Horse Artillery, and rose to the rank of Major. He took part in the Afghan Campaign, 1842 and was decorated for gallantry for his part in the attack on Istalif. In 1845–1846 Richardson also took part in the First Anglo-Sikh War.