Richardson was born in Great Bridge, Tipton and as a youngster played football for Greets Green Boys, Greets Green Prims and Great Bridge Celtic. He turned professional in August 1926, when he joined West Bromwich Albion, and made his league debut in December 1928 in a Division Two match away at Middlesbrough. In 1930–31 he helped the club to achieve promotion to the First Division and played in the 1931 FA Cup Final, in which Albion beat Birmingham 2–1. Richardson also appeared in the 1935 FA Cup Final, but this time earned only a runners-up medal as his team lost 4–2 to Sheffield Wednesday. After making 352 appearances for West Bromwich Albion, he joined Swindon Town for a £200 transfer fee in May 1937, before moving on a free transfer to Dudley Town in September 1939. He joined Vono Sports F.C. in August 1940 but spent just one season with the club prior to his retirement in June 1941. He died in August 1985, aged 77.