Richards was born in Ireland, and partly grew up on the family's estates in County Wexford which had been granted to his grandfather Solomon Richards during the Cromwellian era. He had two brothers Michael Richards and John Richards who also pursued army careers. A sister, Elizabeth, married the politician James Craggs. The family were Protestants, although John converted to Catholicism.