He married on 14 September 1803 Elizabeth, daughter of Lieutenant-General Francis Lascelles. She was born on 30 March 1784, and died at Chislehurst on 29 July 1828. Their children included Anne and Charlotte. Anne's son, Charles Nepean was a Middlesex county cricketer who also played football, and was on the winning side in the 1874 FA Cup Final.