Charles Alfred Lee was born in Parramatta, New South Wales in 1842, the son of Benjamin Lee, a Waterloo veteran, publican and landowner, and Lucy Ann Poulton, who had emigrated to New South Wales in 1829. His elder brother, Benjamin Lee, was also to become a NSW politician and civil servant. Educated at West Maitland Grammar School, Charles entered a Maitland store, rising to become partner. On 18 July 1865 in Penrith he married Clara Jane Tindale. They were to have six sons, Walter Lee (1868–1925), Lieutenant Colonel Charles Arthur Lee (1874–1918), Captain Frederick Edward Lee (1875–?), Reginald Lee (1878–?), Sergeant Lionel Kenneth Lee (1882–1919) and Cecil Lee (1885–1895); and four daughters, Clara Isabel Stuart (1870–1953), Constance Maude Addison (1872–1949), Emilie Mabel Lee (1877–?) and Ruby Violet Lee (1880–1881).