Born in Leeds on 15 September 1847, Ernest was son of John Crofts, Esq. of Adal, near Leeds, a Justice of the Peace, and grandson of the Rev. W. Crofts, B.D., Vicar of North Grimston, near Malton, Yorkshire. One of his maternal uncles was the Rev. William Carr, B.D. of Bolton Abbey, Yorkshire. His mother was Ellen Wordsworth, the daughter of a Leeds industrialist. Ernest studied at Rugby School, for several years, and then headed to Berlin where he developed his interest in art and decided upon a career as a painter. His first acquaintance with war was made in 1864 when he accompanied a Prussian doctor in the Schleswig-Holstein War, and the operations around Düppel. His sister Ellen Wordsworth Crofts married Sir Francis Darwin and was the mother of poet Frances Cornford.