Eveleth was born in New Gloucester, Maine, on May 8, 1821, to James and Hannah Austin Eveleth. His father was a leather tanner and boot and shoe maker and a descendant of early colonial families of Massachusetts. He was a direct descendant of Joseph Eveleth, juror in the Salem Witch trial of John Proctor, and a second cousin of Theophilus Parsons, Chief Justice of the Supreme Judicial Court of Massachusetts. Eveleth married Mary Weeks of Bath, Maine, in 1852.