Taussig was born in St. Louis, Missouri, the son of a wool broker, Charles and his wife, Anna (Abeles), who had emigrated from Austria in 1840. His family was Jewish, but he was brought up in the Unitarian Church. He was appointed to the U.S. Naval Academy during the Civil War and entered on July 23, 1863. His education over the next four years included service on the . Graduating in June 1867 he served on the steam frigate from July to December 1867 and thereafter variously on the , , and from January 1868 to April 1870. He was commissioned an ensign on 18 December 1868. His early sea service was perhaps most remarkable for his time as a passed midshipman on the gunboat when a tsunami washed her far inland at Arica (then part of Peru), on 13 August 1868. He was decorated for his actions during this event.