Williams was born in 1837 in Gibraltar to a military family. After living in the East Indies, West Indies and on the African Gold Coast, he arrived in New York City at age 13 following the death of his parents. He found work at the "New York Times" and stayed there until the start of the American Civil War. He joined the Union Army, rising from private to brevet major and was wounded at the battles of Malvern Hill and the Wilderness.