On June 15, 1850, he married Caroline Catherine Butcher, the daughter of a black mother and half-Native American and half-white father. From 1850 to 1855, he prospected for gold in California while his wife moved to Ontario, Canada, due to fears of being sold into slavery.