 Brown married Lucie Barbee in 1857, but she died the following year. In September 1860, he married Rebecca Hart Dixon, the daughter of former U.S. Senator Archibald Dixon. The couple had eight children.
