He was married twice. He and his second wife, who was White, lived in Birmingham, Alabama, and attended the African-American 16th Street Baptist Church. After the infamous fire bombing of the church in September 1963, he and his wife decided to leave the South and settled in San Francisco, where there was less racial tension and interracial marriage was more accepted. He had an office on Clement Street in the Avenues, and his wife preceded him in death. He was a prolific author with a larger-than-life personality, and continued to study and write until his death in 1992.