McGee was born in Shelby County, Kentucky. His family moved to Kansas City to Clay County, Missouri and then to Jackson County, Missouri where they bought a half section of land of what is now Downtown Kansas City. When McGee was 12 he ran away from home to Texas. He returned to Kansas City in 1841. He made a fortune in the California Gold Rush and used the money to build the Southern House hotel at 16th and Grand. The hotel would become a hotbed of southern sympathizers in the Bleeding Kansas war.