Smith married Shirley Lee of Hoquiam, Washington in 1952. They had nine children, six boys and three girls. Their oldest son, Marine Captain Vincent Lee Smith, was killed in the terrorist attack of Marine barracks in Beirut, Lebanon on October 23, 1983. General Smith died on Friday, September 7, 2012 in Arlington, Virginia. He is buried in Quantico National Cemetery, Quantico, Virginia.