When the Civil War began in 1861, Smith served as an aide de camp to Illinois Governor Richard Yates. On December 26, 1861 he was appointed colonel of the 45th Illinois Volunteer Infantry Regiment, with the rank backdated to July 23, 1861 to replace the previous Colonel who resigned his state Commission when the regiment entered Federal service. He led his regiment at the battles of Fort Henry, Fort Donelson and Shiloh. On November 29, 1862 he was promoted to brigadier general of U.S. Volunteers. He briefly commanded a brigade before taking command of the 8th Division, XVI Corps. When Ulysses S. Grant began his final campaign against Vicksburg, Smith was placed in command of the 1st Brigade, 3rd Division, XVII Corps. He fought at the battles of Port Gibson, Raymond, Champion Hill and in the assaults on Vicksburg. In June, 1863 in the midst of the siege of Vicksburg, Smith was chosen to replace Gen. Isaac F. Quinby in command of the 7th Division, XVII Corps after Quinby became ill and took a leave of absence. In September, 1863 Smith was transferred to command the 2nd Division, XVII Corps and his division was sent with William T. Sherman to aid in the relief of Chattanooga. During the battle of Missionary Ridge, Smith took part in the attacks against the Confederate right flank at Tunnel Hill.