In 1861 Smith entered the Confederate States Army on June 11 as a Private in the Pendleton Rifles, which became Company E of the 25th Virginia Infantry. Elected as Captain on July 1, he and his men were part of the surrender after the Battle of Rich Mountain. Being paroled 2 weeks later he was formally exchanged in March 1862, and was elected as Colonel of his regiment. He was wounded in the Battle of McDowell and received another wound at the Second Battle of Bull Run. After recovering in late 1862 Smith was ordered by the War Department to take several hundred newly exchanged soldiers to Colonel John D. Imboden for service in western Virginia, and was offered the command of them. In January 1863 he became Colonel of the new 1st Regiment Virginia Partisan Rangers, that became the 62nd Virginia Mounted Infantry, as part of General Imboden's Cavalry Brigade and took it into the Jones-Imboden Raid.