When the Civil War broke out Rucker enlisted in the Confederate States Army as a private in Pickett's Tennessee Company of Sappers and Miners. Sent to Kentucky, he was promoted to lieutenant. On May 10, 1862 he was transferred and promoted to captain of Company C, 1st Tennessee Heavy Artillery; his unit composed primarily of men from Illinois. His company worked three 8-inch Columbiads and three 32-pounders as part of the garrison during the Battle of Island Number Ten. When the island fell he escaped and was commended for his valor. Rucker was transferred to the cavalry with the rank of major and assigned to enforce conscription laws in East Tennessee. His unit became the 16th Tennessee Cavalry Battalion. In early 1863 Rucker was promoted to colonel and given command of the newly created 1st East Tennessee Legion, also known as Rucker's Legion. Its components were his battalion as well as 12th Tennessee Cavalry Battalion and a battery of artillery. With his legion he participated in Pegram's Kentucky Raid, the Battle of Chickamauga and the Chattanooga Campaign.