At the start of the Civil War, Sharp enlisted in the 1st Mississippi Battalion, which was later consolidated into the 44th Mississippi Infantry (also known as Blythe's Regiment). He rose through the ranks from private to captain and participated in the Battle of Shiloh, Bragg's Kentucky Campaign, and the Battle of Murfreesboro. Promoted to colonel in August 1863, he led a brigade following the promotion of Brig. Gen. Patton Anderson and was in command of it at Chickamauga and the Battle of Missionary Ridge.