A Unionist in the Virginia Secession Convention of 1861, Carter voted against secession during both the votes on April 4 and April 17. Shortly after Virginia (and Loudoun County's voters) voted for secession, his Virginia Military Institute educated son, Richard Welby Carter, who had already organized a cavalry company to defend his northern Piedmont region, volunteered to join the Confederate States Army. The younger Carter would serve until nearly the war's end, despite two terms as prisoner of war, rising in rank from Captain of Company H of the 1st Virginia Cavalry to Major, Lt. Col. and finally Colonel, and although at war's end he was imprisoned at the Old Capital prison in Washington D.C. and at Camp Delaware. He was ultimately pardoned and married, but his father survived him. The family's farm, Crednal, was near the heart of the Battle of Unison in 1862, as well as during the Battle of Upperville, and was probably burned during the fall of 1864 by troops under Maj. Gen. Wesley Merritt because of Carter's hospitality toward J.E.B. Stuart in 1862 and John Singleton Mosby.