Battle of Kelly ' s Ford The Battle of Kelly ' s Ford , also known as the Battle of Kellysville , took place on March 17 , 1863 , in Culpeper County , Virginia , as part of the cavalry operations along the Rappahannock River during the American Civil War . It set the stage for Brandy Station and other cavalry actions of the Gettysburg Campaign that summer . Twenty - one hundred troopers of Brig . Gen . William W . Averell ' s Union cavalry division crossed the Rappahannock to attack the Confederate cavalry that had been harassing them that winter . Brig . Gen . Fitzhugh Lee counterattacked with a brigade of about 800 men . After achieving a localized success , Union forces withdrew under pressure in late afternoon , without destroying Lee ' s cavalry . Background . When Maj . Gen . Ambrose Burnside was relieved of command of the Union ' s Army of the Potomac ( following the disastrous Battle of Fredericksburg in December 1862 and the fiasco of his Mud March in January 1863 ) , his replacement , Maj . Gen . Joseph Hooker , immediately began reorganizing and training his army , in winter quarters outside of Fredericksburg . One of his most significant actions was to combine smaller cavalry units , spread out across the army , into a single Cavalry Corps , led by Maj . Gen . George Stoneman . Up until this time , the Union cavalry had been consistently outperformed by their Confederate counterparts , commanded by Maj . Gen . J . E . B . Stuart . Although they possessed superior equipment and had the advantage of a plentiful supply of men and federal horses , the Union cavalrymen had lacked the confidence , experience , and leadership to challenge Stuart . On February 25 , 1863 , Confederate cavalry under Brig . Gen . Fitzhugh Lee , one of Stuart ' s key subordinates and a nephew of Gen . Robert E . Lee , led a force of 400 troopers in a raid near Hartwood Church in Stafford County , 9 miles northwest of Fredericksburg . The Federal cavalry was ineffective in pursuing Lee and managed to lose 150 prisoners from the division of Brig . Gen . William W . Averell , ironically one of Fitz Lee ' s closest friends at West Point . Hooker was furious and threatened to relieve Stoneman of his command if he did not stop Confederate raids of this type . At the same time , Fitzhugh Lee was sending his old friend and classmate taunting messages across the river . One of the more challenging messages said " I wish you would put up your sword , leave my state , and go home . You ride a good horse , I ride a better . If you won ' t go home , return my visit , and bring me a sack of coffee . " . Scouts from Averell ' s 2nd Division , Cavalry Corps , detected Confederate cavalry near Culpeper Court House about three weeks later . Averell assembled a force of 3 , 000 cavalrymen and six artillery pieces ( the 6th Battery , New York Light Artillery , under Captain Joseph W . Martin ) and set off for Kelly ' s Ford on the Rappahannock River between Fauquier and Culpeper Counties . After various troops were detached to cover his movements and to engage the enemy ' s pickets at Rappahannock Station , he had 2 , 100 men ready for battle in three brigades , commanded by Col . Alfred N . Duffié , Col . John B . McIntosh , and Capt . Marcus Reno . Facing him was a detached Confederate brigade commanded by Fitzhugh Lee , 800 men in five regiments , with a two - gun artillery section . The " Maryland Scroll , " a graffiti on the wall of the " Graffiti House , " in Brandy Station , Virginia , contains the names of 16 Maryland Confederates who served rifled gun # 1 of James Breathed ' s Battery and were on picket duty in Brandy Station on March 16 , 1863 . The unfurling banner ( also known as the horizontal scroll ) reads : " Rifle Gun " and " No . 1 , Stuart Horse Artillery / Breathed ' s Battery / On Picket - March 16 , 1863 . " Breathed ’ s Battery was heavily engaged at the battle on the next day . Battle . Early on the morning of March 17 , 1863 , Averell ' s advance guard reached Kelly ' s Ford on the Rappahannock and found that felled trees and 60 Confederate sharpshooters opposed their crossing . Three attempts to cross were repulsed under heavy fire , delaying the Union advance by over 90 minutes . Averell ' s chief of staff , Major Samuel E . Chamberlain , eventually forced a crossing led by 20 men of the 1st Rhode Island Cavalry . Chamberlain was wounded in the head . Despite the minor casualties in this action , Averell proceeded cautiously , taking over two hours to cross his men over the swiftly running river . Lee , 10 miles west at Culpeper Court House , was notified of the crossing attempts by 7 : 30 a . m . Assuming that Averell ' s target was Brandy Station on the Orange and Alexandria Railroad , Lee sent his 800 men forward to block the Union advance . They encountered the Union cavalrymen deployed near the C . T . Wheatley house , about 2 miles northwest of Kelly ' s Ford . Duffié ' s brigade was positioned on the left in a woodlot , McIntosh ' s in the center , and Reno ' s two regiments of regulars on the right , behind a stone fence . Jeb Stuart also happened to be at Culpeper Court House that day , attending a court - martial . He decided to ride out to witness the battle , taking with him his artillery chief , Maj . John Pelham . They arrived to find that Lee ' s men were not doing well , outnumbered two to one and facing a well - positioned artillery battery . It was the first time in the war that this Confederate cavalry regiment ( the 2nd Virginia ) had fled in the face of a Union charge . Lee ' s men advanced with the five regiments in line abreast . The 3rd and 5th Virginia Cavalry regiments , led by sharpshooters , ran along the stone fence with the expectation they would find a gap in it somewhere . Pelham moved forward with Lee ' s men , and as he waved them through a gate in the fence , a shell exploded over his head , sending a tiny fragment into his brain , mortally wounding him . He died a few hours later . The Confederate advance was repulsed by carbine fire from the 16th Pennsylvania Cavalry and shelling from Martin ' s battery . On the Union left , Duffié disobeyed Averell ' s orders to hold his position and ordered a charge . The surprise attack forced Lee to withdraw his men back through the woods to a clearing just behind . Lee counterattacked the advancing Union troopers , but once again had to fall back in the face of superior numbers and artillery . A rout of the Confederate position might have been possible , but Reno did not advance in support of Duffié , maintaining his position as ordered earlier by Averell . Aftermath . By 5 : 30 p . m . , Averell , citing his exhausted men and horses , " deemed it proper to withdraw . " He left two Confederate officers who had been wounded and captured by Averell ' s troops , a sack of coffee , and the following message : " Dear Fitz , Here ' s your coffee . Here ' s your visit . How do you like it ? " Some of his fellow officers believed that he lost his nerve , concerned about the presence of Jeb Stuart on the battlefield and , hearing the sound of railroad cars approaching , imagining the possibility of a Confederate infantry force pinning him against the river . The Union advance had covered 2 miles over more than 12 hours and resulted in 78 casualties ( 6 killed , 50 wounded , 22 missing ) . The Confederates lost 133 ( 11 dead , 88 wounded , 34 captured ) ; 71 Confederate horses were killed and 12 were captured . The loss of the youthful Pelham , age 24 , well respected by Robert E . Lee , Stuart , and many veterans of the Battle of Fredericksburg , was a shock . Stuart wrote after the battle , " The gallant Pelham — so noble , so true — will be mourned by the nation . " The battle was technically a Confederate victory because Averell ' s men failed to achieve their objective of destroying Lee ' s smaller force and they withdrew under pressure across the ford at dusk . However the Federal cavalrymen believed they had won a moral victory because , for the first time , they had held their own against Stuart ' s legendary horsemen . They would proceed into the 1863 summer campaigns with increased confidence . One of the participants , Lt . Joseph A . Chedell of the 1st Rhode Island , wrote that Kelly ' s Ford was the " first real , and perhaps the most brilliant , cavalry fight of the whole war . " Both Union and Confederate armies used Kelly ' s Ford extensively during the Civil War . In addition to the role it played in this battle , it was also host to two notable engagements that occurred later that same year : the Battle of Brandy Station on June 9 , 1863 , and one during the Bristoe Campaign ' s Second Battle of Rappahannock Station on November 7 , 1863 .