Between the siege of Charleston in May 1780 and the Battle of Camden in August that year, the North Carolina Line (a loose organizational structure that encompassed all of North Carolina's Continental Army units) was virtually annihilated, suffering substantial casualties and the loss of many men as prisoners of war. During at least part of the intervening time, Sumner was in North Carolina on a recruiting mission. Rather than rebuild the Line, the North Carolina General Assembly determined to rely on militia for the defense of the state. In September 1780, Sumner temporarily transferred to command of the Hillsborough District Brigade, under the statewide leadership of Richard Caswell. As commander of a brigade of North Carolina militia, Sumner was tasked with defending the state from the advances of British General Charles Cornwallis, but the militiamen were poorly equipped and ill-trained.