When the Civil War began and his native Kentucky struggled to maintain its neutrality, Johnson returned home and joined Nathan B. Forrest's cavalry battalion as a scout, fighting with him at his first engagement at the Battle of Sacramento. He escaped capture with Forrest after Fort Donelson, when the Confederate commanders decided to surrender their post to the Union besiegers. He later received a promotion to colonel in recognition of his exploits with his 10th Kentucky Partisan Rangers, a regiment he raised that often operated deep behind Federal lines in Kentucky. Johnson's men harassed Union supply lines and attacked isolated garrisons. In July 1862, in his Newburgh Raid, Johnson captured the town of Newburgh, Indiana, bluffing its sizable Union militia force into surrendering with only twelve of his men and a stovepipe mounted and a burnt black log on the running gears of an abandoned wagon to form a Quaker cannon. His capture of the first Northern city to fall to the Confederates made the news even in Europe, and Johnson's men thereafter nicknamed him "Stovepipe".