When the American Civil War broke out, Duffié enlisted in the Union Army. He first joined the 2nd New York Cavalry Regiment (also known as the Harris Light Cavalry), on August 9, 1861, and was soon promoted to the rank of captain. The somewhat quarrelsome Duffié was placed under arrest several times for confrontations with other officers; in one incident, he challenged General Fitz John Porter to a duel. In July 1862 Duffié was appointed to command the 1st Rhode Island Cavalry, with the rank of colonel, by that state's governor, William Sprague IV. Though the 1st Rhode Island's officers initially refused to serve under a foreign-born leader, Duffié soon won them over and reorganized the 1st into a fine fighting unit. Assigned to the command of General William Averell, they saw action against Confederate troops under Thomas J. "Stonewall" Jackson in August 1862, in fighting near Cedar Mountain, Virginia.