At the outbreak of the Civil War, Smith became a colonel of the 2nd California Volunteer Cavalry in the Union Army, rising early in 1862 to the rank of brigadier general in the United States Volunteers and to the chief command of the cavalry in the Department of the Missouri. From March through July, he served in the same capacity in the Department of the Mississippi. Assigned afterwards to the Army of the Tennessee, he took part in the Battle of Chickasaw Bayou and the capture of Arkansas Post. He commanded a division of the XIII Corps in the Vicksburg Campaign. Later, he led a division of the XVI Corps in the Red River Campaign of Maj. Gen. Nathaniel Banks. He received the brevet rank of colonel in the regular army for his services at the action of Pleasant Hill.