Baldwin was born in Dutchess County, New York, and raised on the family farm there. He attended high school in Red Creek, New York, and later taught school in both Rhode Island and Connecticut. On April 13, 1861, he joined the Union Army along with his four brothers, one of whom would be killed in action at the Battle of Resaca. He was initially elected as captain for Company C, 36th Regiment, Illinois Volunteer Infantry. The regiment was dispatched to Missouri, and in February 1862 he was made quartermaster. In August of that same year he was promoted to lieutenant colonel, over the 8th Regiment Missouri Volunteer Cavalry.