Bliss was born in Tiverton, Rhode Island on 22 July 1837 to James Leonard and Sarah A. Bliss née Stafford. Before the war Bliss completed his law degree at Albany Law School in May 1861 and was called to the bar in New York. That same year he enlisted into the Company B of the 1st Rhode Island Cavalry. The following year he was promoted to captain of Company C. He participated in several Civil War battles including the Battle of Second Bull Run, the Battle of Middleburg within the Gettysburg campaign and the First Battle of Deep Bottom as part of the Siege of Petersburg. In September 1864, while in command of a provost guard in the Shenandoah Valley he performed an act of gallantry that resulted in his capture by Confederate soldiers and earned him his Medal of Honor award after his release from Libby Prison on 5 February 1865.