Rathbone studied law at Union College where he was known to miss many classes and briefly worked in a law partnership in Albany. In 1858 Rathbone entered the New York National Guard where he briefly worked as a judge advocate. Shortly after this he was selected to be sent to Europe as an observer during the Second Italian War for Independence before entering the Union Army at the start of Civil War. During the war, Rathbone served as Captain in the 12th Infantry Regiment and was at the Battle of Antietam and the Battle of Fredericksburg. In 1863 Rathbone was pulled from frontline duty and given a desk job. By the war's end, he had attained the rank of major.