Because of his father's job at Mobil Oil, Rathbone has lived in places ranging from Indonesia to Midland, Texas.
Njegov pradedek, Monroe Jackson Rathbone II, je bil lastnik podjetja Standard Oil v New Jerseyju, ki so ga kasneje preimenovali v Exxon; Jackson Rathbone je tudi daljni sorodnik generala iz ameriške državne vojne, Stonewalla Jacksona.