The year he graduated from law school, on August 28, 1917, Rutledge married Annabel Person. The couple had three children: Mary Lou (1922), Jean Ann (1925), and Neal (1927). Rutledge was the great-great-grandson of Edward Rutledge, South Carolina politician and the youngest signer of the Declaration of Independence. Edward Rutledge's brother, John Rutledge, was a South Carolina delegate to the 1787 Constitutional Convention and twice nominated by President George Washington to the United States Supreme Court.