Rutherford's brothers Sep and Bob were also professional footballers. He was married twice: first to Edith Olive McQueen in May 1908with whom he had a son, John, who was on Arsenal's books at the same time as his father, but only ever played one League match for the cluband then to Blodwen Jones in 1944. His great-grandson Greg Rutherford is the 2012 Olympic long jump champion.