La Farge had eight children: Christopher Grant, Emily Marie, John Louis Bancel, Margaret Angela, Oliver Hazard Perry, Joseph Raymond (who did not survive infancy), Frances, and John, Jr.Christopher Grant La Farge, his eldest son, was a partner in the New York-based architectural firm of Heins & LaFarge.
У них в семье было "Christopher Grant", "Emily Marie", "John Louis Bancel", "Margaret Angela", "Oliver Hazard Perry", "Joseph Raymond" (умер в младенчестве), "Frances" и "John, Jr".