Harmon was the brother-in-law of Ricky Nelson and John DeLorean and is the uncle of actress Tracy Nelson and singers Matthew and Gunnar Nelson of the rock duo Nelson. In 1987, Harmon filed for custody of his nephew Sam, Kristin's son, on the grounds that she was incapable of good parenting.