Ashby is married to actress Susan Walters. The couple met on the set of Loving in 1983, where she was a regular and he was filming a guest appearance (he would later become a regular cast member in 1985). They have two daughters, born in 1991 and 1992.