While it is rare that adultery would impact child custody, it is possible. For example, if a spouse’s adulterous actions showed neglect or harm for a child, such as leaving him or her unattended or inadequately cared for. In such a case, the court may see this as cause for concern over that partner’s ability to care for the child.