Refusing a child's visitation rights can negatively impact custody arrangements, as it may be seen as not acting in the best interests of the child.