If your ex-spouse does not follow the visitation order, it is up to the court to determine what happens. In some cases, the Judge may take away custody of the children from the non-custodial parent. This can only happen if there is a valid reason why the non-custodial parent failed to obey the order.