Yes, you will absolutely get into trouble and be in contempt of court. The two are separate issues and you cannot (read absolutely cannot) use one as a bargaining chip for the other. Case law across the country has dictated this cannot happen. You must allow your children to visit with their father.  This is in the best interests of the children and the court does not take kindly to using the children like I said as bargaining chips. You can, however, file a contempt motion in court against him to force payment and perhaps meet with your county prosecutor to force a garnishment of wages.