Anything within your discretion (e.g. beyond the two days a week he's guaranteed) you can control, so you don't need to  let the children visit with him more than is required in the divorce decree or agreement. To alter the visitation in the decree or settlement, however, you'd have to go to court and get a court order; this would require showing that he is actively harming the children's interests (not just not taking as good a care of them as he should).