So long as there is an agreement as to visitation with respect to your grandchildren on the record a party who refuses to abide by it can be held in contempt of the agreement. You will need to memorialize the agreement into a written order. I suggest that you consult with a family law attorney for help.