Courts are always concerned about child abduction between ex spouses.  But just because you hold passports does not mean that you will be asked to surrender them.  I assume your potential ex also holds a passport. If there is a realistic fear of impending abduction a party may petition the court to surrender the child's passport.  If the party making the petition has proved to the court that their fear is realistic then the court will seal the passport in the court file.  Any future requests to travel abroad with the child must then be made in a court hearing.