It depends upon whether or not there's a Court Order that relates to this issue. Each parent has a right of association with the child, and that includes the right to travel. So, in the absence of any Custody Agreement or Marital Aettlement Agreement stating otherwise, neither parent has the right to stop the other parent from exercising that right.
Unfortunately, it is not uncommon for one parent to threaten to take the child and run. The truth is, it is legally difficult to kidnap your own child. However, if there is a repeat pattern of this behavior, especially if it deprives the other parent of access or in some way traumatizes the child, the court may intervene.