Phone numbers can be used to track your location and even some of your personal information like birthdate, past work experience, family members, and more. This information can be used to blackmail you and harm your family.