Yes, hackers can use your phone number to gain access to accounts that use your number for verification or password reset.