You could get your phone number spoofed Because a scammer knows your phone number is legitimate after you've called a spam number back, they could spoof your phone number. Spoofing attacks over the phone trick people into thinking they are answering a familiar number.