Unless it's those people who like getting phone calls during dinner. Protecting yourself and, if you're an ISP or LAN Manager, your users from unwanted trash isn't as difficult as you've probably been told. Anyone can use any SMTP mailer to send mail from themselves to anywhere with impunity - unless you have rulesets to prevent it. By configuring your system to block unwanted mail, you'll only capture some percentage of spam. However, as more and more systems install these filters, spammers will certainly become more ingenious, but it will also become less and less desirable to spam, as you will have more obstacles to overcome.
People spam now because it's simple and quick.
Here's how to think about who and how to block. Reject delivery only from specific email addresses, IP addresses and ranges, and entire domain names to your local mail users if they are known spammers. Only the top-most Received message, from the relay or spammer to your mailer, is accurate since it can log a remote mailer IP number and name. This often gets their attention where using the other addresses does not. Don't run open, unmoderated list servers.