If your computer is used within a company or at home, the answer is probably yes . If you are running an ISP, the the answer is definitely NO . However, even if you trust your users, you need to be careful. People attacking your computer can attempt to compromise a single user's account, and use that as a springboard for getting more access into your computer. So even if you think you can trust your users, it is best to be careful.
Users get to their accounts through the login program typically using a password .
An attacker has to subvert one of these to get user access into your computer. It is thus very important that all accounts with shell access have good passwords associated with them. Typically the standard password program distributed with Linux systems is fairly light duty. It requires passwords that are somewhat long, but that's about it. Some of the add on packages are more strict and require that the password not look like a name, or a word in a dictionary.