Like with smartphones, you can protect both your phone and USB drive with a password. This method works even if you don't have the password for everyone on the drive.