The private key is never given to anyone else. Then the only way a third party would be able to decode messages encrypted to you would be to get a hold of your private key - and they would have to steal your computer for that. If this ever happens to you, remember to tell your lawyer about the Steve Jackson Games case, which set a precedent for the legal grounds required for a federal agency to confiscate a person's computer. The private key on your computer requires a password to use, which should be something you can remember and don't need to write down. If they have your computer, they can probably hack this password, but it would require serious effort.
Some people encrypt everything on their computer and keep the key on a floppy disk.
That would be the safest way to go. Let's say you want to send me an encrypted message. You will need to have my public key. This public key is public information, and providers are beginning to archive them for their customers. A public key is used by other people to encrypt a message that only you can decode, using the private key that is the other half of the pair.