Yes, the passphrase can be as insecure as a password, but hopefully someone that is concerned with the importance of a digital signature will recognize the importance of the passphrase protecting it. Hopefully soon ISPs will now offer the ability to secure your acount via one or a combination of the following options. A good passphrase not only takes a large number of characters, but also replaces characters with less-used characters. Thankfully, once you type the passphrase a few times it is surprisingly easy to remember the key sequence as you think of the phrase. Of course, a passphrase need not be a phrase at all.
Any combination is fine, but the longer and chaotic the better.
Answer a few questions only the user would know. Can be compromised, but is pretty good for most applications. The user in turn verifies the host's private key with the host's public key which the user already has in his or her possession. Been around for a long time, somewhat obscure, but is making a resurgence. When done properly it is highly reliable and secure.