Fix passphrase cache service

For symmetric keys always assume that they need a passphrase.