Fix passphrase handling logic