Only run keychain if a key exists

On remote servers, we should use agent forwarding, so the key only
exists on our local machine.