Use the public key directly to verify the server host key