Save server roundtrip when checking manually imported public keys