Remove public key if the key is passed as nil, but not otherwise.

There was a weird case with reloading the user in the test if the public key had been unset.