Remove old public key.