Remove generated public key and add to ignore list