Remove the compromised and now deleted keys