Remove configuration file. It exposes private key