Remove old keystore, create config dir if necessary