Remove the configuration key in the correct place