Force remove config dir if we are purging.