Don't delete dirs we need later

Hopefully this will get rid of my folder corruption issues.