Remove exclude paths before removing directories

This will potentially limit the amount of fs.stat calls needed to be made
v0.1.0