Remove now ignored directories