Remove ignored directories