Remove the now ignored directories