Remove the now ignored directories and files