Remove the now ignored directory