Remove directories that had to be ignored