Remove stale directories for those, too