Remove index directories that we don't want to build for a while