Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper shows how full imageNet dataset with over 14M images and 21K classes can be used to train better backbones as part of the pre-training steps for few supervised tasks. The paper claims that the contributions are mainly around cleaning up the dataset, providing official test/train/val splits, Introducing efficient training techniques and providing results on down-stream tasks to show effectiveness of the newly trained backbone. (Side note: The final cleaned dataset has only 11K categories. So probably the name ImageNet21K is a bit confusing when compared to ImageNet-1K.)
strengths: The paper shows how full imageNet dataset with over 14M images and 21K classes can be used to train better backbones as part of the pre-training steps for few supervised tasks. The paper claims that the contributions are mainly around cleaning up the dataset, providing official test/train/val splits, Introducing efficient training techniques and providing results on down-stream tasks to show effectiveness of the newly trained backbone. (Side note: The final cleaned dataset has only 11K categories. So probably the name ImageNet21K is a bit confusing when compared to ImageNet-1K.)
weaknesses: While paper can be found useful to a group of developers and researchers (especially those interested in participating in CV challenges where squeezing accuracy even for 1-2% is critical), the contributions don’t seem sufficient in my opinion for NeurIPS. The paper is an incremental work and the contribution and scope might be best fit for a related workshop. The steps around cleaning the dataset is pretty standard. Training schemes used for pre-trained models are interesting, while some are well known and not novel (e.g Knowledge Distillation) the other one like semantic softmax seems to have some novelty. However, I would still like to see a more thorough literature review and how this approach compare to wide range of work on extreme classification tasks or classification tasks with noisy data (http://manikvarma.org/events/XC16/index.html). Breaking up large classification problems to smaller sub-task classification is not new. Additionally the experiment are not complete in my opinion. You can refer to some of my comments below regarding the experiments. 

Additionally, one might argue the value of such dataset for the community. Increasing the size of the dataset for pre-training might not necessarily be valuable at least for a big portion of research community as it requires a significantly more time to process and pre-train backbones where the reward can be incremental improvements. The resources for pre-training such models might not be available to everyone and smaller dataset might indeed be desirable. This specifically can become a bottleneck when trying new backbones where pre-trained models don’t exist. 


Experiments:

1) One experiment that would be interesting and I wish it was included in the paper is to see is how much adding data actually helps. Do we really need all the 14M and 11K classes to achieve some of the results reported in the paper? For example, if we generate ImageNet1K, 2K, 5K, 10K and repeat the experiments, do we at some  point see that the results plateau. 

2) Majority of the experiments are still around classification. Either in videos or images. Single label and multi-label. These tasks are more likely to benefit from a stronger backbone training for similar tasks on larger images as it can better generalizes. It would be interesting to see how such backbone can increase the quality of other popular CV tasks, like detection, segmentation and etc. 

3)The improvement from official 21K pre-training to the proposed method in this paper is usually within 1-2%. However, the there is one dataset where the improvements are close to 15% (Pascal-VOC, Table 3). Would be useful if authors provide some intuition on why the improvement is much larger in this dataset. 

4)Pre-training used to be an important part of classical computer vision tasks such as detection. Mainly for faster convergence as well as small size of dataset trained for target task. However, the size of the dataset and labels are increasing every year and the impact of pre-trianing may diminish. It would be interesting to see an albation study of such experiments in this paper. 

5)The numbers reported in the paper for training time time, which in my opinion is an important aspect is shown at over 2days on 8x V100 GPUs. However, this is for when models are initialized form ImageNet1K training which is a bit counter intuitive (L:148 We initialize our model from ImageNet-1K training). I would be curious to see how much longer the training would be if ImageNet1k is not used. It is counter intuitive if one needs to include 3 rounds of training for each backbone (ImageNet1K -> ImageNet21K -> Downstream Task)

6)ImageNet-21K dataset labels are still noisy. One may argue that instead of the semantic softmax training or multi-label training one could stick with single-label training and use some higher level of hierarchy where the #labels can be reduced drastically. Would be interesting to see such experiments as well 

correctness: The claims seem to be correct. 
clarity: The paper is well written. 
relation to prior work: Please refer to my comments regarding the experiments and contributions. There are areas that I would like to see more comparison especially around semantic softmax compare with prior work dealing with large number of classes and noisy labels as deviding a large scale classification task into smaller sub-problems is not new. 
additional feedback: I have added some comments on the experiments section and how that can be improved:

Experiments:

1) One experiment that would be interesting and I wish it was included in the paper is to see is how much adding data actually helps. Do we really need all the 14M and 11K classes to achieve some of the results reported in the paper? For example, if we generate ImageNet1K, 2K, 5K, 10K and repeat the experiments, do we at some point see that the results plateau.

2) Majority of the experiments are still around classification. Either in videos or images. Single label and multi-label. These tasks are more likely to benefit from a stronger backbone training for similar tasks on larger images as it can better generalizes. It would be interesting to see how such backbone can increase the quality of other popular CV tasks, like detection, segmentation and etc.

3)The improvement from official 21K pre-training to the proposed method in this paper is usually within 1-2%. However, the there is one dataset where the improvements are close to 15% (Pascal-VOC, Table 3). Would be useful if authors provide some intuition on why the improvement is much larger in this dataset.

4)Pre-training used to be an important part of classical computer vision tasks such as detection. Mainly for faster convergence as well as small size of dataset trained for target task. However, the size of the dataset and labels are increasing every year and the impact of pre-trianing may diminish. It would be interesting to see an albation study of such experiments in this paper.

5)The numbers reported in the paper for training time time, which in my opinion is an important aspect is shown at over 2days on 8x V100 GPUs. However, this is for when models are initialized form ImageNet1K training which is a bit counter intuitive (L:148 We initialize our model from ImageNet-1K training). I would be curious to see how much longer the training would be if ImageNet1k is not used. It is counter intuitive if one needs to include 3 rounds of training for each backbone (ImageNet1K -> ImageNet21K -> Downstream Task)

6)ImageNet-21K dataset labels are still noisy. One may argue that instead of the semantic softmax training or multi-label training one could stick with single-label training and use some higher level of hierarchy where the #labels can be reduced drastically. Would be interesting to see such experiments as well

-------------------------------------------

Post Rebuttal: I appreciate the authors response. The additional experiments and ablation studies are definitely adding more isights. However, my concerns regarding the contribution of the paper and it's novelties are still in place. The work is incremental and doesn't include enough novelty. 

I don't agree with some of the comments on value of the dataset authors provided in rebuttal. There are many datasets out there for pre-training and one can't just publish a paper by using a bigger dataset to pre-train their model and show better results compare to SOTA (as there is no novelty there). So the adaption of a bigger dataset for pre-training for any paper will still require all the baselines and previous SOTAs to also be trained with the new pre-training dataset. This consequently requires a lot of computing resources and I'm surprised authors didn't provide numbers on the training time without using ImageNet 1K as pre-training step. This is the easiest experiments that authors could have conducted in the rebuttal, unless it takes a long time to train/converge such model without pre-training step that didn't fit in the 1 week rebuttal period.

I would like to stand with my original rating.
review_title: While paper can be found useful to a group of developers and researchers (especially those interested in participating in CV challenges where squeezing accuracy even for 1-2% is critical), the contributions don’t seem sufficient in my opinion for NeurIPS. The paper is an incremental work and the contribution and scope might be best fit for a related workshop.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work modifies and improves the ImageNet-21k datasets by:

* preprocessing it
* using WordNet to do semantic multi-labeling of each image
* developing a semantic softmax scheme (not technically an improvement to the dataset I suppose)

In general, this is a nice piece of work. Here are some comments:
* The authors say that "classes with few samples cannot be learned efficiently". Is this true? There's a whole lot of research out there about few-shot learning with as few as 5 or 10 labels. Limiting the dataset to 500 labels per class or more seems ... limiting.
* I think the value of doing the squish-resizing -- having the dataset be smaller -- is not that big compared to the potential loss of various data augmentation techniques that one could do on the original images. ˆ Is there a version of the data that is not resized?
* I don't necessarily buy the argument on line 186 that multi-label training suffers from more local mimima than single softmax training. Has this been shown empirically (or otherwise) by the authors?
* The semantic softmax loss in Section 3.3 seems very tailored to the dataset itself. It's not obvious to me that any of the pieces from it are really usable outside the narrow confines of the hierarchy implicit in this dataset.
* The pretraining experiments seem thorough and the ablations are very detailed. It's clear that this dataset has practical merits so I expect that others will end up actually using it.


strengths: * A practically relevant dataset that I expect to be useful to most researchers doing computer vision & deep learning
* A thorough set of ablations that show the significance of the data vs losses used vs model vs downstream tasks
weaknesses: * I am not convinced by the need to have the images resized the way they were resized
* The semantic/hierarchical softmax loss used in the work is a bit niche, since it's custom for this dataset and I'm not sure how generalizeable it is to anything outside of it.
correctness: Not much to add besides what I already wrote.
clarity: Yes, it is well written.
relation to prior work: yes.
additional feedback: N/A
review_title: interesting improvement to imagenet-21k

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper reforms the ImageNet-21K to clean the object classes resplit the data, and organize the hierarchical class taxonomy labels. Subsequently, the authors propose a semantic softmax scheme to fully utilize the hierarchical labels and avoid the cons of multi-label training. Interesting results are presented on pretraining and transfer learning experiments. Moreover, some details are missing and need to be better clarified. And the experimental contribution is larger than the data contribution.

Overall, this is good work with extensive experiments and can help the community better use the bigger ImagetNet-21K.
strengths: + Reforming the ImageNet-21K really helps the vision community.
+ Hierarchical taxonomy labels and the corresponding scheme are valuable for many vision tasks.
+ The experiments are extensive and show impressive results on some tasks.
+ Interesting results are presented, such as the certain improvements with the reformed dataset and various basic models like MobileNet.
weaknesses: - Some essential details are missing that can make the experiments and proposed scheme more solid. For example:
1. The logit sampling with the semantic Softmax pertaining.
2.  Sec. 3.4, please give more details about the pipeline, and implementation would be better.
3. L260: MSE converges faster. Some analyses or insights behind this? Especially the relationship with the hierarchies.
4. About the difference of improvements on different datasets like ImageNet1K, Food 251, and MS-COCO. I think this is very important to analyze the proposed dataset.
5. L291-L297: Some deeper analysis about the training period, gradient, etc.?
6. Is it possible to conduct some tests to compare the proposed ImageNet-21K-P and the other large-scale datasets? This would further improve the impact and quality of this work. The authors have honestly mentioned JFT-300M in the supplementary (L686). I appreciate this.


correctness: Basically, the evaluation is sound and follows the widely-used metrics.
clarity: Well written and easy to follow.
relation to prior work: Basically good. 
But if there are some discussions about the recent large-scale datasets (object-related datasets) and the downstream tasks would be better.
additional feedback: Advice but may be impractical. Compared to the experimental contribution, the data contribution is relatively smaller. If possible, maybe more annotations to fully build the hierarchical class tree (finer class labels, e.g., the children of the highest class label) and fix the inconsistent tagging would be better. But this could be costly (this would not degrade the rating).

Overall, this paper is beneficial to the vision community, though more details and clarification would make it better to meet the high-quality requirement of NeurIPS. I look forward to the response from the authors and would adjust the rating according to the discussions.

Post-rebuttal:
Thanks for the response from the authors. My concerns about the method and experiment details are basically addressed. And I appreciate the supplemented discussion of the dataset comparison and new results. The other reviews raise some other concerns, but I think a major revision will qualify this work for this track after revision. Though I still think more hierarchy labels will make more dataset contributions as the goal of this track, I stand on my rating.
review_title: A interesting work to reform and improve the usability of ImageNet-21K, more clarifications are essential.

Meta-review: This paper received mixed reviews, leaning positive. The negative review is mainly concerned about the significance of the work. AC believes that the authors' responses have sufficiently addressed these concerns. Overall, AC believes that the paper will be a useful contribution, given that many downstream tasks can benefit. 
Decision: Accept