Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a large-scale meta-dataset that consists of 30 datasets from 10 domains. Extensive competition and challenges have been organized on the benchmark. Currently it focuses on few-shot classification but in the future it has the potential to be used towards the study of other meaningful problem on learning a general visual perception model.
strengths: Strength:
1. The benchmark seems to be constructed soundly and the competition is well-organized.
2. Many design decisions in this benchmark point in the right direction, for example, different versions of the challenge for different computation resources, community-driven benchmark, etc.
3. Empirical discoveries in section 3.2 are informative for future research.

weaknesses: Weaknesses:
1. As vision-language pretraining is developing rapidly, it may be beneficial to explicitly consider a track of cross-domain zero-shot setting or few-shot setting with text information and vision language pretrained models.
2. Uniformly scaling images to the same size may create some difficulty for more fine-grained datasets as they may need larger resolution to provide the necessary cues. However, the design decision is understandable.

correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Overall the reviewer believes the benchmark is very meaningful to the community. Some minor concerns are listed in the weakness section.
review_title: Meaningful Few-shot Benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present Meta-Album, a few-shot image classification benchmark dataset. Meta-Album consists a collection of 40 datasets from multiple domains. All the datasets are pre-processed for accessibility and uniform benchmarking. The authors provide 3 versions (Micro, Mini, Extended) of Meta-Album to facilitate in low resource setting. Further, the Extended version contains more than 1.5M images from 10 domains. Evaluation with benchmark methods like ProtoNet, MAML is given.
strengths: 1. Large number of uniformly formatted datasets from multiple domains
2. Multiple versions with varying sizes of the dataset is provided for better accessibility
3. Baseline methods like ProtoNet, MAML and MatchingNet are evaluated and analysed.
4. Apart from more standard N-way k-shot baseline on any-way any-shot is also provided.
weaknesses: 1. Baseline on more SOTA approaches like iMAML, will be good to have.
2. Domain specific benchmarking is missing, for example how different baseline methods perform on different domains.
correctness: Yes, The benchmark dataset is constructed in a sound way.
clarity: The paper is well written and easy to follow.
relation to prior work: The paper clearly states how it is different from previous datasets in meta-learning.

additional feedback: N/A
review_title: Interesting diverse meta learning benchmark with easy to access versions

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This manuscript introduces an image classification dataset for few-shot learning, transfer learning, and other tasks. It contains multiple domains and features various scales, with large numbers of images. Extensive experiments are conducted on the proposed dataset to accomplish few-shot learning under standard 5-way 5-shot setting, cross-domain setting and any-way any-shot setting, proving its practicability. 
strengths: 1.	The dataset is large and well-organized, stemming from multiple domains with various scales and retaining class hierarchy annotations.
2.	It is extensible and growing.
3.	It comprises 3 different versions: micro, mini, and extended, which could well match users’ computational resources.
4.	The authors adequately compare their work and the prior works.

weaknesses: 1.	This work can be improved by comparing more state-of-the-art cross-domain few-shot approaches, such as [1][2][3][4], instead of only classic in-domain few-shot methods.
2.	I would encourage the authors to check the spellings, such as “protocl” in section 2.1.

[1] Li P, Gong S, Wang C, et al. Ranking Distance Calibration for Cross-Domain Few-Shot Learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 9099-9108.
[2] Li W H, Liu X, Bilen H. Cross-domain Few-shot Learning with Task-specific Adapters[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7161-7170.
[3] Dvornik, N., Schmid, C., and Mairal, J. Selecting relevant features from a multi-domain representation for few-shot classification. In European Conference on Computer Vision . Springer, Cham. 2020: 769-786.
[4] Yue, X et al. "Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation."  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

correctness: It would be better if more cross-domain few-shot approaches are conducted on the proposed dataset. Please refer to the Weakness.
clarity: This paper is well-written and clear stated.
relation to prior work: The manuscript clearly discussed its differences and connections with prior works in detail.
additional feedback: Please refer to the Weakness.
review_title: Review of the Meta-Album

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduced an image classification meta-dataset named Meta-Album for cross-domain few-shot learning. It is consisted of the image data collected from 10 different domains, and organized into three versions (miro/mini/extended) . The dataset is well-processed and formatted, and the guidance is easy to follow. 
strengths: Strong points:
- The dataset is well-processed and formatted.
- The paper is well written and organized.
- The guidance and the code in github is easy to follow. 


weaknesses: Weak points:
- Meta-Album formats all images into 128$\times$128 pixel, how does it influence the difficulity of the meta learning tasks? Please ensure that it doesn't influence the quality of the image data. 
- I'm curious about how does the performance of the meta-trained model vary in different domains? In other word, how to know which domain is more difficult. 
correctness: Yes
clarity: The paper is well written and easy to follow. 
relation to prior work: Yes. 
additional feedback: Please see the weaknesses.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors provide a large-scale dataset and benchmark for multi-domain few-shot learning with 30 sub-datasets from 10 different domains. They have three versions (in scale) of data to meet the users’ needs. They also provide implemented baseline methods and data loaders for the challenge. In the paper, they also compared several baseline meta-learning methods such as MAML, matching networks, and prototypical networks in within-domain and cross-domain experimental settings for k-ways n-shot learning, and any-way any-shot learning tasks. 
strengths: - The paper is well motivated that there’s lack of challenging but realistic cross-domain few-shot learning dataset and benchmark in the literature;
- The paper acquired a very large-scale and relatively-diverse few-shot learning dataset;
- Popular baseline methods and data loaders are well-prepared for users;
- The learning tasks (within-domain, cross-domain, fix-way fix-shot, and any-way any-shot) are well-defined with a range of different levels of difficulty;

weaknesses: - By providing multiple versions of the dataset, it might introduce difficulties for researchers to choose which one to use and to compare their study to existing works (which might leverage different versions);
- The domains in the existing dataset are relatively close to each other. The authors might want to introduce images from the medical domain such as histopathological images, x-ray, and MRI, to make the tasks even more challenging;

correctness: The claims in the paper are valid and the dataset is constructed in a sound way. The experimental design and evaluation metrics are well defined.
clarity: The paper is very clear and well written.
relation to prior work: The authors thoroughly discussed the relevant literature.
additional feedback: N/A
review_title: Comments

Meta-review: metareview: The reviewers were unanimous in recommending acceptance for this paper. Multiple reviewers commented on the dataset/competition being well organized, and the experimental baselines provided in the paper were generally appreciated. There were some suggestions of expanding the baselines in the paper or further diversifying the domains; I tend to agree that these would be nice, but not necessary for the current paper's acceptance. I recommend acceptance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept