Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper unifies a large number (up to 15) of current multi-modal datasets involving 10 modalities and 6 research areas. Being able to support a variety of multi-modal tasks. A comprehensive toolkit called MULTIZOO is released, containing 20 different methods, serving as a good starter code.

strengths: 1.	This dataset integrates previous efforts towards multi-modal learning, covering diverse domains, and could serve as a good basis for benchmarking methods.
2.	There are novel ideas and concerns for the use of this dataset, including qualifying the time and space complexity, assessing the robustness when some modalities are missing, etc. All these could provide more information about the advantages/disadvantages of a given model from different aspects instead of simply considering quantitative results.
3.	Code for various classical multi-model methods is provided, ensuring reproducibility and facilitating future research. Standard evaluation tools are provided to ensure a fair comparison.
4.	Detailed analysis of different factors is conducted (e.g. generalization, performance, complexity, etc.). Limitations are discussed.

weaknesses: In each of the 6 distinct research areas, the annotation modality and focus of each chosen dataset may not be consistent. For example, AV-MNIST and Kinetics are both grouped into “multimedia”, but is it necessary for a model which could understand human actions well to be good at recognizing handwritten digits?


correctness: Are there some logical rules for choosing these research areas? The spanning of these research areas should be systematical with some underlying hierarchies (such as ImageNet). 
clarity: Yes, it is easy to follow.
relation to prior work: This dataset involves multiple research areas, which makes it distinguished from datasets focusing on a single domain. The scale and the number of tasks it could support are also impressively large.
additional feedback: When unifying datasets, semantic granularities are suggested to be considered.
review_title: This work introduces a large-scale unified dataset involving distinct research areas, such effort is a good trial.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark for multimodal learning. It collects data of 10 modalities for 20 prediction tasks in 6 research areas. A set of models are implemented and analyzed. The benchmark targets at evaluating multimodal models' abilites including generalization, efficiency, and robustness.
strengths: 1. The benchmark is well motivated. The writing is clear and easy to understand.
2. The evaluation of the benchmark is sufficient. It analyzes the performance of existing well-known techniques in feature interaction and optimization. The reviewer does get some new insights about these well-known techniques.

weaknesses: The reviewer does have some concerns about the paper.
1. Although the benchmark contains about 10 modalities. The benchmark mainly focuses on simple classification tasks. The reviewer has doubts that whether such data can well represent the complexity of the real world.
2. It is also interesting to extend the existing benchmark to generative tasks like captioning. 
3. Also, since the benchmark contains data of 10 modalities, it will be interesting to see whether the training data in one domain can help the learning of the other domain. For example, it seems that the learning of language can help with audio recognition. However, whether the learning of tabular data can help with the learning of audio recognition seems to be unclear. It will provide more insight to analyze how different kinds of data cooperates with each other and when they will benefit the learning of each other.
correctness: It looks correct.
clarity: Well written.
relation to prior work: Yes, it is.
additional feedback: None.
review_title: A meaningful benchmark for multimodal learning

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a new benchmark for multimodal representation learning. It is composed of 15 different datasets that exist in the literature, covering 10 distinct modalities (language, image, video, audio, etc.) and different prediction tasks. Each dataset in the benchmark is of different size and captures different application areas like affective computing, healthcare, robotics, finance, HCI and multimedia processing, which both require integration of multiple modalities. The authors construct a zoo of multimodal algorithms for data preprocessing, fusion paradigms and optimization objectives, and evaluate their performance, complexity and robustness on the suggested benchmark. The experimental analysis reveals important observations and sheds light on possible future directions for multimodal models.
strengths: + The paper comes with a standard toolkit enabling an easy to use and easy to extend interface for end-to-end training training of multimodal models. 
+ The authors provide a model zoo consisting of implementations of 20 algorithms with various differences, and a leaderboard (not active at the time of this review). 
+ The datasets show great diversity in terms of application areas, modality types as well as prediction tasks. 
+ The experimental evaluation highlights the benefit of having a modular framework to test the effectiveness of certain components (e.g. fusion strategy) on a variety of datasets. 
+ The paper reports state-of-the-art results on 9/15 dataset by exploring the performances of methods/components on domains/datasets not tested on before.
weaknesses: - The paper mostly focuses on evaluating different fusion strategies and algorithms. See my additional comments to the authors.
- The selection criteria of the research areas and the datasets are not explained in a transparent manner. 
correctness: The authors make use of already existing datasets to construct the proposed benchmark. In a nutshell, it considers a modular design, which allows for evaluating different combinations of preprocessing steps, fusion strategies and optimization schemes at ease. Moreover, it includes standard evaluation metrics per each prediction task. In those regards, the claims made and the evaluation setup considered in the submission sound correct.
clarity: The paper is written well. That said, due to the comprehensiveness of the proposed benchmark, most of the technical details regarding the datasets and the models are given in the supplementary material, which decreases the readibility of the paper and there exists some repetitions.
relation to prior work: I am not aware of any such multimodal benchmark of this scale and variety.
additional feedback: Overall, I think the paper is in the right direction but needs a revision considering the points I raised above. 

First and foremost, the decision process for including the selected 15 datasets is not clearly explained. I agree with the authors that they pose certain challenges. However, I think a better way could be to announce an open call to the community for task and dataset proposals, similar to what’s been done for the SuperGLUE benchmark (Wang et al., 2019) and the recently proposed GEM benchmark (Gehrmann et al., 2021).
(The authors carefully addressed my concerns and added a section on their plans to extend their benchmark)

As I mentioned before, the current version of the benchmark solely concentrates on benchmarking various different fusion strategies. As a person interested in multimodal research, I would like to see experiments on evaluating the performances of the methods under a multi-task learning setting. As the datasets include overlapping modalities, this could be a really interesting research direction.
(The authors added new experiments to respond my suggestion)
review_title: Good initial attempt to introduce a multimodal benchmark dataset for representation learning, though the current version mostly focuses on evaluating fusion strategies and the selection process for the included datasets is not very transparent.

Meta-review: This paper proposes a multi-scale benchmark integrating information from multiple heterogeneous sources of data. It is an important area and a very useful contribution to the community to assess the generalization capabilities of the ML models across domains/modalities and modality robustness (robustness to noisy and missing modalities.)

All the reviewers suggested the paper for acceptance without any significant concerns. It seems like the authors of the paper considered the reviewers' suggestions and improved the paper to include:
1) New datasets for retrieval and classification addressing the concerns pointed by Reviewer jKQc.
2) Transfer learning and co-learning results are addressing the comment made by Reviewer jKQc.
3) Multitask learning results addressing the feedback by Reviewer xwdp.
4) Clarifications addressing the concerns of Reviewer nPgz, and others.
5) Call for contributions from the community addressing the feedback by Reviewer xwdp.

I think the updates and the paper's responses in their current form address some of the important issues raised by the reviewers and I think this benchmark can be a good contribution to the multi-modal learning community.
Decision: Accept