Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a multimodal dataset of RGB+depth objects with speech and text descriptions. Experiments conducted on this dataset to analyze the difference between the speech and text descriptions. 
strengths: 1. A public available multimodal (image frames, depth point could, speech and text) dataset of household objects is proposed.
2. Experiment is conducted to analyze the difference between the speech and text descriptions and find that language groundings from transcribed speech perform similarly to models trained on text.

weaknesses: 1. There are some other better speech pre-training models, such as vave2vec2.0 and hubert. I think they could be used instead of vq-wav2vec.
2. Some part of the experiment is not clear:1) I am not sure the queries for grounded language are image frames or depth point. 2) for retrieval, when few instances are selected, how many instances are selected for each query,  andhow many positive and negative samples in them?

correctness: I am curious about the experiment result using some types of objects for testing which are not shown in the training data. Is it much worse than the current setting or not?
clarity: This paper is well written and easy to follow.
relation to prior work: The discussion between this paper and other multimodal datasets should be added.
additional feedback: Is it possible to map inputs of different modalities into a discrete space, such as the VQ space?
review_title: a multimodal dataset of objects with image frames, depth point cloud, speech and text

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper suggests a well-constructed multimodal, multi-labelled dataset for household object description. The modalities include RBG with depth, typed language (ground truth), speech audio and its transcript (total 4). The paper describes why those modalities are chosen, with which content the dataset is created upon, the procedure and some analytic experiments. Beyond just that the proposed dataset contains various modalities, it gives a link between those modalities by grounding on the same description, and roughly shows how those modalities can contribute to the downstream tasks. Whole procedure is clear, and the dataset seems to be a nice contribution for the community that handles grounded language.
strengths: - The relation to the prior work is well stated and why this work is necessary is well demonstrated
- The proposed dataset deals with the transcript with errors, which is far effective in real world setting compared to clean text
- The construction bases on household objects which are welcomed for practical use
- Metadata for speech files may help mitigate the possible bias regarding demographics of the speakers
weaknesses: - Though the dataset contains multimodality, the descriptive sentences does not seem to make the most of multimodal setting due to their content being simple. This does not necessarily harm the merit of this paper, but it would have been better if the spoken descriptions could contain more non-verbal information compared to the text ones.
correctness: The paper incorporates substantial evaluation for the proposed benchmark, with classification and retrieval tasks, and with appropriate models.
clarity: The paper is well written, though the content is a bit difficult to follow for those who are not familiar with grounded language.
relation to prior work: The related work is well organized and it seems that the proposed dataset widens and integrates the prior materials.
additional feedback: It would be better if it is stated that the provider of spoken descriptions are informed of the fact that their voice is to be annotated with their attributes.
review_title: A solid dataset work concerning various modalities, especially RGB, depth, digitized text and spoken language

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a dataset of 207 everyday objects, each captured through pictures and (coloured) 3D point clouds from four different angles. Additionally, every object has been described a number of times by crowdsourced annotators, either in writing or in spoken language. As a result, the dataset lends itself particularly well for researching grounded language acquisition for embodied agents. 

On a mock-up retrieval task, automatically transcribed descriptions performed on par with text-based descriptions proper, and training directly with speech input decreased performance by a few percent points. The dataset therefore could be used to compare grounded training using speech and traditional text-based approaches, but is likely to require coupling with an external task to reach its full potential as a benchmark. 

The authors aim to triple the size of the dataset for publication. 

strengths: The dataset contains coloured 3D point clouds and RGB images of everyday objects - an input likely to be obtained from an embodied agent. Objects also are represented from four different perspectives to mitigate the effect of training based on iconic representations only. Together, this makes the dataset particularly well suited for research on learning in embodied agents. 
Object descriptions are provided in three variants: text-based, in spoken language, and automatically transcribed. As natural audio descriptions are costly to collect, any dataset dedicated to this type of data is a valuable asset. The three-type setup in addition also allows for comparing training with these three different inputs on embodied language learning. This might yield interesting insights because in real-world situations, descriptions are likely to be understood imperfectly and therefore fully correct text-based descriptions might not be obtainable. With their experiments, the authors provide initial evidence that - with current ASR techniques - this might be less of an issue. 

The collection of the data is described clearly, and the data itself is structured well and straightforward to process. Recording audio from crowdsourced participants always is a privacy risk, but the authors did their best to mitigate this by clearly informing participants and collecting fact-based audio descriptions only that - even if traced to a speaker - are likely too benign to cause any harm. The authors even went so far as to hold out large portions of the dataset as the consent form might have malfunctioned for a portion of the data collection. 

Spoken descriptions were annotated with respect to audio and voice quality as well as dialect by the authors. While there is not too much information on this aspect in the paper, this might be an interesting avenue for future research regarding bias in ASR and the processing of spoken language in general. 


weaknesses: A current limitation is the relatively small size of the dataset - something the authors are set to address by aiming to triple the number of entries for publication. 

In terms of evaluation, the paper reports model performance using a mock-up retrieval task only. In order to set a proper benchmark, I would recommend coupling the dataset as a train set with an external (pre-existing) retrieval task to realise its full potential as a benchmark. 

Having conducted a range of data collection studies on AMT myself, I’d like to see a little more info on the (expected) hourly pay of participants, required qualifications (e.g. region, language, education or AMT stats) and whether (and how) the authors address the almost inevitable noise in the data collected. 

The section on annotating the spoken language descriptions needs extending if it is to be made a proper contribution to the work in the paper. 

correctness: The dataset is carefully designed and collected in a sound way. The paper clearly reports the data’s key features and statistics. As mentioned earlier, I would recommend coupling the dataset as a train set with an external (pre-existing) retrieval task to realise its full potential as a benchmark. 

clarity: The paper is clear and well written.

relation to prior work: I don’t have a great overview of the current state of the art or other resources in grounded language acquisition, but my impression is that the authors clearly position their work in relation to previous work and the field in general. 

additional feedback: You clearly want to highlight the different features and potential applications of your dataset. Maybe try to sharpen your focus a little more on a selection of those features to improve coherence. Otherwise it’s a great resource and a well-written paper.
review_title: Small but carefully crafted resource for an emerging field

Meta-review: Reviewers univocally agree that this paper suggests a well-constructed multimodal, multi-labeled datasets for household object description. Paper is also well-written, well-situated with prior work. The dataset is carefully crafted, has high potential for multiple domains. A few questions raised by reviewers are addressed/responded by authors. 
Decision: Accept