Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes two datasets: VCE (samples containing emotion label distributions) and V2V containing pairwise comparative labels in terms of degree of pleasantness. These datasets can be used to predict emotional responses, and wellbeing of viewers. Authors train standard CNNs on these datasets, and observed that while these networks perform well, they do leave a significant performance gap to be fulfilled.
strengths: Although, not altogether novel (since there have similar smallscale datasets), these datasets are largest of their kinds, which would allow end-to-end representation learning. 
In my opinion, these building such dataset would require significant efforts, which adds to the value/contribution of these datasets. Authors have diligently designed dataset collection/annotation protocols. Although, please see Weaknesses section.
weaknesses: Annotators were allowed to use audio signals in order to annotate, so the labels might not be guaranteed to have been grounded in video signal alone. Although, authors did ask annotators to take notes of samples for which they relied heavily on the audio signal. I think for annotators making such choices might be somewhat ambiguous. Such ambiguity might be ultimately reflected in annotations as well. Can authors please lend their views in this regard? 

Also, authors remove the audio signal from the final dataset. Why not provide it to allow multimodal analysis if desired? In my opinion, removing audio signals just hurts the overall utility of the dataset. Unless, the authors want to get another publication (saying this with all due respect) by enhancing this dataset with audio included version of the dataset. I think authors should provide the accompanying audios.
correctness: Authors seem to have quite diligently designed the annotation protocol.
clarity: Yes, the paper is well written.
relation to prior work: Sufficiently discussed prior work.
additional feedback: N/A
review_title: Largescale Multilabel Datasets for Predicting Emotional Responses and Degree of Pleasantness

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: For understanding how viewers feel while watching videos, the authors introduce two large-scale datasets for predicting emotional state and wellbeing of viewers directly from videos. The dataset contains 60,000 videos with human annotations for 27 emotion categories. 
strengths: - The paper highlights the motivation and idea in very efficient manner.
- This paper introduces two large dataset to understand the human emotions after watching a video.
weaknesses: - The background of 400 annotators are not mentioned in the paper, if the majority of people are from a single locality/country then the emotions annotated by them can be biased. As population from one diversity can feel completely different from populatio from different diversity.
- If the videos contains mixed emotions like first sad then happy, such type of videos will have very subjective label depending on the person.
correctness: The experiments seem correct.
clarity: The paper is well written.
relation to prior work: Properly mentioned in the paper.
additional feedback: - Emotions based on different diversity people can be collected.
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors introduce two datasets, Video Cognitive Empathy (VCE) and Video to Valence (V2V). The datasets contain over 60,000 short videos that were annotated with emotional content and general emotional valence by crowdworkers. The authors train machine learning models on the datasets and demonstrate that some models reach good performance, but that there is also room left for further improvement.
strengths: The utility of the datasets is very clear, both for direct application as well as more foundational AI research. The datasets are large and constitute a major contribution. 
weaknesses: I could not identify any signficant weaknesses in this work.
correctness: All claims made seem correct.

clarity: The manuscript is written very clearly. 
relation to prior work: The relation to previous work is established well through in the related work section. 
additional feedback: N/A
review_title: Great dataset, solid paper

final_score: rating: null
review_content: The manuscript states "Our datasets and experiment code can be found at github.com/hendrycks/emodiversity."
Unfortunately, this URL results in a 404 error. Please fix and/or comment!
review_title: Dataset availability

Meta-review: metareview: Overall, the paper is capturing a really timely and increasingly important topic. Although the authors mention that they want to make their dataset particularly useful for academics/future research, I believe, that the topic is of great interest for companies, making the raised ethical considerations even more important. A critical reflection upon possibly ‘unintended uses’ can’t be emphasised enough, as well as possible biases in the data. However, I appreciated that the authors have done a fantastic job in their response and strengthening their article. Moreover, all reviewers recognise the utility and relevance of the proposed dataset and the authors did provide satisfactory responses to all raised concerns/comments. I believe this paper will stimulate some interesting and hopefully critical reflections on how to use/or not use the dataset and future improvements to overcome dataset biases. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept