Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Given that most popular vision datasets are easy for humans, this paper proposes a method to collect ambiguous data by extracting images from videos. Using this method, the authors collect a sizeable dataset, and subjective human event recognition uncertainty scores on a subset of it. They find that SOTA situation recognition models perform poorly on images where humans are <80% confident. Training on ambiguous data alleviates this problem. Finally, using model calibration approaches, they achieve low MSE between model and human uncertainties on a binary event classification task, relative to the baseline.
strengths: 1. Strong motivation: Ambiguous images are very common in the real world and so it is important to include them while testing models to obtain a more accurate measure of real-world performance. Moreover, most existing datasets are simple for humans and therefore do not capture the complete range for human performance. Harder datasets allow for a fairer comparison between models and humans. The dataset proposed in this paper contributes effectively towards both these issues.
2. Extracting images from videos to increase ambiguity and reduce bias towards saliency is a very smart idea.
3. Methods used in this paper are systematic and rigorous. From considering 2 variations of the uncertainty rating task to filtering out bad annotators to using FrameNet templates to measure semantic similarity, the authors do a good job of being objective in their experiments.
weaknesses: The task used to collect data varies a lot between experiments in the paper. The DAI dataset is collected on 20 event types. Human data is collected on only a subset of 6 event types. Humans are asked to report uncertainty scores given ground-truth label while networks, in all but the last experiment, are evaluated on a verb prediction task, assuming that it is completely indicative of situation recognition. Why not just train networks on situation prediction. Training paradigm assessment uses a subset of only 4 event types. Uncertainty quantification task on models is done using a binary classification task. All these variations make it very difficult to interpret results and makes one wonder if say, verb prediction accuracy is indeed a suitable for comparison with human uncertainty ratings. Why not train models on a situation prediction task? It would help if the authors would provide justification for why such subsets were selected and why nature of tasks were so varied.
correctness: All aspects of experimental design and data collection are sound, to the best of my knowledge. 
clarity: The paper is well-written. Figures and captions are useful to understand motivation, experiments and results.
relation to prior work: Relevant prior work has been clearly outlined in the Related Work section.
additional feedback: I have some questions which will help me make a more informed decision:
1. Why is the ResNet50 idea only to select frames for short videos? Why not for all? For long, relatively static videos, how do you ensure diversity in images?
2. Were subjects allowed unlimited response time? 
3. Why is RN+SD (trained on standard dataset) so much better than JSL (trained on SWiG) if both were trained on high-certainty images?
review_title: Useful dataset, smart approach

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces the DAI dataset, a Dataset of Ambiguous Images. While previous work tends to focus on creating datasets with images that are a challenge to models, this work focuses on images that are challenging for people and provides an opportunity for investigating model capabilities on this cognitively interesting set. The dataset is comprised of images extracted from youtube videos so that they capture a wide range of salience and noise, and were selected based on predefined event types. The authors then compare the confidence scores of humans and models as to whether the image represents a certain event, providing a basis for future model evaluations. They also find that training models based on more ambiguous data increases their accuracy, underlining the value of such noisy image datasets.
strengths: - The dataset seems a valuable resource for the situation/image recognition domain.

- The dataset construction seems well thought through.

- The human-perception-centric perspective is a useful angle that allows for the exploration of new questions in the field.
weaknesses: I find that the paper is sometimes not very clearly written. I'm listing my concerns here and I'm happy to increase my rating once they're clarified. [EDIT: I increased the paper score from 5 to 7 since the authors addressed most of my concerns in the author discussion here and I believe that the general proposed changes to the paper will improve its overall quality.]

- Overall, I found the verb/event terminology to be slightly confusing. The work seems motivated through situation recognition which the paper describes as "identify[ing] the verb and corresponding semantic roles (e.g. subject, object, place, reason, etc.)". If I understand correctly, the human subjects were asked for their uncertainty judgments based on the event type (e.g., "wedding") whereas model uncertainty was established based on verbs. If so, it's not clear to me to what extent they can be compared. It would be useful to include a clarifying sentence on the verb/event terminology used and how each of them is used in the experiments and argument of the paper.

- While I enjoyed the analysis of the sources of annotator disagreement (Section 4.2), it seems that the core sub-dataset remains underexplored. Given the aim of the paper to have a dataset of ambiguous images, it seems that images that are rated at around 50% confidence with high agreement comprise the actual dataset that falls under that definition. Was there any specific analysis done on the high-agreement data? The 50% bin is represented in Table 1 and 2 but did specifically the ambiguous data in those middle bins have any effect on the results reported in Table 3? Or is it simply the noisier data that causes the results?

- I'm not quite sure about the goal of the experiment in Table 2. If the goal is to achieve higher accuracy on the verb classification across confidence bins, training on DAI data is clearly beneficial. However, I'm unsure whether another goal was to have a model that is similarly calibrated to human certainty in which case the lowest confidence bin is not well calibrated. A discussion of this would be useful.

- In Section 4.1, the authors use Spearman correlation for analyzing inter-annotator agreement. Since other metrics for inter-annotator agreement such as Fleiss' Kappa or Krippendorff's alpha are specifically designed for such goals, it would be great to include an argument why Spearman correlation was used instead.

- The related works section is not quite clear yet on how the presented work is similar and different from previous work. While it repeatedly states that other work is similar and this paper is different, how exactly it is similar and different would be useful to include. (Specifically in section 2.2 but it's a general recommendation for all of section 2.)
correctness: It seems that the dataset was constructed in a sound way.
clarity: Overall, the paper has a structure that can be easily followed and is writing style is very clear. However, as mentioned in the weaknesses, I still have some remaining questions which I hope can be clarified in the paper as well.
relation to prior work: I think the paper could be more clear about in what ways it comes apart from specific work mentioned in the related works section.
additional feedback: The figure caption for Figure 5 would benefit from a short summary of what it's showing (for instance "Examples for potential sources of annotator disagreements").
review_title: Promising dataset of ambiguous images but with remaining questions

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Focusing on the visual event classification task, this paper presents a benchmark dataset DAI to improve the robustness of existing models against ambiguous event images. The images are scraped from YouTube videos and annotated by carefully selected annotators, which is an admirable work. In the experiments section, the authors provide three use cases of DAI.
strengths: DAI is carefully constructed following a strict process. It enriches the datasets for the visual event classification task.

DAI will likely be used by the community as a real-world test set to (1) assess the model's robustness/generalization ability; (2) improve existing models or develop better models; (3) develop new model uncertainty quantification approaches.


weaknesses: The presented dataset is not yet publicly accessible. 

In Section 4.2, The difference between **background knowledge** and **uncertainty quantification strategies** is not clear. It seems that they are mutually inclusive.

The experiments are extensive but the results discussion can be improved:
1. In Section 5.1, when presenting the results, will it be better by showing top-10 to low human judgments? e.g., for 0-20%, 20-40%, show the top-10 performance.

2. In Section 5.2, Table 2 shows that by augmentation, the model (RN+AUG) can achieve better results on DAI than simply using the standard dataset (RN+SD). Although the performance on DAI is worse than RN+DAI, the performance on the standard test set is better. Thus, I am wondering if increasing the diversity of augmentation methods can further improve the model's performance on DAI and reach a similar level as RN+DAI.

correctness: The construction design and implementation are appropriate.
clarity: The paper is well-written and easy to follow.
relation to prior work: The discussion of related work is good, but I have one doubt: is there no benchmark dataset of ambiguous data for event classification except the medical images? From my understanding of Section 2.1, DAI is the first dataset for this field.

 
additional feedback: Please refer to the **Weaknesses** and **Relation To Prior Work** sections.
review_title: Useful dataset for robust visual event classification

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a new method to generate ambiguous imagery of specific events with human judgements to create a dataset for a robust event recognition benchmark. By applying their strategy, they construct DAI, a dataset of ambiguous 12,000 images, 1,800 of these with human uncertainty judgements. The authors claim that event classification models trained on ambiguous data are more robust to ambiguous images. To demonstrate that, they apply the following steps:

1. The authors use different verb predictors (JSL, GSRTR, CoFormer) on DAI to generate verbs with their probability from ambiguous event images in DAI and transfer them to events using the FrameNet ontology.
2. The authors compare the generated event probabilities with their corresponding human uncertainty judgements and show expected correlations. They show that a higher number of verbs significantly supports the allocation to the true event type (5 vs 1).
3. The authors compare the impact of different training datasets and demonstrate that verb classifiers have the highest accuracy on DAI when they are trained on DAI (9% higher than augmented standard data). They also show that the mean standard error against human judgements is smaller when trained on standard data, which seems contradictory to their hypothesis that training on ambiguous data will  lead to results that resemble human uncertainty.
strengths: - The paper is well written and clearly understandable.
- The authors contribute a human uncertainty dataset of 12,000 images with human uncertainty judgements.
- The authors conduct extensive evaluations with different state-of-the-art methods to have a larger fundament for the evaluation.
weaknesses: The authors successfully show that a model trained on DAI is indeed more accurate on such data (while being less accurate on standard data). However, the experimental results do not strongly indicate that ambiguous training data leads to more robust results or approximates human uncertainty in general. There are numerous studies that the usage of soft labels in training can enhance the robustness in classification tasks. While I appreciate the extensive evaluation of the study, the derived take-home message has no strong new insight.
correctness: The results are reasonable and show no hint of being not correct. 
clarity: The paper is well-written in general and features a clear structure.
relation to prior work: The authors position themselves suitably in prior work and present state-of-the-art methods in their application fields.
additional feedback: A stronger emphasis on how to interpret the results and how they are significant would be helpful to strenghten the impact of your study. How do you interpret the smaller HUJ MSE, when the model is trained on standard data? The 'Experiments' section should start with a brief overview of the three experiments, why they are done and how they are connected to simplify the readability.
review_title: Ambiguous Images With Human Judgments for Robust Visual Event Classification

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors discuss the issue of uncertainty inherent in human annotations. In most cases the uncertainty is not quantified and thus ignored during the training. This leads to a poor model performance due to noisy images and the underyling aleotric uncertainty.

The authors propose a new collection process for ambiguous images with uncertainty quantification, present at dataset of 12,000 images collected via the proposed process and quantify the impact of uncertainty on the classificaiton.
strengths: - The topic is of high importance and often not considered
- The work is clearly structured and gives a good motivation
- The method and results are described in detailed and thus are easy to follow
- they investigated different options like the task variants A & B
- The collection process is easy and reprocducible. I especially liked the criteria for selecting appropriate annotators because it was easy and efficient.
- Issues like the lower performance of RN+SD in line 263 are highlighted and appropriately discussed.
weaknesses: - The evaluation is mainly based on the metric Accuracy (ACC). However the authors want to measure the difference between two distribution. In my opion a score like the Kullback-Leibler Divergence would be more suited. Additionally, the expected calibration error (ECE) (https://arxiv.org/pdf/1706.04599.pdf) would also  be very suited for the analysis.
- The authors discuss the issue of overconfident models and I would recommend adding references to ECE (a metric to measure this overconfidences) at these parts
- The standard deviations are missing which limits the interpretability. This makes it for example difficult to estimate the impact of the uncertainty quantification methods in Table 3.
- As discussed in the limitation only six annotations per image were used and the uncertainty assingments show an interperson variablity. These issue limit the generalizability of the scores.
- Additionally, the issue of intraperson variablitly (over time) is not discussed. I would expect that the same person will rate the same image differently after a while. This issue impacts the complete collection and as such must be addressed. Especially,  in the source of annotator disagreement this topic was not mentioned.
- The authors defined and annotated 20 event types. However often only a selection of classes is evaluated. For example in setion 5.2, why do you use 4 classes instead of all 20. If there is a reason please state it otherwise it looks like you selected the 4 best working classes for demonstration purposes (cherry picking). 

correctness: - In line 191 you state that inter-annotator variance is a form of epistemic (model-driven) uncertainty. I disagree on this conclusion. Inter-annotator variance should be considered aleatoric uncertainty (contained in the data). The epistemtic uncertainty could be lowered with more training samples, however in your example the poor label quality due to the missing uncertainty quantification limits the model performance. The discussion about inter- and intraoberserver variance is also important here because this illustrates that the uncertainty is already present in the data.
clarity: Yes
relation to prior work: In general the background is adequatly described. 

However, some in my opinion highly related papers are not discussed at e.g. other ambiguous datasets (CIFAR10H https://arxiv.org/abs/1908.07086), discussion about ambiguity noise in current dataset (http://arxiv.org/abs/2006.07159), alternative treatments of ambiguity via clustering (https://www.mdpi.com/1424-8220/21/19/6661) and benchmarks for uncertainty quantification (https://arxiv.org/abs/2106.04015).
additional feedback: Please verify that you have the right to use the selected videos to create a new datasets. I'm uncertain if intelluctual property or any other copyright you prevent from the legal usage.
review_title: Ambiguous images classification

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This work presents a method to generate dataset of ambiguous images (DAI) which focuses on images with challenging content. Accordingly, authors generate 12k ambiguous samples, out of which a test set of 1.8k samples have 6 manual annotations of human uncertainty judgements each. The authors empirically show that using DAI improves results for ambiguous samples and the human uncertainty can be used for model calibration.
strengths: The authors tackle a relevant problem in the literature and propose a setup where humans can annotate a "human judgement score" to represent their confidence of an event type based on a snapshot from a video sequence. 
weaknesses: It is not clear whether the image collection procedure is ideal. Authors state in line 551 that if number of time frames for a given video permit, every 240th frame was sampled. If not, then a clustering algorithm is run on 36 evenly sampled frames from a video to decide on the 6 frames to be used from the given event type video. If the former case applies, the snapshots in the datasets can be very arbitrary. If the latter case applies, (i) this implies the video is less than 48s for a 30fps vid, and (ii) the sampled 36 frames are simply ~1s apart snapshots that can again contain arbitrary content, not necessarily strictly related to the event type that the snapshots will be labeled as.

It is not clear whether the annotation procedure is optimal for best value and there are ambiguities. Line 175, variant A implies that an annotator at a given time, gives human judgement score for 6 images of the same event type, which count towards 6 separate annotations. Variant B, however, asks annotations for a given event type despite showing 3 images of another event type. Are the annotations for the other event type ignored? How are they aggregated into the annotations? Based on the manuscript, it seems that each test sample has six annotations of judgement score only for the corresponding class.

A major missing information on the collected data is regarding the standard deviation of uncertainty judgements for images which have a low median judgement score. This could give insight as to whether low judgement scoring samples are in agreement across the 6 annotators or not. Low variation among low judgement scoring samples could imply the sample is simply unrelated to the event type.

It is not clear how the bins in experiments are created: For example, when considering samples that had 0-20% human-judged ambiguity, do the authors look at the average judgement score across 6 annotators and see if it falls within interval [0,20]? If so, is the mean value the right metric to aggregate? 

Sec 5.2. experimental setup is not clear. It seems that different compared setups (SD vs AUG vs DAI) use different number of sample and classes to train the models. From paragraph starting at line 253, it would seem that RN+DAI would be trained on 7 event classes whereas others would be trained on several hundred classes. However, the paragraph starting on line 261 state that RN+SD is only trained on four event types. Furthermore, line 266 mentions "top-5 scores", which would not make sense if the models are trained on four event types. Is the top-5 of Table 1 is compared with top-1 of Table 2?

It is not clear what is the advantage of training on DAI if the training set has no HUJ. If the authors believe training models with samples that are loosely related to an event type is beneficial, they should potentially benchmark their work with respect to a collection of different preprocessing/data augmentation schemes.
correctness: There seems to be a crucial flaw with both the collection and the annotation of the datasets. 
Image collection is done in an automatic way using unconvincing heuristics, with a high chance of collecting unrelated snapshots from a given event type video.
If Variant B is used during annotation, then it implies there are either effectively half the number of annotations for each event type, or authors apply a heuristic not mentioned in the manuscript to account for the lost annotations. Furthermore, it is not clear what percent of the test set is annotated using Variant A (or B).

clarity: Hard to follow overall, especially the Experiments section. The discussion over the results need to be rephrased and the points authors want to make should be clarified. 

relation to prior work: Authors list some key literature from recent years where aleatoric and epistemic uncertainties are modeled through different preprocessing of data and optimization procedures. Furthermore, authors also separately refer to works in cognitive science regarding human uncertainty judgements and state that works in similar nature to this manuscript are very rare.

Nevertheless, novel contributions in this manuscript are not clearly distinguished with respect to findings of prior works.
additional feedback: Potential typo: Fig. 4 caption, "high uncertainty", or received a mean human judgement score at or above 95%,...
review_title: Hard to follow text, not convincing image collection procedure

Meta-review: metareview: Overview:
Accept (poster)

The reviews for “Ambiguous Images With Human Judgments for Robust Visual Event Classification” were originally somewhat mixed, but converged to mostly positive reviews of the revised text / clarifications. Many of the originally noted issues had more to do with the clarity of presentation than issues with the underlying work and with the final revision the outstanding issues were cleared up.

Strengths:
* Important topic and rarely considered. This work makes a strong contribution in the form of a dataset of ambiguous images along with human uncertainty annotations to quantify the ambiguity.
* Clever method of obtaining ambiguous data from video.
* Clear and rigorous annotation procedures

Weaknesses:
* While this provides a clear step forward in evaluating models with-respect-to ambiguous data, it doesn’t provide broadly applicable solutions (focused on event detection, limited number of classes, and training on ambiguous data presents a trade-off with accuracy on standard data). That being said, this is still a clear step forward in an under-considered area and was a well-thought out contribution.


Contributions:
The paper describes a dataset and evaluations which are original in the following ways: First, it contains images which are ambiguous with respect to the event they depict, while most existing computer vision datasets focus on unambiguous images / data. Secondly, they quantify the ambiguity / uncertainty using human raters and human uncertainty judgements. Finally, they show how standard models perform when trained using standard data and the provided ambiguous data and demonstrate improvements in ambiguous scenarios over standard datasets.

Significance:
Ambiguous scenarios are clearly common and relevant in a variety of real world tasks and it is important to be able to evaluate how existing models do in these scenarios. This also allows for better comparisons between models and humans and provides a method of evaluating the calibration of standard models. That being said, the dataset is still rather small for training and has a limited number classes, is focused on event classification, and the provided results suggest a tradeoff between performance on standard data and ambiguous data. So while this work does provide a clear step forward for evaluating / diagnosing how standard models behave with ambiguous images / calibration, it doesn’t provide broadly generalizable solutions.

Quality: 
The data collection is well thought out (using video data for overall clear scenarios, but ambiguous images is a clever idea) and well motivated. The model experiments provide a clear motivation for collecting additional ambiguous data over standard data augmentation techniques and again highlights the importance of calibrating ML models.

Clarity:
The initial paper was challenging to follow, but the reorganization and clarifications provided in the revision were a significant improvement. It’s now much easier to follow contributions and experiments.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept