Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper provides a novel dataset and its code to increase the number of samples. The dataset is named Gamified Association benchmark to challenge Vision-and-Language models (WinoGAViL), inspired by Winograd Schema Challenge [9]. The dataset is collected via an online game resembling the well-known Codename game. In the game, given image candidates, a spymaster associates a cue word for a subset of them. Spymaster's goal is to come up with a set of images and their cue word that AI cannot guess and that humans can guess correctly.
strengths: - This benchmark is interesting since it highlights the gap of ability for commonsense reasoning between humans and AIs.
- The authors provide a dataset and the online game, allowing researchers to collect further data.
- Experimental results show that the benchmark is challenging for AIs while humans can solve.
weaknesses: - The game, though inspired by Codename, is a different one. The reviewer is a little concerned that the incentive for spymasters to look for a unified representation for more images may not appear. For example, if the spymaster comes up with a word for two images and the AI misses them all, the Jaccard index is, of course, 0%. If the spymaster thinks of a word for three images and the AI misses them all, the Jaccard index is still 0%. In the original Codename, the number of cards guessed by other than the spymaster is directly scored, so the spymaster thinks of abstract expressions suitable for more cards. However, in the game in this paper, no such incentive is created through the Jaccard index. In order to address human commonsense reasoning, incentives to assign words to more images would have been necessary.
- Although the title includes Vision-and-Language, it is slightly overclaimed since the associated representation seems to be a single word. (If the authors mean that the dataset includes not only words but also sentences as a cue, the reviewer would like them to point this out.) Cues did not have to be limited to words only, as in Codename. In the sense of association and commonsense reasoning, spymasters may generate a more flexible representation for a subset of candidates. Some relations between cue length and performance of AIs and humans would also be interesting.
correctness: The reviewer thinks this paper has sufficient correctness.

clarity: There are reasonable descriptions of how the data set was collected and processed and experimental methods.
relation to prior work: Adequate citations and comparisons to related works have been made.

additional feedback: When introducing technical words such as the Turing test and Jaccard index, the authors should avoid referring to Wikipedia since correctness is not guaranteed.
review_title: Interesting trial for better commonsense and association abilities

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper introduces a new vision-and-language (V&L) benchmark based on gamification. The gamification framework is very similar to Codenames (a board game), and in this benchmark the aim is to associate _k_ images from a set of candidates for given input concept. First, the _spymaster_ selects k images from a set of candidates, and the solvers try to predict selected images. To do so, a game is designed for AMTs, and a dataset is collected from the logs of this game. Finally, a couple of known V&L models are evaluated on this dataset.
strengths: The idea is quite novel and interesting. I am certain that this benchmark will be a beneficial resource for future researchers. NLP community is interested in problems that require commonsense reasoning and/or external knowledge recently, and this benchmark falls into this category. The problem is easy for humans and challenging for machines (which should be the case for all created datasets). The paper includes many different categorical analyses (e.g. Table 2, Figure 4). Overall, the paper is well-written.
weaknesses: I can't find any major weakness, but I think the paper has minor issues which can be improved (see the Correctness section).
correctness: I think there are several minor issues,
- For the observation of _Models struggle with associations that are not visually salient_: Is there any further analysis on this result? 60% model performance on visually salient examples is still low.
- For the observation of _Training is effective when the task is difficult_: How is the 10 & 12 task more difficult than 5 & 6? Since the humans achieve similar performance on both tasks, could we really say this? This statement contradicts with the _easy for humans, challenging for models_ claim. The models might be benefitting more from negative associations in finetuning (just a possibility), and those tasks contain more negative examples (in terms of percentage). Do you have an analysis of which examples / cases are improved with finetuning?
- For the observation of _Solving data collected with WinoGAVIL with textual models is not beneficial_: This makes me confused because actually (pseudo) text-only baselines perform similar to vision-language baselines.
- Could you explain why does CLIP-ViL perform worse with comparison to the rest of the models? (especially on the 10 & 12 task)
- A suggestion: could you also perform categorical evaluation on the skills and observed patterns (Table 2, great analysis by the way) required to solve the task? In this way, people can better compare the models.
clarity: The paper is very well-written. Easy to follow from the beginning to the end.
relation to prior work: I think the related work section is adequate.
additional feedback: N/A.
review_title: A challenging V&L benchmark inspired by Codenames board game

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes an online game WinoGAViL to collect vision and language associations, (e.g., werewolves to a full moon), used as a dynamic benchmark to evaluate state-of-the-art models. Specifically, the paper uses the game to collect 3.5K instances, finding that they are  intuitive for humans (>90% Jaccard index) but challenging for state-of-the-art AI models, where the best model (ViLT) achieves a score of 52%, succeeding mostly where the cue is visually salient, which is interesting and insightful.  Moreover, those experimental results in this paper also indicate that the collected associations require diverse reasoning skills,  including general knowledge, common sense, abstraction, and more. Overall, this paper is well-motivated and well-written. I believe it can promote the development of visual-language models.
strengths: 1. This paper proposes WinoGAViL, an online game to collect vision and language associations, used as a dynamic benchmark to evaluate state-of-the-art models. To me, the proposed approach is novel and has a great contribution.

2. This paper is well-written and well-motivated and has insightful discussions.

3. This paper also releases the dataset, the code and the interactive game, aiming to allow future data collection that can be used to develop models with better association abilities.
weaknesses: I do not find any weaknesses in this paper.
correctness: I do not find any incorrectness in this paper.
clarity: This paper is well-written and easy to follow.
relation to prior work: Yes.
additional feedback: No.
review_title: A very interesting paper for evaluating vision-and-language models

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors introduce WinoGAViL, a game which acts as a framework to collect language-vision associations that are difficult for existing vision-language models to solve but solvable by humans. The authors design WinoGAViL in the style of Codenames, where the "spymaster" is provided a set of images, and has to come up with a language cue that is related to a certain subset of the images. The authors initially collect a set of 3.5K association instances, but more instances can be flexibly added to this. The authors analyze the collected associations and the challenges involved in solving them, and analyze how existing vision-language models perform on this dataset.

This paper has one of the most novel and intriguing ideas I have seen in vision-language data collection, but after reading the paper and playing around with the demo myself, I have a few doubts that make me skeptical of this work until they are addressed. The strengths of this paper are very strong ones, but unfortunately, the weaknesses in certain methods are also glaring. If the questions listed are adequately answered, I would be willing to change to Clear Accept.
strengths: Strengths:
1. The idea is a very clever one, and the codenames-style data collection game is well formulated to collect difficult vision-language associations. 
2. The framework and game are made public, so that people can continually add associations.
3. The evaluation of the collected associations is well done, and the results are convincing that these associations are harder to solve for existing VL models but highly solvable by humans (but still less than the SWOW Baseline)
4. The paper is very well written and structured. It was honestly a pleasure to read.

weaknesses: 1. I find it odd that the authors collect only one image per concept from Google Images (L100). Collecting only one image per concept results in cues for concepts that are too specific to the single image collected for it. This means that the cues. corresponding to the various associations that a concept occurs in, largely contain information about the specific visual context of the concept's image, and so the concept's evaluation overfits to the visual properties of that image. While collecting a larger set of images (say, 5) per concept would also result in cues that are specific to the images that in their instances, since the same image is not used for each instance the overall evaluation for that concept's associations does not overfit to a single image. 

2. The various visual skills in Table 2 only make it more confusing exactly what we are evaluating with the collected associations. Do snowmen stare? Is this a general association that we want the model to know? It seems like the associations being collected are too image-specific (related to point 1 above). Even if we only want to evaluate the associations that the model can express conditioned on a visual context (i.e. not a general concept-level association like "snowmen are white" but an image-specific one like snowmen staring in that particular image), the overall evaluation still only evaluates associations specific to that one image of the snowmen - how do I know if the model will express snowman-related associations for a different image of the snowman?

3. Further, it's not clear what visual associations for all concepts derived from SWOW will look like. For instance, what do the images of "green" and "dirty" (concepts from the first row of Table 2) look like? How does that affect the particular associations that we collect for them?

4. The choice to not discard associations that are solvable by the AI (L148) is confusing - the whole point of the framework is to collect associations that are difficult for models but solvable by humans, so why keep samples that do not meet the first criteria? The authors could average model performances from a few different AI models if they are concerned about model dependence of their collected data.

5. It's not clear in Section 3.1 how the SWOW Baseline dataset is different from the WinoGAVIL one - specifically, what are properties of how the Baseline is collected that WinoGAVIL does better? (the explanation in L253-258 would go well in S3.1, because it's not clear how these datasets differ till we compare the results on them) Some visual examples that compare associations from these two datasets would be nice.

6. In L274-276, when filtering visually salient cues, I don't understand why the authors do not retain only those cases where both the annotators agree - 88% is still a sizable number, and gives us more confidence about the results on this subset — in the current setup, the results appear to be muddled by those examples where annotators disagreed on the visual salience of the cue.
correctness: I have questions about certain design decisions when creating the benchmark, which I have outlined above.
clarity: The paper is very well written, and easy to follow.
relation to prior work: Yes
additional feedback: -
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposed a new benchmark called WinoGAViL to evaluate the commonsense reasoning ability of the state-of-the-art vision-and-language models. In WinoGAViL, each task is composed of a textual cue, a number $k$ and a set of candidate images, and its goal is to select the $k$ images most associated with the textual cue. To collect novel and challenging associations, this work proposed a web gamification framework where a spymaster composes a new association cue that can be easily solved by other human players but can easily fool the AI model (such as CLIP-RN50). Experiments showed that current vision-and-language largely fall behind human performance.

strengths: (1) The WinoGAViL is an interesting benchmark for multimodal commonsense reasoning, and it can be dynamically updated with the proposed gamification framework. 

(2) The experiments well demonstrated the gap between human and machine performances, implying the challenges of the designed tasks for current AI methods. 

(3) The analysis of the tasks in the WinoGAViL dataset reveals that a diverse set of reasoning skills are required to achieve high performance. 

(4) An alternative approach to collecting the data using the SWOW dataset is presented in this work to show that the gamification framework creates more challenging associations. 

weaknesses: (1) It is unclear why not use the model scores in the data selection. My intuition is that removing data that can be easily solved by the AI model (i.e., the CLIP-RN50) will make the benchmark more challenging to current vision-and-language models. Is that true? If not, I wonder how to explain this phenomenon and whether there is any experiment to justify it.

(2) In experiments, it would be more interesting to show how various methods behave differently across tasks that require different reasoning skills. It is an extension of the existing experiments of comparing “non-visual” and “visual” settings to more fine-grained observed patterns. 

(3) What are the chance performances in Table 4? I am curious how the methods compare with the random guess.

correctness: The benchmark is constructed in a sound way and the evaluation methods and experiment design well support the claims.
clarity: Yes
relation to prior work: Yes
additional feedback: See the above comments
review_title: An interesting benchmark for testing the commonsense reasoning of vision-and-language models

Meta-review: metareview: The paper presents a new vision-and-language benchmark that associates images and text, collected using a crowdsourced game similar to Codemaster. All reviewers were positive on the paper. They praised the cleverness and novelty of the idea, the clarity of the writing, that the benchmark and game are publicly available, and the quality of the experimental evaluation. Reviewer o1Pk initially raised some issues, but was largely convinced by the authors’ response. Overall the AC agrees that the paper is interesting and well-written, and will be a welcome addition to the NeurIPS Datasets and Benchmarks Track this year.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept