Measuring CLEVRness: Black-box Testing of Visual Reasoning Models
Authors: Spyridon Mouselinos
Henryk Michalewski
Mateusz Malinowski
Venue: ICLR 2022 Poster (2022)
Keywords: Visual Reasoning
Visual Question Answering
Black Box Testing
Computer Vision

Abstract: How can we measure the reasoning capabilities of intelligence systems? Visual question answering provides a convenient framework for testing the model's abilities by interrogating the model through questions about the scene. However, despite scores of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architectures can actually reason remains open to debate.
To answer this, we extend the visual question answering framework and propose the following behavioral test in the form of a two-player game. We consider black-box neural models of CLEVR. These models are trained on a diagnostic dataset benchmarking reasoning. Next, we train an adversarial player that re-configures the scene to fool the CLEVR model. We show that CLEVR models, which otherwise could perform at a ``human-level'', can easily be fooled by our agent. Our results 
put in doubt whether data-driven approaches can do reasoning without exploiting the numerous biases that are often present in those datasets. Finally, we also propose a controlled experiment measuring the efficiency of such models to learn and perform reasoning.

Published as a conference paper at ICLR 2022
MEASURING CLEVRNESS: BLACK-BOX TESTING OF VISUAL REASONING MODELS
Spyridon Mouselinos
Henryk Michalewski University of Warsaw, Google Oxford, U.K. henrykme gwwgle.oo..ooooooo com com
University of Warsaw
Warsaw, Poland S. w.mouselinos@uus edu.pl
Mateusz Malinowski
DeepMind London, U.K. ma mateuszme deepmi nd. com
ABSTRACT
How can we measure the reasoning capabilities of intelligence systems? Visual question answering provides convenient framework for testing the model's abil- ities by interrogating the model through questions about the scene. However, de- spite scores of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architec- tures can actually reason remains open to debate. To answer this, we extend the visual question answering framework and propose the following behavioral test in the form of a two-player game. We consider black- -ox neural models of CLEVR. These models are trained on diagnostic dataset benchmarking reasoning. Next, we train an adversarial player that re-configures the scene to fool the CLEVR model. We show that CLEVR models, which otherwise could perform man level", can casily fooled by our agent. Our results put in doubt whether data-driven approaches can do reasoning without exploiting the numerous biases that are often present in those datasets. Finally, we also propose a controlled ex- periment measuring the efficiency of such models to learn and perform reasoning.
"hu-
INTRODUCTION
Are our artificial intelligence systems capable of reasoning? Or like Clever Hans, they use var- ious cues only tangentially related to the task and rely on rote memorization with poor general- ization? (Pfungst 1911 Johnson et al. 2017a) This work revisits such question and proposes an interactive framework with the communication channel between two players. The first player, which reasoning capabilities
about test, performs visual reasoning tasks, call it Visual- QA Player. The second player, which we call the Adversarial Player, is manipulating the scene so that it fools the first player even though those changes still lead correct reasoning steps among hu- mans. Both players interact with each other only through questions, answers and the visual scene as shown in Figure D If the Adversarial Player manipulating the scene causes the Visual-Q2 QQ Player to change its answer even though the new scene is still valid for the same question and answer, it is then the reasoning failure. It is similar the following situation. Imagine a box placed between two spheres. If you ask a question, is there a box between two spheres?, the answer should be positive. Now, if we move the box anywhere so it does not cross any of the spheres, and ask the same ques- tion, the response should remain unchanged. In other words, we postulate that reasoning outputs of agents need to be invariant under scene configurations that are consistent with the questions-ant -nnswer pairs. Moreover, in the spirit of generic adversarial attacks, seek configurations that also pose little if any reasoning challenges for humans.
We propose automatic and agnostic pipeline to benchmark the reasoning capabilities of various models, only assuming they can communicate by answering questions about the scene. Due to the recent stream of research in vision-and- languuuge (Zhang et al. 2021 J02ngJiang al. 2020 Guo al.

---

Published as a conference paper at ICLR 2022
Chen et al., 2020). we believe there will be an increasing number of vision models that operate through language. Moreover, also consider the visual question answering framework set-up two-player system as an excellent benchmarking pipeline. We perform all tests by scene manipulations and observing how tested model behaves under such changes. The pipeline does not require any knowledge of the internals of the tested model. It also does not manipulate the sensory information of such a model, e.g., pixels in the images, and all the manipulations are physically meaningful. Even though our current pipeline uses synthetic scenes as only those can easily be automatically manipulated, our results have also weal-world ramifications. change [T in scene configurations, in synthetic setting, there are valid concerns that real-world robots could also be prone to manipulation of objects in a room. Finally, our work also questions the possibility of training and benchmarking networks in purely data-driven and offline, static manner.
2019b Su et al. 2020 Wang et al. 2020 Kamath et al. 2021 Tan & Bansal 2019
models are susceptible to semantically meaningless
Contributions. The main contributions of our work could be summarized in three points.
First, we propose strong black-box adversarial test, which makes assumptions about the under- lying mechanics of a tested model, formulated as a game between two players. Our test does not require any direct access the tested model, even through its sensory information. In particular, it does not require gradients, output probabilities, or any access to the perceived image. Our work also deviates from bounded perturbations and instead focuses on global scene manipulations that are still consistent with the task constraints, and can change the behavior of tested model.
Second, we reformulate visual reasoning by integrating visual question answering with zero-sum two-player game frameworks. Under our novel formulation, visual and adversary agents compete against each other through content manipulation. We believe that this is an initial step towards more sophisticated frameworks that integrate computer vision with multi-agent systems.
Third, we explore the limits of the data-driven approaches in synthetic visual scenarios, and demon- strate that current CLEVR models are lacking the efficiency to learn robust reasoning steps.
2 RELATED WORK
Our work touches upon various research directions, which we briefly describe here.
Visual QA. Introduced as a visual counter-part of the Turing Test (Malinowski & Fritz 2014 Geman
et al., 2015). it became computer vision task that requires holistic visual understanding. Many other works have extended the task to larger datasets or videos or adversarial train-test splits (An-) al. 2016] Yu al.) 2019). More recently, we also observe the generalization of the task to become a part of the vision-plus-language suite of
tol et al. 2015 Agrawal al.) 2018 Tapaswi
benchmarks (Lu et al.] 2019: Chen et al., 2019 Wang al., (022). Johnson et al. (2017a) have
introduced a synthetic variant of the Visual QA problem that is more focused on reasoning and the control of the experimentation. Although it was shown that traditional methods (Yang et al. 2016) are not enough to solve that dataset, newer methods can pass the human baseline on this task. These methods are trained to operate on pixels and text directly (Santoro et al. 2017): Perez et al.) 2018
Hudson& & Manning 2018 Malinowski et al., 2018 Kamath et al.) [202] 1)
or they induce programs
as an intermediate representation (Johnson et al.; 2017b] Mascharka et al. 2018 Yi et al. 2018)
Hudson & Manning (2019) have shown that Visual QA models lack some reasoning robustness but their approach is mostly linguistic and still static. Some other works also observe issues with static evaluation and proposed more dynamic benchmarks, c.g., with human-in- -ie---iittttooooo loop (Khashabi et al. 2021 Nie et al. 2020 Li et al. 202T Sheng et al. 2021). We extend the line of research on Visual QA by reformulatinn aatin it as interactive two-agents system, and show that CLEVR under such new setting is still an unsolved problem. Our work also differs from previous works in that it focuses on visual reasoning, is interactive and fully automatic.
Adversarial perturbations. SSzeeddy et al. 2): Goodffllow et al. (2015) Kurakin et al. (016));
Carlini & Wagner (2016) have introduced early methods that can 'fool' networks by performing tiny, visually imperceptible, perturbations of the input image. MMoosavi-Dezfoli et al. (2017) have also shown universal and untargeted adversarial perturbations that are more transferable than previous approaches. Most methods perform white-box attacks, where an adversarial model has access to the target model's parameters or gradients. In black-box attacks such access is removed and, c.g., only output probabilities are available (Guo et al. 2019a). Other black-box attacks involve the use
Changes that are consistent with the task constraints.
2

---

Published as a conference paper at ICLR 2022
of surrogates to estimate gradients of the real target (Papernot et al. 2017: Cheng et al. 2019) or gradient- methods (Alzantot et al. 2018). More related to our work. Cheng et al. (201 8 describe black-box system, which only assumes access to outputs of the network. However, all the pertur- bations described above are continuous-
where individual pixels are changed independently. This leads to improbable images and assumes access to the sensory inputs of the neural network. In contrary, semantic perturbations (Joshi et al., 2019; Zeng et al., 2019), operate on semantically meaningful chunks of the input, leading to more plausible and physically interpretable perturbations. However, they still assume direct access to either images or ideally differentiable renderers. We can interpret our work within the adversarial perturbations framework, where the adversarial model can change the original scene by its manipulations, with zero access to the target model, including its sensory information and not limited to tiny changes but instead to semantically and globally mean- ingless modifications.
Reinforcement learning. The game between the Visual-QA Player and Adversarial Player is a zero-sum two-player game. Conneau al. (2017) use conceptually similar two-player game to find a better word alignment between two languages without parallel corpus. Inspired by such a problem formulation, We optimize the Adversarial Player with Advantage Actor-Critic (A2C) algorithm (Sutton & Barto 2018 Degris et al.) 2012).
consider our two-player game tool to achieve better reasoning models.
Probing and measuring intelligence. Turing Test (Turing 2009) is considered to be among the ear- liest works on measuring the intelligence of artificial systems. Crucially to us, is also formulated as a two-player game with an interrogation protocol. Legg (008) systematizes and relates the no- tion of intelligence from different fields with the main focus on 'universality' PJhnnson et al. (2017a) have transferred the notion of intelligence onto the visual ground and emphasized reasoning rather than universality. seems there is no widely accepted dataset or definition that encompasses our intuitions about intelligence. However, there are few recent directions showing the lack thereof, mainly highlighting bias amplification or absence of mathematical capabilities (Hendricks et al., Piekos et al. 2021
2018 Bhardwaj et al. 2021
3 PRELIMINARIES
In this section, we explain briefly how the CLEVR dataset (Johnson et al. 2017a) is constructed and introduce notation and definitions.
CLEVR is a synthetic visual question answering dataset introduced by Johnson et al. (201a)) which consists of about 700k training and 150k validation image-qusstion-aswwe triplets. Images are tificially constructed and rendered from scene graphs - a special structure containing information about object attributes such position or color. Such scene graph also used synthesize the trouh question-anse answer pairs by expanding templates according to the ordering. Ambiguous scenes are rejected. Each image represents an isometric view of the scene containing from two to ten objects. There are three classes of objects, spheres, cubes and cylinders. Each object can also be either large or small and has one color out four (brown, purple, cyan, yellow). It can also be either metallic or rubber-made. Every object has and y coordinates that are confined within the (-3,+3 +3) range. We use the same generation process to render modified scenes. CLEVR models. Various models have been introduced to work with the CLEVR dataset, some even 'solving' the dataset by achieving near perfect performance. Despite the strong offline performance, test if those models' performance perpetuates in the more interactive setting where configu- rations of the scene could be changed. Whenever possible, we use pre-trained CLEVR models. Otherwise, train the remaining models from scratch by making sure we achieve results similar to published accuracy numbers on the validation set. We summarize all the models in Table [ We show the accuracy on the CLEVR dataset (Accuracy), indicate if an architecture is trained from scratch (Re-trained), briefly describe how multi-modal fusion and reasoning is conducted (Reason- ing Mechanism), and indicate any extra privileged information required during the training process (Extra). For instance, some models require extra access to functional programs used during the dataset generation, use scene graphs as a supervisory signal (states), or always operate on scene graphs (input-states). Otherwise, the models were trained only from image-question-answer triples. Mini-games. We formulate our problem as a Game between two players, Visual-QA QA Player and Adversarial Player. The Visual-QA Player takes input question-imanl immag pairs and provide swers to such questions. Some models use states (scene-gradl (scene-graphs) graphs) that replace images or require programs JJhnssn et al. 2017a). The whole game consists of all CLEVR data points. For our
3

---

Published as a conference paper at ICLR 2022
purpose, extend the notion of the Game into Mini- fanii. The rules of Kini-games identical to the whole Game. The only difference is that each Mini-game operates on a subset of the CLEVR dataset. We define the size of a ..MMiiggam game. We sample data points for each Mini-game randomly and mutually exclusively. Mini-games have analogies in the adversarial perturbations literature. Mini-gamen of size one resemble per- image adversarial perturbations (Goodfellow et al. 20155 Deeosaiii et al. 2016) whereas Mini-game that has all data pointsis similar to universal adversarial perturbations (Moosavi-Dezfooll et al., 2017). In this work, we investigate various Mini-game sizes but due to the sheer scale we were unable to use the whole game as the Mini-game. Larger Mini-games make the optimization process more difficult as the domain where the Adversarial Player needs to operate increases. The training also much more time-consuming and a sequential process. Instead, we can train multiple players on different Mini-games, independently and thus massively. We leave the arduous training of the universal Adversarial Player on the whole Game as a possible future direction.
by the number of datapoints that attached to that Mini-
Table I: CLEVR models that we use as Visual-QAa QA Players.
Model Name
Accuracy
Re-trained
Reusoning Mechanism
Extra Data
SAN
Yang et al. 2016
72.1
Attention
FiLM
(eerez et al 2018
96,2
Feature Conditioning Relational
RN
Santoro et al. 2017
93.2
IEP
Johnson et al.
2017b
96,9
Neural Program Induction
Progrums
TbD
Mascharka el al
201
99.1
Neural Program Induction
Programs
Mdeii
Kamath el a 2021
99.7
Multimodal Transformer Querying
States
State-Inpul Transtormer (ours)
96.8
Cross Attention
Input-States
4 ENVIRONMENT
We need to ensure that Adversarial Players create valid scenes that are consistent and in-distribution. Both properties are guaranteed by our environment enforcers.
Consistency. Since scene manipulation may change the answer for a given question, we need to ensure this does not happen. That is, the new scene still consistent with the question-answer pair. The guesticn enforcer achieves that by running functional programs associated with each question (Johnson et al., 2017a) on the modified scene-graph. Hence, it gets the new ground-truth answer. The enforcer rejects the new scene if that new answer differs from the previous one. In this way, it guarantees the newly generated scenes give the same answers as the original scenes on the same question. Thus, we can generate equivalent scenes containing the same objects that have identical answers for the same questions. Using that enforcer, we can test if the Visual-QU QA Player's answers invariant under such an equivalent class of scenes.
In-distributiin. Even with the question-relevance enforcer, the Adversarial Player may still pro- duce undesired outputs. For instance, it can stretch the whole scene thus violating the scene bound- aries from the original CLEVR dataset, making, e.g., everything to look very small (Section [A.9 in the appendix). Although
still an interesting form of adversarial scene manipulation, we focus rather on the in-distribution scene manipulations that respect the original boundaries. To enforce that property, we use scene-constraint enforcer that checks the boundaries of the scene. Without that enforcer, the Adversarial Player would quickly resort to stretching the whole scenes, achieving a form of adversarial attack that uses distribution shifts rather than content manipulation. does so, e.g., by moving the camera away until objects are barely visible. We give few such examples in the appendix (Section A.9..
5 Adversarial Player
Meaningful scene manipulations require not only generic scene understanding, but also the ability to distinguish which objects to displace and how. Hence, the player is composition of a multi-modal module, which creates input representation, and decision maker, which decides how to control the scene. Figure| illlustrates the Adversarial Player and the game between both players.
Multi-modal module. We have experimented with the same multi-modal modules as those in Ta-
ble i but found out we have better performance and the convergence rate if the Adversarial Player
4

---

Published as a conference paper at ICLR 2022
operates the scene-grapha, (states) instead of pixels. For that, we state-input variant of Re- lation Networks (Santoro et al. 2017). The model receives as the input 10 * 6 object tokens, and question tokens. Every object token represents one-out-of-ten possible objects in the scene by its attributes such as position, color, shape, material and size. If the scene has fewer than ten objects, we use 0 token to indicate that, which also acts padding. We also have special tokens that separate questions from the objects which we add as a latent embedding, c.g., emb material) + emb(object). Such an input encoding similar our State-Input Transformer and described in Section A..3 The embedded vectors are given to the Relation Network (RN). Finally, we train that network on the CLEVR visual question answering task, where we achieve 97.6% on the validation set, and use the representation just after the last relational layer for the decision maker.
Decision maker. Inspired by the work on reinforcement learning (Mnih et al. 2016 Wu et al., 2017
Lillicrp et al. 2016 Wang et al., 2016) SSchulman et al., 2015). we use an actor-critic module that acts on scenes. The actor is a general
general-purpose purpose fully connected layer with ten object-specific heads. Each head is randomly assigned to a unique object in the scene for its manipulation. Every head pro- duces a displacement in æ and coordinates of the corresponding object. Although we have initially experimented with the continuous output space, we have found out the following simple strategy is more effective. First, we discretize all the I and coordinates into N bins each. Now, each head produces two N-dimensional vectors that are next projected into a probabilistic space via softmax. Next, we sample displacements in and axis independently from both softmax distributions. Note that, even though we do not model the joint distribution explicitly due to computational reasons, both samples condition on the common head and thus are only conditionally independent of each other. We discretize the scene where each axis has values in [-3,3) 3] onto N bins per axis. Our critic is a simple three layer feed-forwar network (with relu as activations) that predicts a reward score between -1 and +1 via tanh activation (1.2 x tanh for better numerical properties).
The game of scene manipulations. Due to our formulation of Adversarial Player and the envi- ronment, we can benchmark various reasoning models purely in the black-box setting via a series of questions about the scene. Adversarial Player manipulates the scene so that it is still consistent with the question-answe answerr pair. The manipulations are applied to evene-graphs, and the resulting scene-graph is evaluated by the environment enforcers described in Section Invalid scenes thus discarded. In this way, we ensure the in-distribution and consistency in the scene generation. Original image-question pairs are fed to a Visual- QA Player that produces corresponding answers. We refer to that answers as old answers. After the scene manipulation, new images paired with the same questions are also given to the Visual-QA Player that produces new answers. We construct rewards based on old answers, new answers and ground-trute answers.
If the Adversarial Player forces the Visual-QA -A Player to change the answer, i.c. an old answer is different than new answer, it gets Consistency Drop Reward (cr). If an old answer is the ground- truth answer, it gets instead Accuracy Drop Reward (dr). Both rewards differentiate between simply confusing the model and causing drop in its performance. If Adversarial Player produces an
Original Scarro
mappet ad Head ad
Maripuutttion Heore
Actor
Manipxiated Soone
jectect
Scane and Question
Valid:
FFN
Malttaboial Mkodille
Object
Enforser
Rendering
Engine
Quection: How
FFN
Critic
Adversarial Player
A2C
many brown items there? Answer:
Quastion: HaW
Visual OA Player
are tiare?
Answwr: 2
Heward
Calculation
Figure 1: Our game between two players: Adversarial Player and Visual-QA Player. Adversarial Player uses multi-modal module to extract features conditioned on the visual and textual inputs. After transforming such features with feed-forward architecture, it samples corresponds to manipulating the corresponding object the scene. the case of missing objects. we use an @ token. After alternating the original scene graph, we use various environment enforcers to ensure validity of the constructed scene. A valid scene graph is rendered and introduced to the Visual- QA Player together with the original image. Finally, collect responses of the Visual- QA Player and calculate suitable rewards based on them, and we repeat the whole cycle during the training phase.
action using object-specific heads. Each action
5

---

Published as a conference paper at ICLR 2022
invalid scene it gets Invalid Scene Reward (isr). This reward encourages producing scenes that pass the environment enforcers tests. Finally, if Adversarial Player does not manage to fool the model, it gets Fail Reward (fr). We use the following values: dr= 1, cr= 0r1, -0.1, isr -0.8. Training algorithm. To train Adversarial Player we use the A2C algorithm with the episode length set to as we do not need model long-range consequences of the decision-making mechanism. Batches contain images, question, answers, programs and scene-graphs. We train the Adversarial Player for each Mini-game independently using the same architecture. We experiment with the following Mini-game game sizes 10, 100, 1000. All Mini-game are constructed randomly. Under our discretization scheme, the action space is N where N is the number of bins and k the number of objects in the scene. In practice, it is up to 49
6 EXPERIMENTS
We consider Adversarial Player with or directly from pixels (pixel-input). We train Adversarial Player for each Mini-gume independently. We use different Mini-game sizes in our experiments: 10, 100 and 1000. To obtain statistically sig- nificant results, we run each Mfni-game thirty times with randomly initialized Adversarial Players, each with different seed per trial. We report the results that are averaged over all those runs. We compute two metrics for each Mini-game, Consistency refers to the fraction of times Adversarial Player has changed the Visual-QA Player's 's answer, regardless that answer was correct. Drop refers to the fraction of times Adversarial Player has changed the correct Visual-QA Player's an- swer. We aggregate the results using two statistics. Average Accuracy averages accuracies over all Mini-games. Maximal Accuracy refers the worst (best) case of a single Mini-game from the Visual- Matal-QA Player's (Adversarial Player's) perspective. We also computed p-values of T-Test, with the null hypothesis that a manipulation is unsuccessful. In almost all cases we reject the null hy- pothesis with small p-values. The p-values are available in the appendix (Section A. Quantitative results. We show our results in Table 2 for the state-input and pixel-input Adver- sarial Players, and for each Visual-QA Player presented in Table D All the Visual-QA Players in the comparison are trained from pixels, apart form our custom State-Input Transformer architecture (Section A.3 in the appendix). When comparing both Adversarial Players, we can observe that the state-inpan Adversarial Player is significantly more successful than the pixel-input one. We hypoth- esize that this is the effect of having more structured and less ambiguous input information. It could also be the case that multi-modal modules trained from pixels ignore features in the same way and hence smaller discrepancy between all models operating games increases, it is becoming more difficult for the Adversarial Player to successfully manipulate the model on all examples from the Mini-game
module pre-trained either on states (state-input)
pixels. Moreover,
the size of Mini-
This is an expected behavior resembling standard adversarial perturbations [Goodfellow et al. 2015) and their universal variants (Moosavi-Dezfooll al., 2017). The highest performing CLEVR models, TbD and MDetr, are the most robust under the manipulation. However, they are still susceptible to the scene manipulations in some Mini games as the Maximal Consistency metric indicates, especially if our Adversarial Player uses states as the input. Section A.8 (appendix) also shows richer performance statistics. Our detailed investigation shows that the models above are particularly sensitive to manipulations of scenes associated with counting or existence questions. Such question are often more complex, hence, increasing the like- lihood of the reasoning failure at some stage (Section A.6 in the appendix).
Qualitative results. Figure 3 qualitative results of the scene manipulations for the three high- est performing and distinctive models from Table e.g., simple feed-forward or program induction models. For the reader's convenience, we provide in Section A.2 (appendix) examples of CLEVR object attributes. We can observe that the scene manipulations are surprisingly minimal and are semantically meaningless, i.e.. they should not 'fool' human players.
Reasoning steps and visual cues. To better understand the source of errors (reasoning steps vs visual cues), we have conducted the following experiments. In the first experiment, shown in Sec- tion B.4 (appendix,, we visualize the attention mechanisms of Visual-QA Players before and after the scene manipulations. We observe changes in the attention map of all models but MDetr. The results suggest, most models's 's perception is impacted by the manipulations. However, all objects are still correctly identified by MDetr, thus suggesting some issues are also stemming from the reason- ing steps. In the second experiment, we deployed Slot-attention object detector (Locatello) 2020), and show that the detector identifies the same properties before and after manipulations. This re- sult (Section B.6, appendix) shows that Adversarial Player does not produce 'comne cases' of the

---

Published as a conference paper at ICLR 2022
Table 2: Results of the games: Model dataset performance refers models' accuracy on the CLEVR dataset. Model Mini-game performance refers to the model accuracy on the Mini-game examples where the largest (maximal) performance drop was detected, before manipulations took effect. The Average Accuracy column reports the performance of models averaged over all runs of the respec- tive Mini-game sizes. Maximal Accuracy reports the worst model performance among the respec- tive Mini-gume size runs. We also report in brackets the relative performance drop, in percentages X-Y%.
X %. Average Accuracy is compared against Model dataset performance (X = Model dataset performance/ / Y
tatec/Pixc-IInpuute Maximal Accuracy is compared against Model Mini- Mini-game game performance (X = Model Mini-game performance, Y = State/Pixel-Inppe Note that in the case of state-input and pixel-input Adversarial Player the worst Mini-game might not come from the same Mini-game instance.
Average Accuracy
Maximal Accuracy
Mini-
Model gume size
Model datuset
State-Input
Pixel-Input
Model Mini-game performance
State-Input
Pixel-Input
performance
SAN
(-14.22))
(-11.4%)
80.0/
28.0 (-65.00))
43.0 (-46.2%)
100
72.1
66.1 (-8.3%)
69.2 (-3.9%)
74.0 75.0
47.6 (-35.6%)
66.3 (-11.5%%)
1000
70.2 (-2.5%)
71.0 (-1.4%)
72.3/ 72.3
68.4 (-5.3%%
70.8 (- (-1.9%)
FiLM
10
(-12.7%)
93.6 (-2.6%)
100.0/ 100.0
48.0 (-52.0%)
86.0 (-14.0%)
100
96.2
89.1 (-7.3%)
94.8 (-1.4%)
98.0/ 100.0
75.6 (-22.8%))
92.4 (-7.5%)
1000
93.8 (-2.4%)
95.8 (-0.4%)
96.4/ 96.1
90.8 (-5.7%)
94.7 (-1.4%)
RN
10
80.5 (-13.6%)
86.4 (-7.2%)
100.0/ 100.0
47.0 (-53.0%%
74.0 (-26.0%)
100
93.2
85.5 (-8.2%)
(-2.5%)
94.0/
(-32.66))
87.4 (-7.9%)
1000
90.5 (-2.8%)
91.6 (-1.7%)
/
90.0 (-3.2%%
91.0 (-2.4%)
IEP
10
84.3 (-13.0%)
94.4 (-2.5%%
100.0/ / 100.0
48.0 (-52.00)
87.0 (-13.0%)
100
96.9
90.3 (-6.8%)
95.6 (-1.3%)
98.0/ 96.0
74.4 (-24.0%)
93.5 (-2.6%)
1000
94.1 (-2.8%)
96.6 (-0.3%)
97.1 97.3
93.9 (-3.2%)
95.7 (-1.6%)
TbD
10
94.0 (-5.1%)
99.1 (-0.0%)
100.0/ 100.0
(-31.0%)))
99.0 (-1.0%)
100
99.1
96.6 (-2.5%)
98.7 (-0.4%%
100.0/ 100.0
91.1 (-8.8%)
98.0 (-2.0%)
1000
98.0 (-1.1%)
99.0 (-0.1%)
99.4 99.7
(-3.9%)
98.8 (-0.9%)
Mdetrr
93.7 (-6.0%)
99.7 (-0.0%)
100.0 100.0
60.0 (-40.00)
99.0 (-1.0%)
100
99.7
96.1 (-3.6%)
98.6 (-1.1%)
100.0/ 100.0
86.4 (-13,5%)
91.8 (-2.2%)
1000
98.5 (-1.2%)
99.4 (-0.3%
99.5 100.0
94.5 (-5.0%)
99.1 (-0.9%)
State
89.3 (-7.7%)
96.0 (-0.8%%
100.0/
(-23.-2)
91.0 (-9.0%)
Input
100
96.8
94.7 (-2.1%)
96.1 (-0.7%%
97.0/ 99.0
92,6 (-4,5%)
95.0 (-3.9%)
Transf:
1000
95.7 (-1.1%)
96.5 (-0.3%)
97.2 96.4
95.1 (-2.1%)
95.8 (-0.5%)
perceptual system. We also have similar conclusions based on our small-scale human experiment (Section B.5 appendix).
Adversarial training. We have adapted a widely used adversarial training schema (Goodfellow et al. 2018), 8), and included manipulated scenes in the training protocol of Vixual-QA Player. We use pixel-input RN (Santoro et al., 2017) as the Visual-QA Player. Using this method, we have obtained only marginal improvements in robustness at the cost of slight performance degradation on the original CLEVR dataset.
Limitations of data-driven reasoning models. Due to the data-driven nature of our Visual- QAA Players, we pose the following question, How many examples of Adversarial Player manipulations would be enough for visual reasoning model to train on, in order for it be robust against any unseen ones? Intuitively, if the network has enough capacity, and was trained on all possible data points (or its manipulations), it could rely on a look-up strategy solve the problem accurately. We propose the following experiment to address such a question. We created series of datasets as follows. We treat a scene as discrete 7 x 7 grid (manipulations are restricted only to this grid). Thus each dataset contains 492 = 2401, 493 = 117649, 49 = 5764801 scene combinations, cov- ering all possible scene manipulations with two, three, and four objects respectively. In the case of four objects, due to computational reasons, we keep single object stationary (thus it also has 493 scene manipulations). The set of questions associated with each image dataset requires the model to perform either one reasoning step (Onehop), two reasoning steps (Twohop) or a mixture of both (Mixhop). We use RN, FiLM and TbD as Visual-QA. QA Players. We train them on X% data and next evaluate (100- X)% remaining, unseen data. We have conducted ten trials by forming X%
7

---

Published as a conference paper at ICLR 2022
training data randomly. We notice that the behavior of all tested models is similar report joint average. Results are presented in Figure 2 As the number objects and diversity of question increases, does the amount of training examples needed in order to achieve robustness.
Complexity vs Performance
100
Test accuracy (100-X9; catoseti
90
70
Orehop 1 Question -2 Objects
Orehop
Question 33 Oojects
Twchop
Question 3 Oojects
50
Midhop
Questions 3 Objects
Mixhop
Questions 3 Objects
Mixhop
Questions 3 Objects
Muxhop 10 Questions Objects
Mixhop 10 Questions Objects
300
10
15
30
35
Training percantage (X% of dataset)
Figure 2: Data-driven limitation experiment: Unseen perturbation accuracy depends heavily the number and complexity of introduced questions, followed by the number of objects in the dataset examples/scenes. A model could be considered robust when achieving close to 100% test accuracy.
CONCLUSIONS
We present a novel approach to isolate and benchmark the reasoning capabilities of visual models by formulating a game between two players. Both players are independent and can only communicate through a common interface, an environment and question-answe sarial Player, is trying to manipulate the environment so that the second player, named Visual-QA Player, can't answer the given question anymore. We show that popular models with solid results on CLEVR are susceptible to scene manipulations, and their performance may degrade significantly. Moreover, we also conducted controlled study on the generalization capabilities of such models. We did by creating synthetic and simplified datasets of all possible scene manipulations. We show that the existing models are susceptible to reasoning gaps and require extra data, proportional their task complexity in order to generalize all possible configurations. One possible direction to increase efficiency could be the use of stronger inductive biases. Another to change the training interactive one. However, the environment synthetic, and could be that learning from real-world and multi-modal data is somewhat more efficient. Finally, we believe that melding two-player games with the visual question answering framework is both natural and bene- ficial for testing the reasoning capabilities and potentially can also be generalized to other settings.
pairs. One player, named Adver-
paradigm into
ACKNOWLEDGMENTS
All experiments were performed using the Entropy cluster funded by NVIDIA, Intel, the Polish National Science Center grant TMM0-201726////S5//00622 and ERC Starting Grant TOTAL. The work of Spyridon Mouselinos and Henryk Michalewski was supported by the Polish National Sci- ence Center grant UMO-2018/29///S56000555
8

---

Published as a conference paper at ICLR 2022
What the color of the block that left of the big blue matte ball?
How many other ohjects the same shupe the big gray matte object?
How many other things the same shape the big metallic thing?
FILM Before
Purple
FILM After
Brown
2 there big object of the shape the green thing?
What number tiny matte cylinders there?
Are there any other things that have the material the big yellow ball?
ThD Before
Yes
Yes
ThD After
No
No
The red thing is the material as the brown cube is what size?
there anything clse that has the material the large green cube?
What number of things large metallic blocks the left side of the brown metallic thing big shiny objects?
MDelr Before
Large
No
2
MDett After
Small
Yes
Figure 3: Manipulations of CLEVR models. We show results before and after scene manipulations.

---

Published as a conference paper at ICLR 2022
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971-4980, 2018.
Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, and Mani B. Srivastava. Genattack: Prac- tical black-box attacks with gradient- free optimization. CoRR, abs/1805.11090, 2018. URL
http://arxiv http://xww....aa////////a///
11090
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2015.
Rishabh Bhardwaj, Navonil Majumder, and Soujanya Poria. Investigating gender bias in bert. Cog- nitive Computation, pp. 1-11, 2021.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. CoRR, abs/1 ht//1608.0P00 046444444, 2016. URL BRtthtt::/aaxiie wwtp:///rggbbbbbbbbbes/e//// 04644
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2019.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning, In ECCV, 2020.
Minhao Cheng, Thong Le, Pun-Yu Chen, Huan Zhang, JinFeng Yi, and Cho-Jui Hsieh. Query- efficient hard-label black-box attack: optimization- approach. In International Confer- ence on Learning Representations, 2018.
Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box ad- versarial attacks with a transfer-based prior. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL htRLttsssstt.. httptttttpppttttt.etttttteoeeeet neurips...
Alexis Conneau, Guillaume Lample, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint
2011 2017.
Thomas Degris, Patrick M. Pilarski, and Richard S. Sutton. Model-free reinforcement learning with continuous action in practice. In 2012 American Control Conference (ACC), pp. 2177-2182, 2012. doi: 10. 10.1109/AC..2012.631
Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. Visual turing test for com- puter vision systems. Proceedings of the National Academy of Sciences, 112(12):311-3362, 2015.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference Learning Representations, 2015. URL http:// arxiv. wwwp/bs/g///../eoooeree 6572
Ian Goodfellow, Patrick McDaniel, and Nicolas Papernot. Making machine learning robust against adversarial inputs. Communications of the ACM, 61(7):56-66 2018.
Chuan Guo, Jacob R. Gardner, Yurong You, Andrew G. Wilson, and Kilian Q. Weinberger. Sim- ple black-box adversarial attacks, 2019a. URL https://ppeneeeiewweee//ooo
IJeZS3RcYml
Dalu Guo, Chang Xu, and Dacheng Tao. Graph reasoning networks for visual question answering. CoRR, abs/1907. 009811, 2019b. /tt.
907. 1907.09115
FR Helmert. Die genauigkeit der formel von peters berechnung des wahrscheinlichen beobach- tungsfehlers director beobachtungen gleicher genauigkeit. Astronomische Nachrichten, 88:113, 1876.
10

---

Published as a conference paper at ICLR 2022
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European Conference Computer Viston (ECCV), pp. 771-787, 2018.
Drew Hudson and Christopher Manning. Compositional attention networks for machine rea- soning, arXiv preprint arXiv: 1803.03067 2018.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709 (700-670,,400 2019.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv preprini arXiv:2 2021.
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. In defense of grid features for visual question answering. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017a.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017b.
Ameya Joshi, Amitangshu Mukherjee, Soumik Sarkar, and Chinmay Hegde. Semantic adversarial attacks: Parametric transformations that fool deep classifiers. In Proceedingy of the TEEF/CVF International Conference Computer Vision, pp. 4773-4783, 2019.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Modulaeodullted detection for end-to-end multi moti-lmddal understanding. arXiv preprint arXiv:2104. 12763, 2021.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A Smith, and Daniel Weld. Genie: A leaderboard for human-in-the-loop evaluation of text generation. arXiv preprint arXiv:2101.06500 2021.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint arXiv:161..012550 2016.
Shane Legg. Machine super intelligence. PhD thesis, Università della Svizzera italiana, 2008
Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. Adversarial vqa: new benchmark for evaluating the robustness of vqa models. arXiv preprint arXiv:2106.00244 2021.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016. /ttttt::
org/ wttpoorgg/aa/o/ooooeee abs//a/9.....///////// 02971
Locatello. Object-centric learning with slot attention. In Neural Information Processing Systems (NeurIPS), 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for rision-and-languaar tasks. arXiv preprint
2019,,
J Lüroth. Vergelichung von zwei werthen des wahrscheinlichen fehlers. Astronomische Nachrichten, 87:209, 1876.
Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about real- world scenes based on uncertain input. Advances in neural information processing systems, 27: 1682-1690, 2014.
11

---

Published as a conference paper at ICLR 2022
Mateusz Malinowski, Carl Doersch, Adam Santoro, and Peter Battaglia. Learning visual question answering by bootstrapping hard attention. In Proceedings of the European Conference on Com- puter Vision (ECCV). pp. 3-20, 2018.
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Clos- ing the gap between performance and interpretability in visual reasoning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedingg of The 33rd International Conference on Machine Learning. volume of Proceedings of Machine Learning Research, pp. 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. httppt:: //proceedings. press / /48/ www.mmn...hhtccmmmmmmmm html
Seyed-Mohsen Moosavi-Dezfoolli Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2016.
Seyed-Mohsen Moosavi-Dezfooli: Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1673, 2017.
Nie, A Williams, Dinan, M Bansal, Weston, and D) Kiela. Adversarial nli: A new 521 benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the 522 Association for Computational Linguistics, pp. 4885-4901, 2020.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM Asia conference computer and communications security, pp. 506-519, 2017.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
Oskar Pfungst. Clever Hans:(the horse of Mr. Von Osten.) contribution to experimental animal and human psychology. Holt, Rinehart and Winston, 1911.
Piotr Piekos, Henryk Michalewski, and Mateusz Malinowski. Measuring and improving bert's math- ematical abilities by predicting the order of reasoning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the I Ithh International Joint Conference on Natural Language Process- ing, ACL/IJCNLP 2021, (Volume Short Papers), Virtual Event, August 1-6, 2021. Association for Computational Linguistics, 2021.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Leurning Research, pp. 1889 1897, Lille, France, 07-09 Jul 2015. PMLR. URL https::/ppoceeeeeeeee
.mlr.
mttas//33/schhlmaane html
Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Alberto Lopez Magana, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Human-adversarial visual question answering. arXiv preprint Ntt::2100602280. 2021.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre- training of generic visual-linguisti representations. International Conference on Learning Representations, 2020. URL htt.sttt::
12

---

Published as a conference paper at ICLR 2022
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http:://icccmllleeeidaaaaaaee//s html
2nd.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014.
Hao Tan and Mohit Bansal. Lxmert: Learning mross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through guestion-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4631-464444 2016.
Alan M Turing. Computing machinery and intelligence. In Parsing the turing test, pp. 23-65. Springer, 2009.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008 2017.
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-cnn. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10760- 10770, 2020.
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkow, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. CoRR, abs/1611.01122 2016. (ttLptt::///xttt org abapg//as/1111 c1224
Yuhuai WU, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust- region method for deep reinforcement learning using kronecker-factored approximation. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL. htttps://proces ings neur ps. htc//pappe/201//11ee 361 44442287666bb48
9018455f415225 Pappssprr--st-pppppr_....p pdf
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. arXiv preprint arXiv:1810.02338, 2018
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang, Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi-Keung Tang, and Alan L Yuille. Adversarial attacks beyond the image space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4302-4311, 2019.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5579- 5588, June 2021.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 URLS OF CLEVR MODELS
Table 3 shows the URLs models used in our investigations (also also report if we re-trained model from scratch (type Architecture) or used already trained models (type Model). Please note that the latter type proves that our testing procedure fully black-box
Table the main paper). We
Model Name
Link
Type
SAN
Yang et al. 2016
https://github.com com/faaccbookrreeecccccceeerrrr
Model
IEP
Johnson al.
htlpss://githhb http:/aacccooorreeeencceeeeeeeg
Model
FiLM
[Percz al.
https:
Architecture
RN
(Santoro
2017
https://ithhbb www.e//wwwsesiee/oooooooooootote
Architecture
ThD (Mascharka
2018
https://gituhh http://///hhhhhhoo///r/rrdiiiia http://ghhh...oommmmdiddnnnaa/aa
Model
Mdetr (Kumath et al.
2021
https://gthu...oo////ttt wwwp://www..o./eeeaaaaaaamnetttt
Model
URLs to Models. Architecture denotes that we use the code but re-trained the model on CLEVR. Model refers to already trained models.
Table 3:
A.2 OBJECT CATEGORIES
Figure
shows Aigurs and materials. Small objects are rendered at approximately 40% of the size of their large counter- parts. Metallic materials can be identified by the high albedo property and 'shininess' of the object. We provide it as the reference point for reader to make it easier to compare our qualitative results.
made-up image that contains all available CLEVR object categories: shapes, sizes,
Large Red Rubber Sphere
Large Blue Metal Cube
Small Metal Green Cylinder
Small Rubber Brown Cylinder
Figure Object categories the CLEVR dataset.
14

---

Published as a conference paper at ICLR 2022
A.3 STATE-INPUT TRANSFORMER
To test our two-player game against a state-inpul Visual-QA Player, we have designed a transformer- based architecture (Vaswani al. 2017) that receives six types of input features object sizes, object shapes, object materials, object colors, object positions, and question tokens and uses a cross-modal attention mechanism. In that mechanism, queries from one modality attend to keys and values from the other modality. Each data point contains variable-length inputs, describing all objects in the scene. also contains question tokens that compose the question itself. Hence, we do padding with special token 0 to the maximal input length. We set the maximal length for the objects tokens to be 10 6 = 60 and 50 for the question tokens. All all, we have 110 input tokens. Each token type is projected into a different embedding space and learnable type embedding is added to it, separately for object and question tokens. For instance, emb (materiaal + emb(object). Furthermore, three special learnable embeddings as queries to reduce the overall computational costs of the cross-attention mechanism. The same mechanism is also used in Perceiver (Jaegle et al. 2021). The input tokens (concatenated object and question tokens) form keys and values. In every transformer block, we apply cross-attention between all input embeddings and those three special tokens. This repeated five times. As the final block, a feed-forward network (classifier) receives as inputs the three latent tokens and outputs answer probabilities. We show that architecture Figure
used as additional input. We them
"Green"
FFN Classifler
Special Embeddings
Cross Attentio Transformer Block X
Size Type Emdedding
Add LayerNorm
Add LayenNorm
Add + LayerNorm
Add LayerNorm
Shape Type Embedding
Malerial Type Embedding
Taberl
Green
Gry
Gerai
cuee
Gender
Soane Description
Color Type Embedding
Graph
Add LayorNorm
Question Type
Question
De
Geoce
Embedding
What color is the small cylinder?
Figure 5: Our Input Transformer that operates on the graph scene (states).
A.4 MANIPULATIONS OF State-Input Visual-QA Player
Figures 6 and show manipulations of Visual- QA Player that is trained to take states instead of pixels the input. Such state-input Visual-QA Player receives direct overview of the scene, bypassing any need for any image renderings. We use State-Inpul Transformer (Section A.3) as multii-oddl component of that player. Since such model gets the perfect visual information the input, makes it more robust under scene manipulations. As consequence, our Adversarial Player tends to manipulate the scene that objects placed closely together.
15

---

Published as a conference paper at ICLR 2022
there equal number of small things that and large cyan matte balls?
State Before
False
State After
True
the number of big blocks that are in front of the brown metallic cylinder
the same as the number of tiny cylinders on the right side of the rubber cube?
State Before
False
After
Figure 6: Qualitative results state-input Visual-QA Player. For each example, top-down view of the
presented. The eye the bottom right each view represents rendering camera. It corresponds to Visual- QA Player's viewpoint. Each seene also rendered and presented side-by by- side for comparison.
16

---

Published as a conference paper at ICLR 2022
How many other things are both behind the purple rubber sphere and to the right the gray cylinder?
State Before
State After
What the material of the cube that is behind the rubber cube?
State Before
Metal
State After
Rubber
Figure 7: Qualitative results on state-input Visual- QA Player. each example, a top-down view of scene presented. The eye at the bottom right of each view represents the rendering camera. It corresponds to the Visual- QA Player's viewpoint. Each scene is also rendered and presented side-by-side for comparison.
17

---

Published as a conference paper at ICLR 2022
A.5 ALGORIHTMS
We show pseudo-algorithms that we use to (Algorithm calculate rewards, (Algorithm 2 train Adversarial Player, (Algorithm 3) and play a game.
Algorithm 1 Calculate Rewards
1: new answer Answer produced by Visual Agent on perturbed image
2: ondwere Answer produced by Visual Agent on original image
3: gt onswwrr Ground Truth Answer
4: Drop Reward
5: Consistency Reward
6: fr Fail Reward
7: reward 0
8: if new answer * old answer then
9: if old answer gt answer then
10:
reward = reward + or
else
12:
reward = reward + dr
end if
14: else
reward reward fr
16: end if
return reward
Algorithm 2 Training Pipeline
1: M Minigames
2: Va Visual Agent, Fe: Feature Extractor
3: A Actor, C' Critic
4: Qre Question Relevance Enforcer, Sce Scene Constraint Enforeer
5: for batch in M do
6: rewards, state values, logprobs <- DataGame(M, Va, Fe, A, C, Qre, Sce)
7: n Irewarddds
8: advantage + rewards - state_volues
9:
ploss -logrrobs x advantage
10: vloss + (state.values - stop-grad(yewardss} /n
11: loss + ploss vloss
12:
backprop(loss, A, C)
13:
end for
14: return
18

---

Published as a conference paper at ICLR 2022
Algorithm 3 Game between players (Game)
1: M MMiiiigan Hamm
2: Pvoa Visual Agent, Sy Multi-modal Backbone
3: A Actor, C Critic
4: Qre Question Relevance Enforcer, See Scene Constraint Enforcer
5: isr Invalid Scene Reward
6: for batch in do
7: rewards +- {}
8: state values k- 0
9: logprobs +- ()
10: for (image, scene, question, program, gt_answer) in batch do
old_anse old-answer + PvQA (image, question)
12:
extracted.featuree + Sy (scene, question)
dr. dx.probs, dy-psobs t Aleatracted.featureee
14:
CulX Calegorical((z probb))
CatY + Calegoricalldy probs)
16:
state_value <- Cllettracted_ fetuurs)) dr + sample CatX
17:
18:
dy sample CatY
19:
new scene +- Perturbate (scenc, dx, dy)
20:
check _eccccssceeeeee <- preenewwsceend
21:
check question + Qre(new scene, program, pl_nnswer)
22:
if check A check squestion True then
23:
new new imagee +- Render(newsscenn
new_onswer PvOA (new_image, question)
25:
rewards + rewards CalcRewards(neu answer, old answer, gt answer)
else
27:
rewards < rewards isr
28: end
29:
stale_values state_volues) state_value
30:
logprob + logprobll (og(pld log(p(dy (Ctt))))
31:
end for
32:
end for
33: return rewards, state values, logprobs
19

---

Published as a conference paper at ICLR 2022
A.6 SENSITIVITY TO QUESTION TYPES
Figure 88 shows the susceptibility of CLEVR models to scene manipulations of Adversarial Player. We observe that especially counting and existence questions the 'back-doors' for scene ma- nipulations. Questions of those two types typically involve multiple steps of reasoning. Adversarial Player's scene manipulations make them more likely that reasoning will fail some stage.
125
Frequency
Frequency
134
9p
(Comboned))
Qveryy Mdee
Eruutt shapes
Querr materia
Exst
Gaunt
(Cantiond)
Query cater
Query matertal
Esnall shope
Cavery sice
Esual mustriia
Geerr shane
Exves soe
Expail
Edpalt
Equail
Gaerr
Qvery
(a) FiLM
(b) TbD
Frequency
Other Comboned)
Qvery color
Query materaal
Equal museriaa
Equel shape
Ganey shone
East
Equai
Emaal
Query
Mdetr
Figure 8: Histograms of the model's susceptibility to scene manipulations conditioned on question type.
20

---

Published as a conference paper at ICLR 2022
A.7 CONVERGENCE
Figure 9 shows convergence plots of Adversarial Player during training against FiLM, TbD and MDetr. We use different initialization seeds and 30 trials. Adversarial Player is trained either from states or pixels. The former yields higher performance and sometimes better convergence. We use the Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated. The higher Drop, the more successful manipulations are.
Mini- Mini-game game size 100
Mini-gama size 1000
Dop
10
Film
00A plarke
plank
TbD
0.5
Eooch
Easch
MDeer
Figure 9: Performance of Adversarial Player for different training snapshots. We show the mean and variance
over thirty trials. Adversarial Player is either trained from states (blue) or pixels (orange).
21

---

Published as a conference paper at ICLR 2022
A.8 PERFORMANCE DISTRIBUTION PLOTS
Figures Figures[[O and I1 show the distribution of the Adversarial Player performance, where we show how often (y-axis) the given Drop score is achieved (x-axis). The Drop metric measures the accuracy drop after the Visual-QA Player is manipulated. The higher Drop, the more successful manipulatiins are. Larger Mini-games lead more consistent (narrower distributions) but lower performance. This behavior suggests that scene diversity present in larger Mini-games leads to more stable scene manipulations. However, they do so at the cost of increasing the complexity of the optimization problem.
Mini-game size 100
Mini-game size 1000
Deneity
Dereity
Film
Drap
Drop
Demeity
Demsity
TbD
Drop
Drop
Dersity
Dereity
MDeer
Drop
Drop
Figure 10: Mini-games performance distribution for state-input Adversarial Player.
22

---

Published as a conference paper at ICLR 2022
Mini-game game size 100
Mini-game size 1000
Density
Deneily
Film
Drop
Drop
Density
Density
TbD
Drop
Drap
Density
Densiyy
MDetr
Drop
Drop
Figure 11: Mini games performance distribution pixel-input Adversarial Player.
23

---

Published as a conference paper at ICLR 2022
A.9
MANIPULATIONS
If we do not provide extra constraints on the scene generation process with our environment en- forcers, Adversarial Player may find easy manipulation that changes the camera pose loca- tion, effectively 'zooming out' the whole scene. It does so by stretching all the object coordinates in the scene. Figure [2] iillustrates that. For instance, large objects look much smaller than typical examples. As such constructed scenes are quite different from the ones that Visual-QA Player has observed during training. we categorize such manipulations to nut-of-distribution. Note that, due to our in-distribution environment enforcers, such scenes are prohibited in our pipeline.
(a) Shape of the blue cube is misclassified as sphere.
(b) Large, brown cylinder is misclassified small.
(c) Large, yellow cube misclassified as small.
Figure 12: Out-of-distribution examples.
24

---

Published as a conference paper at ICLR 2022
A.10
ADDITIONAL QUALITATIVE RESULTS
In Figures 13 21
we provide more qualitative results.
Is the number of small cubes that are right of the blue rubber block greater than the number of small balls that are in front of the cyan object?
What is the color of the metallic thing that is on the right side of the shiny thing behind the large gray object?
SAN Before
Yes
Gray
SAN - After
No
Green
Figure 13: Manipulations of CLEVR models. We show results before and after scene manipulations.
25

---

Published as a conference paper at ICLR 2022
There is cube on the left side of the brown object that is behind the red ball; what size is it?
What shape is the thing that is in front of the small yellow cylinder and to the right of the rubber thing?
SAN - Before
Large
Cylinder
SAN - After
Small
Sphere
Figure 14: Manipulations of CLEVR models. We show results before and after scene manipulations.
26

---

Published as a conference paper at ICLR 2022
How big is the shiny cylinder that is behind the rubber cylinder?
There is a thing that is the same color as the cube; what is it made of?
IEP- - Before
Large
Rubber
IEP- - After
Small
Metal
Figure 15: Manipulations of CLEVR models. We show results before and after scene manipulations.
27

---

Published as a conference paper at ICLR 2022
Does the cyan metal object that is behind the tiny gray block have the same size as the block behind the small cyan metallic objec?
What is the big thing that is behind the large red shiny sphere made of?
IEP - Befoore
No
Rubber
IEP After
Yes
Metal
Figure 16: Manipulations of CLEVR models. We show results before and after scene manipulations.
28

---

Published as a conference paper at ICLR 2022
There is a metallic object; does it have the same size as the rubber block right of the tiny shiny cube?
How many other small shiny objects are the same shape as the red object?
RN - Before
No
RN - After
Yes
2
Figure 17: Manipulations of CLEVR models. We show results before and after scene manipulations.
29

---

Published as a conference paper at ICLR 2022
Are there any other things that are the same shape as the big metallic object?
Are there any other things that are the same size as the ball?
RN - Before
No
No
RN After
Yes
Yes
Figure 18: Manipulations of CLEVR models. We show results before and after manipulations.
30

---

Published as a conference paper at ICLR 2022
How many objects are there of the same size as the brown object?
There is another object that is the same size as the blue rubber object; What is it made of?
MDetr - Before
Rubber
MDetr- - After
Metal
Figure 19; Manipulations of CLEVR models. We show results before and after scene manipulations.
31

---

Published as a conference paper at ICLR 2022
What number of other objects are the same color as the block? ?
What number of metal objects are small objects or big cyan objects?
MDetr - Before
MDetr - After
Figure 20: Manipulations CLEVR models. show results before after scene manipulations.
32

---

Published as a conference paper at ICLR 2022
Is there an equal number of cubes in front of the large shiny thing behind the cyan block and behind the yellow cube?
Are there less matte objects on the left of the green block than metal things?
TbD - Before
No
Yes
TbD - After
Yes
No
Figure 21: Manipulations of CLEVR models. We show results before and after scene manipulations.
33

---

Published as a conference paper at ICLR 2022
A.11 QUANTITATIVE RESULTS WITH P-VALUES
We extend the quantitative results of the main paper by also reporting p-values of the one-sample hypothesis testing. We the Accuracy Drop metric that measures the accuracy drop after the Visual-QA Player is manipulated, and the Consistency Drop that measures how many times the manipulated Visual-QA Player changes its answer, independently if that is correct or wrong an- swer. The higher Drop Accuracy or Consistency Drop, the more successful manipulations are. We use T-Test (Helmert 1876. Lüroth 1876) for each metric all the runs. Our null hypothe- sis is that the population mean of all the games is zero, indicating that manipulation is unsuc- cessful, and the results of our Adversarial Players are extreme cases of a good performance, i.e, Ho μ4 0,/ E [Drop, Consistency]. As an alternative hypothesis, we assume that the population mean is greater than zero, i.e. HA? it > 0. I E Drop, Consistency]. The same setup stands for both the Consistency and the Drop metric. Tables 4 and 5 show the full table with p-values computed. Note that only in two cases, Adversarial Player does not manipulate the scene convincingly (TbD and Mdetr for the pixel-inpul Adversarial Player).
Table 4: Results of the game with the state- imput Adversarial Player.
Average
p-value
Model
Mini-game size
Consistency Drop %
Accuracy Drop %
Consistency Drop
Accuracy
Drop
SAN
10
-23.9
-14.2
0.000
0,000
100
-10.7
-8.3
0.000
0.000
1000
-3.5
-2.5
0.001
0.000
FiLM
10
-14.8
-12.7
0.003
0.006
100
-7.8
0.000
0.000
1000
-3.1
-2.4
0.000
0.000
RN
-20.9
-13.6
0.000
0.000
100
-10.5
-8.2
0.000
0.000
1000
-3.3
-2.8
0,000
0.000
IEP
10
-13.9
-13.0
0.001
0.003
100
-6.9
-6.8
0,000
0.000
1000
-2.9
-2.8
0.000
0.000
TbD
10
-5.3
-5.1
0,011
0.012
100
3.8
-2.5
0.000
0,000
1000
-1.1
-1.1
0.000
0.000
Mdetr
10
-6.6
-6.0
0.018
0.021
100
-4.9
-3.6
0.000
0.000
1000
-1.3
-1.2
0.000
0.000
State
-8.5
-7.7
0.020
0.034
Input
100
-2.3
-2.1
0.000
0.000
Model
1000
-1.2
-1.1
0.000
0.000
34

---

Published as a conference paper at ICLR 2022
Table 5: Results of the game with the pixel-input Adversarial Player.
Average
p-value
Model
Mini-game size
Consistency Drop %
Accuracy Drop %
Consistency Drop
Accuracy
Drop
SAN
10
-12.8
-11.4
0.008
0.018
-6.4
-3,9
0.000
0.001
1000
-233
-1.4
0.000
0.000
FILM
-3.62
0.056
0.154
100
-1.7
-1.4
0.000
0.000
1000
-0.6
-0.4
0.000
0.000
-9.4
-7.2
0.021
0.065
100
-3,2
-255
0.000
0.000
1000
-2.3
-1.7
0.000
0.000
IEP
10
-2.9
-2.5
0.111
0.171
100
-1.5
-1.3
0.001
0.004
1000
-0.4
-0.3
0.000
0.000
TbD
10
0.0
0.0
1.000
1.000
100
-0.6
-0.4
0.017
0.078
1000
-0.2
0.000
0.000
Mdetr
10
0.0
0.0
1.000
1,000
-1.2
0.002
0.003
1000
-0.4
-0.3
0.000
0.000
State
-0.9
0.033
0.038
Input
100
-0.7
-0.7
0.031
0.031
Model
1000
-0.3
-0.3
0.000
0.000
35

---

Published as a conference paper at ICLR 2022
B APPENDIX B
B.1 MODEL PERFORMANCE ON MAXIMAL. DROP MINIGAMES.
In the table below, present model accuracy each of the Mini-games games where the maximal perfor- mance drop was observed. The third and fourth columns (Original Data section) refer to the model accuracy unmanipulated image-question pairs of the respective Mini-game. The fifth and sixth columns (Manipulated Data section) refer to the model accuracy in those Mini-game examples after the Adversarial Player manipulation.
Original Data
Manipulated Data
Model
Mini- Mini-game size
Mini-game Accuracy (State)
Mini-game Accuracy (Pixel)
Mini-game Accuracy (State)
Mini-gane
Accuracy
(Pixel)
SAN
10
80,00
80.00
28.84
43.26
100
74.00
75.00
47.65
66.32
1000
72.30
72.20
68.42
70.87
FiLM
10
100.0
100.0
48.10
86.58
100
98.00
100.0
75.61
92.44
1000
96.40
96.10
90.81
94.75
RN
100.0
100.0
46.60
74.56
100
94.00
95.00
63,28
87.42
1000
93.10
93.30
90.03
91.05
IEP
10
100.0
100.0
48.45
87.21
100
98.00
96.00
74.41
93.50
1000
97.10
97.30
93.99
95.73
TbD
10
100.0
100.0
69.37
99.10 10
100
100.0
100.0
91.17
98.00
1000
99.40
99.70
95.53
98.80
Mdetr
10
100.0
100.0
59.82
99.70
100
100.0
100.0
86.43
97.80
1000
99,50
100.0
94.51
99.10
State
10
100.0
100.0
77.44
91.96
Input
97.00
99.00
92.63
95,05
Transf.
1000
97.20
96.40
95.15
95.83
B.2 GENERATING 'VISUALLY FAIR' SCENES.
We define a scene as visually fair if there are no occlusions and all objects are within the field-of- view. The ground truth answer of a posed question on visually fair scene remains the same before and after object manipulation.
For the generation of new images/scenes (v2.79b), and the original 3D models of the CLEVR dataset. Scenes suggested by our Adversarial Player, encoded in the form of arguments for Blender, which renders the new images according to those directives. In order to guarantee visual fairness, and question validity, we employ two mod- ules (scene-constraint enforcer question relevance enforcer), presented briefly in Section in the Consistency and In- n--Ditribution paragraphs.
the open-source Blender Graphics Engine
The scene-constrain enforcer responsible for ensuring the creation scene/image that respects all constraints and statistical properties of the original dataset. This piece of code is borrowed from the original CLEVR codebase and as in the original CLEVR paper the following constraints checked (Lines 66-83 in the python code):
Blender a 3D modeling and rendering package. https:://wwwttt https: / thub. com http///aacooookkeerrrkaeeee clvr gen/bl ob ma m /aainn//nnn /magge.
generati ontrratioo//rennnn on/render paatderr_mmaess_ppp py
36

---

Published as a conference paper at ICLR 2022
1. Minimum / Maximum Number of Objects: 3/ 10.
2. Minimum allowed distance between object centers: 0.25
3. Margin along with cardinal directions: 0.4 (This ensures lack of ambiguity by keeping objects at least 0.4 points of distance apart. This distance a Blender-specific measure that spans from -3 to +3.)
4. Min pixels per object: 200 (This ensures that an object will not be occluded, by requiring at least 200 pixels of it being visible.)
The question-relevance enforcer checks if the newly generated scene is indeed answerable and has the same "ground truth" answer as the old scene. By not enforcing this, every manipulation will be falsely "successful" as the question (which is kept the same before and after the attack) will have different answer in the newly generated scene.
This is also resolved by using the same question engine module provided by the CLEVR authors in their codebase The module receives a question and scene and calculates the ground truth answer based on a set of functional programs that are apriori known and used to generate the questions. (Supplementary Material A in the CLEVR paper (Johnson et al. 2017a))
We use the following pipeline. Our Adversarial Player initially receives an input image-question pair. Then, it suggests new scene, that passes through the scene-constraint enforcer. If the new scene respects all boundaries and constraints is then evaluated alongside the question by the question-relevancce enforcer. If the new ground truth answer to the suggested scene/image is the same as the original answer, the amage-question pair is marked as in-distribution and valid. Other- wise, it is removed from the training pipeline.
37

---

Published as a conference paper at ICLR 2022
B.3 COMPARISON WITH RANDOMLY SAMPLED EQUIVALENT SCENES.
Arguably, out of the semantically equivalent space of images given a question and an accompany- ing answer, the Adversarial Player effectively discovers cases for which the model under the test can be fooled. This, however, does not indicate how rare those examples are, neither the degree of robustness of the model against other examples in this space. Let us limit ourselves on a specific image-question pair. Then, a tractable estimate of how rare "fooling examples" are would be the number of manipulated scenes (F) that are able to fool the model, divided by all the possible per- mutations of available objects (N) in our discrete 7x7 grid. Rarity = 49
F .N E [3, 10]. A scene of
only objects and discovered successful manipulation would make it 0. 0.00085% possible to find by chance and would indicate 99.9915% robustness against that sphere of equivalent image-question pairs.
Of course not all of those configurations valid due to scene constraint violations, well as objects can theoretically move freely on the scene. As a way to approach this question, we compare our Adversarial Player against random scene generator (RSG). The random scene generator is able to manipulate any object freely, under the constraint that the resulting scene is valid under the scene constraint enforcer well as the question relevance enforeer. Manipulations of lighting or camera angles, as well as addition or removal of objects is not permitted. In order to reduce the huge rendering times needed for each random search, allow the RSG operate game where scene manipulations that cause accuracy drop have been discovered by the Adversarial Player. For each image-question pair the RSG has a budget of 5000 queries, and if not successful proceeds to the next Mini-game entry.
10 length Mini-
Model
CLEVR Accuracy
Adv.Player Accuracy (Drop %)
RSG Accuracy (Drop %) 71.59 (-0.7%)
SAN
72.1
61.86 (-14.2%)
FiLM
96.2
83.98 (-12.7%)
95.81 (-0.4%)
RN
93.2
80.52 (-13.6%)
92.73 (-0.5%)
IEP
96.9
84.30 (-13.0%)
96.70 (-0.2%)
TbD
99.1
94.04 (-5.1%)
99.10 (-0.0%)
MDetr
99.7
93.71 (-6.0%%
99.70 (-0.0%)
ST
96.8
89.34 (-7.7%)
96.50 (-0.3%)
As it can be seen, randomly question-uninformed suggested object placements do not seem to be effective into fooling the models under test. This further supports the argument that models un- der test theoretically and statistically seem robust, but nevertheless have vulnerabilities that can be effectively exploited by our Adversarial Player.
38

---

Published as a conference paper at ICLR 2022
B.4 VISUALIZATIONS.
Here, we try to isolate reasoning gaps from visual causes after the scene manipulations; with the following setup. Each Visual-QA Player is introduced with image-question pairs. The first consists of the original image and the second of a scene created by our Adversarial Player that successfully fools the Visual-QA Player. We then inspect the visualization of the Visual-QA. -QA Player's inner-workings in both pairs. We are especially interested in patterns that suggest the root of failure. As each Visual-QA Player operates under different assumptions and architectural designs, we choose to visualize:
1. The attention maps of Stack-Attention Networks.
2. The gradient-weighted activations of the pre-ultimate and ultimate convolutional blocks of FiLM.
3. The programs that are synthesized alongside their soft-attention maps of each module block in TbD network.
4. The objects that were detected during the computation steps of MDetr model.
In Figure 22 visualize the pre-ultimate (column 2) and ultimate (column 3) attention maps of the Stack-Attention Network. We observe that the model focuses on a single final object manipulated examples. In the original ones, the relational jumps required for the answer are still visible the attention map. The final attention map that is presented (column 3) is passed to fully-com connecconnected classifier in order for the answer to be created. We observe limited focus all the necessary objects / locations that creates a lack of necessary features for the correct answer derivation.
In the of FiLM model in Figure 23 observe similar scenario. In the first example, (rows and 2) the gradient-weighted activations of the model seem to be focusing on multiple irrelevant objects regarding the final answer, while in the second example (rows and 4) the model seems to include the purple cube mistakenly as one of the possible spheres of the image. In both cases above it is difficult to exactly pin down the root of (reasoning or visual) failures.
MDetr model can be seen in Figure 24 and Figure 25 Here, specific pattern could not be identified. All objects were correctly classified, It suggests that the cause of failures stem rather from the reasoning steps.
Tbd is presented in Figure 26 In both the original and manipulated scenes the programs that the model generates are exactly the same. This is to no surprise since the program generator module using exclusively the question as input. Nevertheless, a similar pattern of scattered model focus is presented here as well. The attention maps in both examples are the same between the original and manipulated scenes, in all but the final reasoning step. the first example (rows and 2) the attention is leaked towards the gray cube that has nothing do with the correct answer, while in the second (rows 3 and 4) the attention is spread among many matte objects but peaks near the big yellow ball. This is wrong since the word "other" that exists in the question should drive the model focus exclusively to other objects apart from the starting point the reasoning chain.
39

---

Published as a conference paper at ICLR 2022
Is the number of small cubes that right blue rubber block greater than the number small balls that are in front of the eyan object? Yes
No
Original Scone
Manipulated Scene
What shape is the thing that is in front of the small yellow cylinder and to the right of the rubber thing? Cylinder
Sphere
Original Scene
Manipulated Scone
Figure 22: Visualization of attention maps in SAN model on original and manipulated scenes.
40

---

Published as a conference paper at ICLR 2022
How many other objects the same shupe the big gray matte object? - 0
Original Scene
Manipulated Scene
How many other things are the same shape as the big metallic thing? -
Original Scene
Manipulated Scene
Figure 23: Gradient weighted activation visualization of FiLM model's pre-ultimate and ultimate blocks original and manipulated scenes
41

---

Published as a conference paper at ICLR 2022
The red thing that the same material
as the brown cube is what size? Large + Small
redt thing same material: 1.00
brown 1.00
Original Scene
red thing sa me materil: 1.00
brown cube:
Matdp Scene
Figure 24: Visualization of MDetr model's detection stages on original and manipulated scenes.
42

---

Published as a conference paper at ICLR 2022
Do the object behind the big matte sphere and the thing on the right side of the red thing have the same material? Yes - No
Lojject behind: 0.97 big matto sphere: 0.7]
thing right side of:
thing: 1.00
Original Scene
object behind: big matte sphere: 0.71
thing on the right side of: 1.00
red thing: GO
Manipulated Scene
Figure 25: Visualization of MDetr model's 's detection stages on original and manipulated scenes.
43

---

Published as a conference paper at ICLR 2022
Is there a big brown object of the same shape as the green thing? Yes No
filer calorfgreen)
same Shape
Fharr (intlarg))
fiter (arrjaanl
Manipulated Scene Original Scene
Shee cololgues)
witee Shape
simelenge
filee filegrur pur Innon]
Are there any other things that have the same material the big yellow ball? Yes
No
vieflange)
Manipulated Scene Original Scene
fieer sizel sizellagee
fiter: sol.rfylolllll
fiter stapelophere!
meteril
Figure 26: Visualization of TbD model's program generation and attention map visualization on original and manipulated scenes
44

---

Published as a conference paper at ICLR 2022
B.5 HUMAN-IN-THE-LOOP EXPERIMENT
Can our Adversarial Player 'fool' a human-agent? The perceptual abilities of human observers have been previously measured (92.6%) this dataset, however, human robustness is different question. We, hence, decided to assess human robustness by simple experiment. In our setting, human observer is presented with set of two images and question. Then the observer is required to answer to each image question pair, (image before question / 'mage after- question), and report whether the answer has changed remained the same. Participants were first made familiar with the concepts of the CLEVR dataset (colors / shapes materials / sizes) and then were left unsupervised to answer a set 20 such examples. Each contained one CLEVR question and two CLEVR scenes, before and after the Adversarial Player manipulations. Half of those scenes successfully fooled models and half those did not. An example of what survey question looks like given in Figure 27 together with the confusion matrix Figure 28 (results of our survey).
There is cube on the left side of the brown object that behind the red ball. What size Is it?
Same
Different
Figure 27: Questionnaire example: The observer is presented with two images and a question. Then they have pick if their answer would remain the same if the question referred cach image respectively.
As we can see in Figure 28 human observers exhibit 91% precision identifying cases where scene manipulations not causing changes the questions answer, and 95% precision in identifying cases where the manipulations do cause a change. While this small scale experiment may not reflect universal truth about human robustness in VQA scenarios, it suggestion that human observation is still reliable after our agent's 's manipulations.
45

---

Published as a conference paper at ICLR 2022
Confusion Matrix
Same
91
Predicted labels
Different
Same
Different
True labels
Figure 28: Confusion matrix of human experiment: Participants identified 91 % of cases where manipulations were not causing a change in the ground-truth answer. All of these managed to fool Visual-QA -A Players and made up half of the survey. When faced with manipulations that changed the ground-truth trudt--rthhhhh answer (the other half of the survey), they identified 95% of them.
B.6 OBJECT DETECTION EXPERIMENT
As extra test confirm the visual fairness (see section B.2 of Adversarial Player manipulations we run the following experiment. We employed an unsupervised object discovery model, called Slot Attention (Locatello) (2000,, pre-trained on CLEVR. together with classifier trained iden- tify object attributes. Next, we used that model to identify objects and attributes on original and manipulated scenes. Ultimately, we could not find any pairs which the number of objects or their attributes were different before and after the manipulations. The results for two pairs can be seen in Figure 29 and Figure (30 We thus conclude that our Adversarial Player manipulations create "visually fair' scenes, without producing any corner cases of perceptual system.
46

---

Published as a conference paper at ICLR 2022
Detected: Smed,Small,Meti Metal,Cuber
<Cyan, Metal,
Original
Matten,, LGray,Large,Metalluube
Detected:
( Cube> (Cyan,Large,Metallcy/ll Laree,,Larg,,aatespphee Metal, Cube>
Mutipulated
Figure 29; Visualization of Slot Attention model's 's detection results
Detected: Metal, <Cyan, Ball> Setay,Sall Murrple,Small,Metal/ Cube>
Original
Detected:
<Cyan, <Purple,Small, Metal, S Metal, Meea,,
Manipulaed
Figure 30: Visualization of Slot Attention model's 's detection results
47


Reviews:
[{"review_id": "w6wCXKXO2CU", "reviewer": "Reviewer_8VmJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper proposes a general framework to evaluate the reasoning capabilities of current VQA models. This evaluation could be useful as previous VQA models are likely to overfit the dataset bias instead of understanding and reasoning the question.\n\n\nWeaknesses:\n\nThe experimental results don't reveal much new information. The early models(SAN, FiLM, and RN) are simple and do not consider the reasoning abilities. TbD and Mdetr perform object-level reasoning. Thus it is predictable that they can maintain the average performance given the manipulated samples. Other State-of-the-art models on CLEVR, such as MAC[1] and NS-CL[2], are also likely to have consistent answers. The experimental results do not expose any new weakness of the evaluated VQA methods.\n\n[1] Drew A. Hudson, Christopher D. Manning, Compositional Attention Networks for Machine Reasoning, ICLR 2018\n\n[2] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, Jiajun Wu, The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision, ICLR 2019", "summary_of_the_paper": "This paper proposes to measure the reasoning capabilities of VQA models by training an \"adversary player\" that can manipulate the image and fool the evaluated models. Given a sample, the \"adversary player\" moves objects such that the groundtruth answer remains valid and the scene is in-distribution. If the evaluated VQA model produces a different output, the \"adversary player\" receives a positive reward and is then optimized with a reinforcement learning algorithm.", "summary_of_the_review": "Overall, this paper proposes a feasible way to evaluate the reasoning abilities of any black-box VQA model. Although the experimental results haven't revealed new problems, the proposed method can inspect current largely pretrained vision-language models to see if they perform reasoning on various tasks. Thus I endorse acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "o6W65GE0ho7", "reviewer": "Reviewer_ELDV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Prior work**\n* **Prior works about adversarial evaluation**: This type of adversarial analysis is not new in ML for QA models in particular (e.g. has been explored for reading comprehension over SQuAD and also over real-images) but it still interesting and unexplored in the context of CLEVR.\n* **Prior work about consistency**: The particular sort of adversarial analysts that the paper explores is actually about measuring consistency - asking the same question over different scenes that hold the same relevant relations to make the question's answer unchanged, and checking wether the model indeed keeps its answer consistent over the different scenes. Such sort of evaluation has been proposed e.g. as part of the GQA dataset a couple of years ago, by pre-computing variety of semantically-equivalent questions and then measuring the consistency over them. \n* **Related work**: The related work section gives a good coverage though in describing prior papers in related fields.\n\n**The approach**\n* **Problem statement**: The authors propose a clearly-defined approach and present it with a good degree of detail and preciseness. \n* **adversarial scene manipulation**: A strength of the approach is that compared to other adversarial experiments that tend to change the inputs to be valid but not natural (e.g. a random or unreadable sequence of words that fails a textual QA model, or changes to pixel values that impact a visual models abut can't be noticed by a person), the sort of adversarial modifications performed by the proposed approach are completely natural and explainable (manipulating objects within the scene). \n* **Perceptual vs. reasoning failures**: At the same time, I do worry about whether maybe some of the adversarial modifications are potentially just too visually-difficult for the model rather than really exposing real reasoning failures. There are multiple causes the could lead for that: (1) moving an object to be too occluded by another object so to be hard to identify, (2) or moving an object to be near another object of the same color in a particular position that will make the model harder to distinguish between them, (3) or making objects too similar in x or in y position such that it's hard to determine if one it behind/in front / to the left/right of another, or (4) putting a metallic object at a very particular position in the scene and/or partially occluded such that it will be hard to notice it's metallic reflection. Some of the qualitative examples indeed look like that is potentially the source for the change in the model answer.  I think there are 2 ways to mitigate this potential issue: First, constrain the adversarial model so to prevent it from making valid but \"visually unfair\" modifications (e.g. making object too close to each other etc), and second quantitatively measuring how the modification impact the performance of a pre-trained CLEVR object detector. On the standard CLEVR data a pre-trained detector can reach almost a perfect accuracy, and in the cases its performance will go down too due to the adversarial modification, we will know that the source of the failure was in perception rather than in reasoning. Overall I recommend the authors to explore these options further in order to give isolate the reasoning failures and strengthen the claims in the paper.\n* **Other datasets**: Another important way to increase the confidence that the failures happen due to reasoning and not perception, which is always a good practice in general - is to test the approach also on another dataset. e.g. the NLVR or other 2d scenes could be good candidates, since their perceptual elements is easier than in clevr and so they are less prone to adversarial perception failures.\n* **Adversarial vs. pre-computing questions**: I actually think that the pre-computed version as in GQA potentially allows simpler means for carrying out the consistency evaluation (by simply testing a trained model over a set of test questions). On the other hand, the adversarial conditions do enable looking for questions that will be especially hard for the particular evaluated model -- on a sense these settings therefore are more challenging, since they focus on looking for particular blind spots of the model, rather than measuring its consistency over randomly sampled equivalent scenes -- i.e. it could be that out of the space of semantically equivalent scenes, the model stays consistent in 95% of them, and the adversarial model finds those few rare cases it doesn't, not giving quantitative indication to how rare these cases are. Random sampling of multiple equivalent scenes would in contrast give a better estimation of the degree of consistency. Overall, I think while each approach has positives and negatives, it could be nice to include comparison in model's performance in the adversarial vs. randomly sampled equivalent scenes settings, in order to give a bigger more detailed picture about models' degree of consistency.\n* **Black-box evaluation**: The black-box aspect of the approach -- the fact that it measures models performance and reasoning capabilities just by looking at their textual answers can be both a strength and a weakness: on the one hand, it make the evaluation more easily performed on a wider variety of models, but on the other, it gives a narrow view into why models fail when they do. Especially for CLEVR, there are multiple models that are e.g. based on  computing attention maps, executable programs, structured representations and other indications for their ongoing reasoning rather than just predicting an answer. I think it will be good to pick couple of such models and explore in the paper how they perform internally when failing vs. when being consistent -- are the any interesting patterns in the computed programs or attentions when models fail? Are there particular settings, scenes or questions that make models to be particularly vulnerable? having such sort of analysis would help a lot getting further insight not just into whether models are good or bad reasoners, but also for why are they not good enough and what could be done to improve them.\n\n**Writing quality**\n* The writing clarity of the paper is good, the idea and content are easy to follow, the motivation is clearly explained, and there are good visualizations that help understanding. \n* **Spaces**: One small note regarding presentation of the manuscript: I guess due to space constraints there are no paragraphs at all -- the whole paper after the first couple pages is a one long block of text -- which makes it harder to read visually, so strongly recommend getting it back to the standard format in terms of spaces etc.\n", "summary_of_the_paper": "The paper explores measuring VQA models under adversarial settings, by having two competing models, when reasoning over a CLEVR scene as usual, and the other seeks to make the scene more challenging to answer. These settings reveal the weaknesses of visual reasoning models, and demonstrate that their apparent almost perfect performance on the standard CLEVR dataset may give imprecise impression about their “true” reasoning capabilities, as the adversarial settings may better reveal. \n\nUpdate: Following the detailed and thorough response by the authors that addressed most of my concerns, I'm happy to increase my score.", "summary_of_the_review": "Overall, I think the paper proposes an interesting evaluation but is still missing empirically in multiple aspects that could strengthen the paper:\n* Tracking potential reasons for the failure source (perceptual or reasoning) and making sure to reduce ways for the adversarial model to place objects in ways that make them too hard from the perceptual side to answer correctly.\n* Comparing adversarial to randomly-sampled scenes to measure the degree of consistency given a question rather than penalize the model for the existence of a single way per question to fool it. \n* Explore the model at least one more additional environment (e.g. 2d)\n* Explore additional indications about models behavior when they do and do not fail, to give further insight about potential reasons for why they are in/consistent in different cases.\n\nI therefore give the paper a marginal score, and encourage the authors to update the paper to make the evaluation more extensive! ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kxQ9mx-Axk", "reviewer": "Reviewer_u1a9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To this reviewer, this is an interesting paper, that passes the bar of acceptance in ICLR. The experiments are thorough and well-explained. The writing is clear and detailed and the results are interesting. However, there are a few concerns that prevent me from giving a better score. Details Below\n\nStrengths:\n\nS1. Thorough Experiments: The choice of both data-driven (RN, FiLM), ground-truth program based (IEP, TBD) as well as a host of experiments clearly show the efficacy of the proposed model. There is very little to complain about ablations, details, statistical tests, and thoroughness in experiments.\n\nS2. Clear Writing: The writing is clear and easy to understand. It also has enough details to easily allow reproduction by an expert.\n \nS3. Interesting Results: The results are not always surprising. As expected, the program-based models are more robust to adversarial attacks compared to \"from-scratch\" methods. Regardless, some of the results for several strong non-program-based models still come as a surprise to me and raise interesting questions about the tested algorithms as well as learning as a whole. To me, this is the biggest strength of the paper.\n\nWeaknesses:\n\nW1. Experimental Validation has some weird issues: Maximal scores are, as I understand it, measure the loss of accuracy in the \"worst\" minigame compared to overall accuracy. I think that is an incorrect way to report it. In my opinion, it should be compared to the accuracy in *THAT* particular mini-game as opposed to comparing it with average accuracy for the model. Please clarify this choice. \n\nW2. Some details are missing: How are the manipulations done and are steps taken to avoid unintended issues, e.g., occlusions, that would make a question unanswerable/ambiguous? \n\n\n", "summary_of_the_paper": "The paper proposes a black-box adversarial attack on models trained on the CLEVR dataset. CLEVR dataset is widely considered solved, especially with program synthesis-based models which seem to achieve near-perfect accuracy on CLEVR with a fairly small amount of training examples. Moreover, due to their synthetic and limited nature, it can be reasonably assumed that they would be able to answer *any* variation of CLEVR scenes within the bounds of the CLEVR-universe (Barring intentionally OOD test sets such as CoGENT). However, this paper shows that it is possible to fool all classes of the CLEVR model using an adversarial player that simply re-configures the scene (which should have the same answer as before). While the drops inaccuracy is not drastic, it is still an interesting finding, especially for program synthesis-driven methods. The paper also ends with an experiment about whether a purely data-driven approach can ever learn to be fully \"robust\". ", "summary_of_the_review": "Overall, this is a solid paper with really interesting results. There are a few issues, noted in weaknesses, that prevent me from giving it a higher score. I will happily assign a higher score after rebuttal if those points are clarified.\n\n\nUPDATE: I have read the author's responses and all other reviews carefully. Like others, I had some concerns about scene occlusion, etc. issues, but the authors have satisfactorily answered that. I also appreciate the authors' commitment to update the paper according to the reviews. That being said, I do not think the revisions have made for any *new* information for me to change my scores. I still think this is an interesting paper, albeit a bit limited in scope. I tend to keep my score as is ( ~ a weak accept)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Introducing an adversarial agent that re-configures the rendered scenes of CLEVR to demonstrate that models that appear achieve super-human performance are actually easily fooled due to their lack of ability to reason, provides a nice insight into limitations with existing approaches and correspondingly how we evaluate on some benchmarks.  There is a persistent concern that the results are only on CLEVR, and that the adversarial examples are not really disproving reasoning but rather issues with vision.  However, overall reviewers were generally positive about the aims the work.
Decision: Accept (Poster)