Measuring CLEVRness: Black-box Testing of Visual Reasoning Models

1 INTRODUCTION . Are our artificial intelligence systems capable of reasoning ? Or like Clever Hans , they use various cues only tangentially related to the task and rely on rote memorization with poor generalization ? ( Pfungst , 1911 ; Johnson et al. , 2017a ) This work revisits such a question and proposes an interactive framework with the communication channel between two players . The first player , which reasoning capabilities we are about to test , performs visual reasoning tasks , we call it VisualQA Player . The second player , which we call the Adversarial Player , is manipulating the scene so that it fools the first player even though those changes still lead to correct reasoning steps among humans . Both players interact with each other only through questions , answers and the visual scene as shown in Figure 1 . If the Adversarial Player manipulating the scene causes the Visual-QA Player to change its answer even though the new scene is still valid for the same question and answer , it is then the reasoning failure . It is similar to the following situation . Imagine a box is placed between two spheres . If you ask a question , is there a box between two spheres ? , the answer should be positive . Now , if we move the box anywhere so it does not cross any of the spheres , and ask the same question , the response should remain unchanged . In other words , we postulate that reasoning outputs of agents need to be invariant under scene configurations that are consistent with the questions-answer pairs . Moreover , in the spirit of generic adversarial attacks , we seek configurations that also pose little if any reasoning challenges for humans . We propose an automatic and agnostic pipeline to benchmark the reasoning capabilities of various models , only assuming they can communicate by answering questions about the scene . Due to the recent stream of research in vision-and-language ( Zhang et al. , 2021 ; Jiang et al. , 2020 ; Guo et al. , 2019b ; Su et al. , 2020 ; Wang et al. , 2020 ; Kamath et al. , 2021 ; Tan & Bansal , 2019 ; Chen et al. , 2020 ) , we believe there will be an increasing number of vision models that operate through language . Moreover , we also consider the visual question answering framework set-up as a two-player system as an excellent benchmarking pipeline . We perform all tests by scene manipulations and observing how a tested model behaves under such changes . The pipeline does not require any knowledge of the internals of the tested model . It also does not manipulate the sensory information of such a model , e.g. , pixels in the images , and all the manipulations are physically meaningful . Even though our current pipeline uses synthetic scenes as only those can easily be automatically manipulated , our results have also real-world ramifications . If models are susceptible to semantically meaningless changes1 in scene configurations , in a synthetic setting , there are valid concerns that real-world robots could also be prone to manipulation of objects in a room . Finally , our work also questions the possibility of training and benchmarking networks in a purely data-driven and offline , static manner . Contributions . The main contributions of our work could be summarized in three points . First , we propose a strong black-box adversarial test , which makes no assumptions about the underlying mechanics of a tested model , formulated as a game between two players . Our test does not require any direct access to the tested model , even through its sensory information . In particular , it does not require gradients , output probabilities , or any access to the perceived image . Our work also deviates from bounded perturbations and instead focuses on global scene manipulations that are still consistent with the task constraints , and can change the behavior of a tested model . Second , we reformulate visual reasoning by integrating visual question answering with zero-sum two-player game frameworks . Under our novel formulation , a visual and adversary agents compete against each other through content manipulation . We believe that this is an initial step towards more sophisticated frameworks that integrate computer vision with multi-agent systems . Third , we explore the limits of the data-driven approaches in synthetic visual scenarios , and demonstrate that current CLEVR models are lacking the efficiency to learn robust reasoning steps . 2 RELATED WORK . Our work touches upon various research directions , which we briefly describe here . Visual QA . Introduced as a visual counter-part of the Turing Test ( Malinowski & Fritz , 2014 ; Geman et al. , 2015 ) , it became a computer vision task that requires a holistic visual understanding . Many other works have extended the task to larger datasets or videos or adversarial train-test splits ( Antol et al. , 2015 ; Agrawal et al. , 2018 ; Tapaswi et al. , 2016 ; Yu et al. , 2019 ) . More recently , we also observe the generalization of the task to become a part of the vision-plus-language suite of benchmarks ( Lu et al. , 2019 ; Chen et al. , 2019 ; Wang et al. , 2021 ) . Johnson et al . ( 2017a ) have introduced a synthetic variant of the Visual QA problem that is more focused on reasoning and the control of the experimentation . Although it was shown that traditional methods ( Yang et al. , 2016 ) are not enough to solve that dataset , newer methods can pass the human baseline on this task . These methods are trained to operate on pixels and text directly ( Santoro et al. , 2017 ; Perez et al. , 2018 ; Hudson & Manning , 2018 ; Malinowski et al. , 2018 ; Kamath et al. , 2021 ) or they induce programs as an intermediate representation ( Johnson et al. , 2017b ; Mascharka et al. , 2018 ; Yi et al. , 2018 ) . Hudson & Manning ( 2019 ) have shown that Visual QA models lack some reasoning robustness but their approach is mostly linguistic and still static . Some other works also observe issues with static evaluation and proposed more dynamic benchmarks , e.g. , with a human-in-the-loop ( Khashabi et al. , 2021 ; Nie et al. , 2020 ; Li et al. , 2021 ; Sheng et al. , 2021 ) . We extend the line of research on Visual QA by reformulating it as an interactive two-agents system , and show that CLEVR under such a new setting is still an unsolved problem . Our work also differs from previous works in that it focuses on visual reasoning , is interactive and fully automatic . Adversarial perturbations . Szegedy et al . ( 2014 ) ; Goodfellow et al . ( 2015 ) ; Kurakin et al . ( 2016 ) ; Carlini & Wagner ( 2016 ) have introduced early methods that can ‘ fool ’ networks by performing tiny , visually imperceptible , perturbations of the input image . Moosavi-Dezfooli et al . ( 2017 ) have also shown universal and untargeted adversarial perturbations that are more transferable than previous approaches . Most methods perform white-box attacks , where an adversarial model has access to the target model ’ s parameters or gradients . In black-box attacks such access is removed and , e.g. , only output probabilities are available ( Guo et al. , 2019a ) . Other black-box attacks involve the use 1Changes that are consistent with the task constraints . of surrogates to estimate gradients of the real target ( Papernot et al. , 2017 ; Cheng et al. , 2019 ) or gradient-free methods ( Alzantot et al. , 2018 ) . More related to our work , Cheng et al . ( 2018 ) describe a black-box system , which only assumes access to outputs of the network . However , all the perturbations described above are continuous-based where individual pixels are changed independently . This leads to improbable images and assumes access to the sensory inputs of the neural network . In contrary , semantic perturbations ( Joshi et al. , 2019 ; Zeng et al. , 2019 ) , operate on semantically meaningful chunks of the input , leading to more plausible and physically interpretable perturbations . However , they still assume direct access to either images or ideally differentiable renderers . We can interpret our work within the adversarial perturbations framework , where the adversarial model can change the original scene by its manipulations , with zero access to the target model , including its sensory information and not limited to tiny changes but instead to semantically and globally meaningless modifications . Reinforcement learning . The game between the Visual-QA Player and Adversarial Player is a zero-sum two-player game . Conneau et al . ( 2017 ) use a conceptually similar two-player game to find a better word alignment between two languages without parallel corpus . Inspired by such a problem formulation , we consider our two-player game as a tool to achieve better reasoning models . We optimize the Adversarial Player with Advantage Actor-Critic ( A2C ) algorithm ( Sutton & Barto , 2018 ; Degris et al. , 2012 ) . Probing and measuring intelligence . Turing Test ( Turing , 2009 ) is considered to be among the earliest works on measuring the intelligence of artificial systems . Crucially to us , it is also formulated as a two-player game with an interrogation protocol . Legg ( 2008 ) systematizes and relates the notion of intelligence from different fields with the main focus on ‘ universality ’ . Johnson et al . ( 2017a ) have transferred the notion of intelligence onto the visual ground and emphasized reasoning rather than universality . It seems there is no widely accepted dataset or definition that encompasses our intuitions about intelligence . However , there are a few recent directions showing the lack thereof , mainly highlighting bias amplification or absence of mathematical capabilities ( Hendricks et al. , 2018 ; Bhardwaj et al. , 2021 ; Piekos et al. , 2021 ) . 3 PRELIMINARIES . In this section , we explain briefly how the CLEVR dataset ( Johnson et al. , 2017a ) is constructed and introduce our notation and definitions . CLEVR is a synthetic visual question answering dataset introduced by Johnson et al . ( 2017a ) , which consists of about 700k training and 150k validation image-question-answer triplets . Images are artificially constructed and rendered from scene graphs – a special structure containing information about object attributes such as position or color . Such a scene graph is also used to synthesize the ground-truth question-answer pairs by expanding templates according to the depth-first-search ordering . Ambiguous scenes are rejected . Each image represents an isometric view of the scene containing from two to ten objects . There are three classes of objects , spheres , cubes and cylinders . Each object can also be either large or small and has one color out of four ( brown , purple , cyan , yellow ) . It can also be either metallic or rubber-made . Every object has x and y coordinates that are confined within the ( −3 , +3 ) range . We use the same generation process to render modified scenes . CLEVR models . Various models have been introduced to work with the CLEVR dataset , some even ‘ solving ’ the dataset by achieving near perfect performance . Despite the strong offline performance , we test if those models ’ performance perpetuates in the more interactive setting where configurations of the scene could be changed . Whenever possible , we use pre-trained CLEVR models . Otherwise , we train the remaining models from scratch by making sure we achieve results similar to published accuracy numbers on the validation set . We summarize all the models in Table 1 . We show the accuracy on the CLEVR dataset ( Accuracy ) , indicate if an architecture is trained from scratch ( Re-trained ) , briefly describe how multi-modal fusion and reasoning is conducted ( Reasoning Mechanism ) , and indicate any extra privileged information required during the training process ( Extra ) . For instance , some models require extra access to functional programs used during the dataset generation , use scene graphs as a supervisory signal ( states ) , or always operate on scene graphs ( input-states ) . Otherwise , the models were trained only from image-question-answer triples . Mini-games . We formulate our problem as a Game between two players , Visual-QA Player and Adversarial Player . The Visual-QA Player takes as input question-image pairs and provide answers to such questions . Some models use states ( scene-graphs ) that replace images or require programs ( Johnson et al. , 2017a ) . The whole game consists of all CLEVR data points . For our purpose , we extend the notion of the Game into Mini-games . The rules of Mini-games are identical to the whole Game . The only difference is that each Mini-game operates on a subset of the CLEVR dataset . We define the size of a Mini-game by the number of datapoints that are attached to that Minigame . We sample data points for each Mini-game randomly and mutually exclusively . Mini-games have analogies in the adversarial perturbations literature . Mini-games of size one resemble perimage adversarial perturbations ( Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ) whereas a Mini-game that has all data pointsis similar to universal adversarial perturbations ( Moosavi-Dezfooli et al. , 2017 ) . In this work , we investigate various Mini-game sizes but due to the sheer scale we were unable to use the whole game as the Mini-game . Larger Mini-games make the optimization process more difficult as the domain where the Adversarial Player needs to operate increases . The training is also much more time-consuming and a sequential process . Instead , we can train multiple players on different Mini-games independently and thus massively . We leave the arduous training of the universal Adversarial Player on the whole Game as a possible future direction .

Summary: The paper proposes a black-box adversarial attack on models trained on the CLEVR dataset. CLEVR dataset is widely considered solved, especially with program synthesis-based models which seem to achieve near-perfect accuracy on CLEVR with a fairly small amount of training examples. Moreover, due to their synthetic and limited nature, it can be reasonably assumed that they would be able to answer *any* variation of CLEVR scenes within the bounds of the CLEVR-universe (Barring intentionally OOD test sets such as CoGENT). However, this paper shows that it is possible to fool all classes of the CLEVR model using an adversarial player that simply re-configures the scene (which should have the same answer as before). While the drops inaccuracy is not drastic, it is still an interesting finding, especially for program synthesis-driven methods. The paper also ends with an experiment about whether a purely data-driven approach can ever learn to be fully "robust".