Measuring CLEVRness: Black-box Testing of Visual Reasoning Models

1 INTRODUCTION . Are our artificial intelligence systems capable of reasoning ? Or like Clever Hans , they use various cues only tangentially related to the task and rely on rote memorization with poor generalization ? ( Pfungst , 1911 ; Johnson et al. , 2017a ) This work revisits such a question and proposes an interactive framework with the communication channel between two players . The first player , which reasoning capabilities we are about to test , performs visual reasoning tasks , we call it VisualQA Player . The second player , which we call the Adversarial Player , is manipulating the scene so that it fools the first player even though those changes still lead to correct reasoning steps among humans . Both players interact with each other only through questions , answers and the visual scene as shown in Figure 1 . If the Adversarial Player manipulating the scene causes the Visual-QA Player to change its answer even though the new scene is still valid for the same question and answer , it is then the reasoning failure . It is similar to the following situation . Imagine a box is placed between two spheres . If you ask a question , is there a box between two spheres ? , the answer should be positive . Now , if we move the box anywhere so it does not cross any of the spheres , and ask the same question , the response should remain unchanged . In other words , we postulate that reasoning outputs of agents need to be invariant under scene configurations that are consistent with the questions-answer pairs . Moreover , in the spirit of generic adversarial attacks , we seek configurations that also pose little if any reasoning challenges for humans . We propose an automatic and agnostic pipeline to benchmark the reasoning capabilities of various models , only assuming they can communicate by answering questions about the scene . Due to the recent stream of research in vision-and-language ( Zhang et al. , 2021 ; Jiang et al. , 2020 ; Guo et al. , 2019b ; Su et al. , 2020 ; Wang et al. , 2020 ; Kamath et al. , 2021 ; Tan & Bansal , 2019 ; Chen et al. , 2020 ) , we believe there will be an increasing number of vision models that operate through language . Moreover , we also consider the visual question answering framework set-up as a two-player system as an excellent benchmarking pipeline . We perform all tests by scene manipulations and observing how a tested model behaves under such changes . The pipeline does not require any knowledge of the internals of the tested model . It also does not manipulate the sensory information of such a model , e.g. , pixels in the images , and all the manipulations are physically meaningful . Even though our current pipeline uses synthetic scenes as only those can easily be automatically manipulated , our results have also real-world ramifications . If models are susceptible to semantically meaningless changes1 in scene configurations , in a synthetic setting , there are valid concerns that real-world robots could also be prone to manipulation of objects in a room . Finally , our work also questions the possibility of training and benchmarking networks in a purely data-driven and offline , static manner . Contributions . The main contributions of our work could be summarized in three points . First , we propose a strong black-box adversarial test , which makes no assumptions about the underlying mechanics of a tested model , formulated as a game between two players . Our test does not require any direct access to the tested model , even through its sensory information . In particular , it does not require gradients , output probabilities , or any access to the perceived image . Our work also deviates from bounded perturbations and instead focuses on global scene manipulations that are still consistent with the task constraints , and can change the behavior of a tested model . Second , we reformulate visual reasoning by integrating visual question answering with zero-sum two-player game frameworks . Under our novel formulation , a visual and adversary agents compete against each other through content manipulation . We believe that this is an initial step towards more sophisticated frameworks that integrate computer vision with multi-agent systems . Third , we explore the limits of the data-driven approaches in synthetic visual scenarios , and demonstrate that current CLEVR models are lacking the efficiency to learn robust reasoning steps . 2 RELATED WORK . Our work touches upon various research directions , which we briefly describe here . Visual QA . Introduced as a visual counter-part of the Turing Test ( Malinowski & Fritz , 2014 ; Geman et al. , 2015 ) , it became a computer vision task that requires a holistic visual understanding . Many other works have extended the task to larger datasets or videos or adversarial train-test splits ( Antol et al. , 2015 ; Agrawal et al. , 2018 ; Tapaswi et al. , 2016 ; Yu et al. , 2019 ) . More recently , we also observe the generalization of the task to become a part of the vision-plus-language suite of benchmarks ( Lu et al. , 2019 ; Chen et al. , 2019 ; Wang et al. , 2021 ) . Johnson et al . ( 2017a ) have introduced a synthetic variant of the Visual QA problem that is more focused on reasoning and the control of the experimentation . Although it was shown that traditional methods ( Yang et al. , 2016 ) are not enough to solve that dataset , newer methods can pass the human baseline on this task . These methods are trained to operate on pixels and text directly ( Santoro et al. , 2017 ; Perez et al. , 2018 ; Hudson & Manning , 2018 ; Malinowski et al. , 2018 ; Kamath et al. , 2021 ) or they induce programs as an intermediate representation ( Johnson et al. , 2017b ; Mascharka et al. , 2018 ; Yi et al. , 2018 ) . Hudson & Manning ( 2019 ) have shown that Visual QA models lack some reasoning robustness but their approach is mostly linguistic and still static . Some other works also observe issues with static evaluation and proposed more dynamic benchmarks , e.g. , with a human-in-the-loop ( Khashabi et al. , 2021 ; Nie et al. , 2020 ; Li et al. , 2021 ; Sheng et al. , 2021 ) . We extend the line of research on Visual QA by reformulating it as an interactive two-agents system , and show that CLEVR under such a new setting is still an unsolved problem . Our work also differs from previous works in that it focuses on visual reasoning , is interactive and fully automatic . Adversarial perturbations . Szegedy et al . ( 2014 ) ; Goodfellow et al . ( 2015 ) ; Kurakin et al . ( 2016 ) ; Carlini & Wagner ( 2016 ) have introduced early methods that can ‘ fool ’ networks by performing tiny , visually imperceptible , perturbations of the input image . Moosavi-Dezfooli et al . ( 2017 ) have also shown universal and untargeted adversarial perturbations that are more transferable than previous approaches . Most methods perform white-box attacks , where an adversarial model has access to the target model ’ s parameters or gradients . In black-box attacks such access is removed and , e.g. , only output probabilities are available ( Guo et al. , 2019a ) . Other black-box attacks involve the use 1Changes that are consistent with the task constraints . of surrogates to estimate gradients of the real target ( Papernot et al. , 2017 ; Cheng et al. , 2019 ) or gradient-free methods ( Alzantot et al. , 2018 ) . More related to our work , Cheng et al . ( 2018 ) describe a black-box system , which only assumes access to outputs of the network . However , all the perturbations described above are continuous-based where individual pixels are changed independently . This leads to improbable images and assumes access to the sensory inputs of the neural network . In contrary , semantic perturbations ( Joshi et al. , 2019 ; Zeng et al. , 2019 ) , operate on semantically meaningful chunks of the input , leading to more plausible and physically interpretable perturbations . However , they still assume direct access to either images or ideally differentiable renderers . We can interpret our work within the adversarial perturbations framework , where the adversarial model can change the original scene by its manipulations , with zero access to the target model , including its sensory information and not limited to tiny changes but instead to semantically and globally meaningless modifications . Reinforcement learning . The game between the Visual-QA Player and Adversarial Player is a zero-sum two-player game . Conneau et al . ( 2017 ) use a conceptually similar two-player game to find a better word alignment between two languages without parallel corpus . Inspired by such a problem formulation , we consider our two-player game as a tool to achieve better reasoning models . We optimize the Adversarial Player with Advantage Actor-Critic ( A2C ) algorithm ( Sutton & Barto , 2018 ; Degris et al. , 2012 ) . Probing and measuring intelligence . Turing Test ( Turing , 2009 ) is considered to be among the earliest works on measuring the intelligence of artificial systems . Crucially to us , it is also formulated as a two-player game with an interrogation protocol . Legg ( 2008 ) systematizes and relates the notion of intelligence from different fields with the main focus on ‘ universality ’ . Johnson et al . ( 2017a ) have transferred the notion of intelligence onto the visual ground and emphasized reasoning rather than universality . It seems there is no widely accepted dataset or definition that encompasses our intuitions about intelligence . However , there are a few recent directions showing the lack thereof , mainly highlighting bias amplification or absence of mathematical capabilities ( Hendricks et al. , 2018 ; Bhardwaj et al. , 2021 ; Piekos et al. , 2021 ) . 3 PRELIMINARIES . In this section , we explain briefly how the CLEVR dataset ( Johnson et al. , 2017a ) is constructed and introduce our notation and definitions . CLEVR is a synthetic visual question answering dataset introduced by Johnson et al . ( 2017a ) , which consists of about 700k training and 150k validation image-question-answer triplets . Images are artificially constructed and rendered from scene graphs – a special structure containing information about object attributes such as position or color . Such a scene graph is also used to synthesize the ground-truth question-answer pairs by expanding templates according to the depth-first-search ordering . Ambiguous scenes are rejected . Each image represents an isometric view of the scene containing from two to ten objects . There are three classes of objects , spheres , cubes and cylinders . Each object can also be either large or small and has one color out of four ( brown , purple , cyan , yellow ) . It can also be either metallic or rubber-made . Every object has x and y coordinates that are confined within the ( −3 , +3 ) range . We use the same generation process to render modified scenes . CLEVR models . Various models have been introduced to work with the CLEVR dataset , some even ‘ solving ’ the dataset by achieving near perfect performance . Despite the strong offline performance , we test if those models ’ performance perpetuates in the more interactive setting where configurations of the scene could be changed . Whenever possible , we use pre-trained CLEVR models . Otherwise , we train the remaining models from scratch by making sure we achieve results similar to published accuracy numbers on the validation set . We summarize all the models in Table 1 . We show the accuracy on the CLEVR dataset ( Accuracy ) , indicate if an architecture is trained from scratch ( Re-trained ) , briefly describe how multi-modal fusion and reasoning is conducted ( Reasoning Mechanism ) , and indicate any extra privileged information required during the training process ( Extra ) . For instance , some models require extra access to functional programs used during the dataset generation , use scene graphs as a supervisory signal ( states ) , or always operate on scene graphs ( input-states ) . Otherwise , the models were trained only from image-question-answer triples . Mini-games . We formulate our problem as a Game between two players , Visual-QA Player and Adversarial Player . The Visual-QA Player takes as input question-image pairs and provide answers to such questions . Some models use states ( scene-graphs ) that replace images or require programs ( Johnson et al. , 2017a ) . The whole game consists of all CLEVR data points . For our purpose , we extend the notion of the Game into Mini-games . The rules of Mini-games are identical to the whole Game . The only difference is that each Mini-game operates on a subset of the CLEVR dataset . We define the size of a Mini-game by the number of datapoints that are attached to that Minigame . We sample data points for each Mini-game randomly and mutually exclusively . Mini-games have analogies in the adversarial perturbations literature . Mini-games of size one resemble perimage adversarial perturbations ( Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ) whereas a Mini-game that has all data pointsis similar to universal adversarial perturbations ( Moosavi-Dezfooli et al. , 2017 ) . In this work , we investigate various Mini-game sizes but due to the sheer scale we were unable to use the whole game as the Mini-game . Larger Mini-games make the optimization process more difficult as the domain where the Adversarial Player needs to operate increases . The training is also much more time-consuming and a sequential process . Instead , we can train multiple players on different Mini-games independently and thus massively . We leave the arduous training of the universal Adversarial Player on the whole Game as a possible future direction .

Summary: This paper proposes to measure the reasoning capabilities of VQA models by training an "adversary player" that can manipulate the image and fool the evaluated models. Given a sample, the "adversary player" moves objects such that the groundtruth answer remains valid and the scene is in-distribution. If the evaluated VQA model produces a different output, the "adversary player" receives a positive reward and is then optimized with a reinforcement learning algorithm.