Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests

1 INTRODUCTION . Psychologists have employed mental rotation tests for decades ( Shepard & Metzler , 1971 ) as a powerful tool for devising how the human mind interprets and ( internally ) manipulates three dimensional representations of the world . Instead of using these test to probe the human capacity for mental 3D manipulation , we are interested here in : a ) understanding the ability of modern deep neural architectures to perform mental rotation tasks , and b ) building architectures better suited to 3D inference and understanding . Recent applications of concepts from 3D graphics to deep learning , and vice versa , have led to promising results . We are similarly interested in leveraging models of 3D image formation from the graphics and vision communities to augment neural network architectures with inductive biases that improve their ability to reason about the real world . Here we measure the effectiveness of adding such biases , confirming their ability to improve the performance of neural models on mental rotation tasks . Concepts from inverse graphics can be used to guide the construction of neural architectures designed to perform tasks related to the reverse of the traditional image synthesis processes : namely , taking 2D image input and inferring 3D information about the scene . For instance , 3D reconstruction in computer vision ( Furukawa & Hernández , 2015 ) can be realized with neural-based approaches that output voxel ( Wu et al. , 2016 ; Nguyen-Phuoc et al. , 2019 ) , mesh ( Wang et al. , 2018 ) , or point cloud ( Qi et al. , 2017 ) representations of the underlying 3D scene geometry . Such inverse graphics methods range from fully-differentiable graphics pipelines ( Kato et al. , 2018 ) to implicit neural-based approaches with learnable modules designed to mimic the structure of certain components of the forward graphics pipeline ( Yao et al. , 2018 ; Thies et al. , 2019 ) . While inverse rendering is potentially an interesting and useful goal in itself , many computer vision systems could benefit from neural architectures that demonstrate good performance for more targeted mental rotation tasks . In our work here we are interested in exploring neural “ mental rotation ” by adapting a well known standard benchmark for visual question-and-answering ( VQA ) through answering questions with respect to another viewpoint . We use the the Compositional Language and Elementary Visual Reasoning ( CLEVR ) Diagnostic Dataset ( Johnson et al. , 2017 ) as the starting point for our work . While we focus on this well known benchmark , many analogous questions of practical interest exist . For example , given the camera viewpoint of a ( blind ) person crossing the road , can we infer if each of the drivers of the cars at an intersection can see this blind person crossing the street ? As humans , we are endowed with the ability to reason about scenes and imagine them from different viewpoints , even if we have only seen them from one perspective . As noted by others , it therefore seems intuitive that we should encourage the same capabilities in deep neural networks ( Harley et al. , 2019 ) . In order to answer such questions effectively , some sort of representation encoding 3D information seems necessary to permit inferences to be drawn due to a change in the orientation and position of the viewpoint camera . However , humans clearly do not have access to error signals obtained through re-rendering scenes , but are able to perform such tasks . To explore these problems in a controlled setting , we adapt the original CLEVR setup in which a VQA model is trained to answer different types of questions about a scene consisting of various types and colours of objects . While images from this dataset are generated through the rendering of randomly generated 3D scenes , the three-dimensional structure of the scene is never fully exploited because the viewpoint camera never changes . We call our problem formulation and data set CLEVR-MRT , as it is a new Mental Rotation Test version of the CLEVR problem setup . In CLEVR-MRT alternative views of a scene are rendered and used as the input to a perception pipeline that must then answer a question that was posed with respect to another ( the original CLEVR ) viewpoint.1 This gives rise to a more difficult task where the VQA model must learn how to map from its current viewpoint to the viewpoint that is required to answer the question . This can be seen in Figure 1 ( b ) . Figure 1 ( a ) depicts a real world situation and analogy where the answers to similar types of questions may help different types of systems make consequential decisions , e.g . intelligent intersections , cars , robots , or navigation assistants for the blind . The fact that MRTs are a classical tool of Psychology and the link to these different practical applications motivated us to create the controlled setting of CLEVR-MRTs depicted in Figure 1 ( b ) . ( a ) ( Left ) A view of a street corner . ( Middle ) a CLEVR-like representation of the scene with abstractions of buildings , cars and pedestrians . ( Right ) The same virtual scene from another viewpoint , where questions concerning the relative positions of objects after a mental rotation could be of significant practical interest . ( b ) Random views of an example scene in CLEVR-MRT . The center image is the ‘ canonical ’ view , which is the unseen point of view for which questions must be answered using only one of the other views as input . Figure 1 : ( a ) A real-world example where the ability to perform mental rotations can be of practical utility . ( b ) Images from the CLEVR-MRT dataset . Using our new mental rotation task definition and our CLEVR-MRT dataset , we examine a number of new inverse-graphics inspired neural architectures . We examine models that use the FILM ( Feature-wise Linear Modulation ) technique ( Perez et al. , 2017 ) for VQA , which delivers competitive performance using contemporary state-of-the-art convolutional network techniques . We observe that such methods fall short for this more challenging MRT VQA setting . This motivates us to create new architectures that involve inferring a latent feature volume that we subject to rigid 3D transformations ( rotations and translations ) , in a manner that has been examined in 3D generative modelling techniques such as spatial transformers ( Jaderberg et al. , 2015 ) as well as HoloGAN 1Dataset and code will be available at https : //github.com/anonymouscat2434/clevr-mrt ( Nguyen-Phuoc et al. , 2019 ) . This can either be done through the adaptation of a pre-trained 2D encoder network , i.e . an ImageNet-based feature extractor as in Section 2.2.1 , or through training our encoder proposed here , which is obtained through the use of contrastive learning as in Section 2.2.3 . In the case of the latter model , we leverage the InfoNCE loss ( Oord et al. , 2018 ) to minimise the distance between different views of the same scene in metric space , and conversely the opposite for views of different scenes altogether . However , rather than simply using a stochastic ( 2D ) data augmentation policy to create positive pairs ( e.g . random crops , resizes , and pixel perturbations ) , we leverage the fact that we have access to many views of each scene at training time and that this can be seen as a data augmentation policy operating in 3D . This in turn can be leveraged to learn an encoder that can map 2D views to a 3D latent space without assuming any extra guidance such as camera extrinsics . 2 METHODS . We begin here by describing simple and strong baseline methods as well as upper bound estimates used to evaluate the performance of different techniques on this dataset . We then present our new approach to learning 3D features and two different ways to address this task . 2.1 FILM BASELINES . The architecture we use is very similar to that proposed by FILM ( Perez et al. , 2017 ) , in which a pre-trained ResNet-101 classifier on ImageNet extracts features from the input images which are then fed to a succession of FILM-modulated residual blocks using the hidden state output from the GRU . As a sanity check – to ensure our models are adequately parameterised – the simplest baseline to run is one where images in the dataset are filtered to only contain canonical views . In this setting , we would expect the highest validation performance since the viewpoint given is the same as the canonical viewpoint ( the viewpoint wrt which the question must be answered ) . The second and third baselines to run are ones where we use the full dataset , with and without conditioning on the viewpoint camera via FILM , respectively . This is illustrated in Figure 2 , where we can see the viewpoint camera also being embedded before being concatenated to the question embedding and passed through the subsequent FILM blocks . Note that in the case of the canonical baseline , the viewpoint and canonical view is the same thing , so no camera conditioning is necessary . If we let S denote a scene consisting of all of its camera views ( images ) X , the camera c , the question q , and its corresponding answer y , we can summarise this as the following : S = ( X , q , c , y ) ∼ D ( sample a scene ) x ∼ X ( sample a random view ) h : = ResNet ( x ) ecam : = embed ( film ) φ ( c ) egru : = GRUφ ( q ) ỹ : = FILMφ ( h , [ egru , ecam ] ) ` cls : = ` ( y , ỹ ) , ( 1 ) where the encoder ( a ResNet ) is frozen and we do not update its parameters during training . 2.2 LEARNING 3D FEATURE REPRESENTATIONS FROM SINGLE VIEW IMAGES . So far we have been operating in 2D , based on the pre-trained ResNet-101 ImageNet encoder which outputs a high-dimensional stack of feature maps ( a 3D tensor ) . To work in 3D , we would either need to somehow transform the existing encoding into a 4D tensor ( a stack of 3D feature cubes ) or use a completely different encoder altogether which can output a 3D volume directly . Assuming we already had such a volume , we can manipulate the volume in 3D space directly by having it undergo any rigid transformation that is necessary for the question to be answered . In Section 2.2.1 we illustrate a simple technique which simply takes the existing ImageNet encoder ’ s features and runs it through a learnable ‘ post-processing ’ block to yield a 3D volume , and in Section 2.2.3 we propose a self-supervised contrastive approach to learn such an encoder from scratch . 2.2.1 PROJECTING 2D FEATURES INTO 3D FEATURES . To exploit the power of pre-trained representations , we start with a pre-trained ResNet encoder and transform its stack of feature maps through an additional set of 2D convolution blocks using the ‘ post-processor ’ shown in Figure 3 , right before reshaping the 3D tensor into 4D . In other words , we learn a module which maps from a stack of feature maps h to a stack of feature cubes h′ . Since the post-processor is a learnable module through which the FILM part of the pipeline is able to backpropagate through , it can be seen as learning an appropriate set of transforms that construct 3D feature volumes h′ . Through back-propagation it learns to perform well when manipulated with camera controllable FILM operations either as is or , more interestingly , when also subjected to rigid 3D transformations as we will see shortly in Section 2.2.2 .

Summary: The paper studies visual question answering focusing on answering questions in a reference image of a different viewpoint. They propose a new dataset CLEVR-MRT drawing motivation from the well-known visual reasoning dataset CLEVR to illustrate the idea in which they have full control of the changes of viewpoints in an image. They then propose to use a volumetric encoder to represent 3D image features of an image via either 2D-to-3D projection or a contrastive-based encoder and further adapt an existing method (FiLM) to handle 3D tensors. Experiments on the CLEVR-MRT show that the use of the 2D features and 3D features of an image is complementary to each other.