Probabilistic Neural-Symbolic Models for Interpretable Visual Question Answering

1 INTRODUCTION . Exciting progress has been made in recent years in deep representation learning ( Hochreiter & Schmidhuber , 1997 ; LeCun et al. , 1998 ) , which has led to advances in artificial intelligence tasks such as image recognition ( Krizhevsky et al. , 2012 ; He et al. , 2015 ) , machine translation ( Sutskever et al. , 2014 ) , visual question answering ( Agrawal et al. , 2015 ) , visual dialog ( Das et al. , 2016 ) , and reinforcement learning ( Mnih et al. , 2015 ) , etc . While deep neural networks achieve such impressive performances , many aspects of human cognition such as compositional generalization and reasoning are harder to model with state-of-the art deep learning approaches ( Battaglia et al. , 2018 ; Barrett et al. , 2018 ) . Symbolic approaches ( Newell & Simon , 1976 ) on the other hand provide strong compositional and reasoning capabilities which are challenging to model with neural networks ( Lake et al. , 2017 ) . Consequently , a rich line of work in neuro-symbolic processing sought to build AI systems with both strong learning and reasoning capabilities ( Valiant , 2003 ; Evans et al. , 2018b ; Bader & Hitzler , 2005 ) . As we tackle higher-level tasks which involve reasoning , such as visual question answering ( Agrawal et al. , 2015 ) , planning ( Andreas et al. , 2016b ; Gregor et al. , 2016 ) etc. , it is natural to desire the ability to provide instructions to models to guide them . Symbolic instructions , by their very nature are easier to specify and more interpretable than specifying the parameters of a neural network . For such high-level tasks , a sensible approach is to specify “ what ” to do using symbols and learn how to do the task using modern representation learning techniques . For the example shown in Figure 1 , given a question “ Is a square to the left of a green shape ? ” , one can ask the model to reason about the answer by first applying a find [ green ] operator , then find [ left ] , then And the result together with a find [ square ] operator in order to predict the answer , in other words specifying “ what ” are the computations we desire to be executed , in the form of a “ program ” . ( Johnson et al. , 2017 ; Hu et al. , 2017 ; Andreas et al. , 2016a ) . The network can then learn “ how ” to execute such a program from data using deep representation learning ( Johnson et al. , 2017 ; Hu et al. , 2017 ; Andreas et al. , 2016a ) . This paper addresses a very natural desiderata for such neuro-symbolic models , in the context of visual question answering – can we retain the intepretability of “ what the network did ” expressed in terms of the syntax and lexicon that we are interested in , while specifying minimal teaching examples of “ what to do ” given an input question ? We propose to approach this problem with a treatment of programs z as a latent variable , embedded into a generative model of questions x and answers a given images i as visualized in Figure 2 . This model class has certain desirable properties : first , a proper treatment of z as a stochastic latent variable is helpful especially in the presence of a limited number of expert human instructions , as we model the uncertainty associated with z explicitly , leading to more interepretable latent spaces ( Section 5 ) ; second , the model makes intuitive independence assumptions with regards to the question answering task – conditioned on the program z , questions x are independent of the answers a for an image , meaning that the program must be a sufficient statistic for predicting the answer given a question1 . Third , we parameterize the p ( a|i , z ) term , which predicts an answer given an image and a program using a popular , related class of question answering models called neural module networks ( NMNs ) ( Hu et al. , 2017 ; Johnson et al. , 2017 ; Andreas et al. , 2016a ) which previous work has shown leads to better fit between the program ( or “ what the model is doing ” ) and the execution ( or “ how it is doing what it is doing ” ) . Given the close ties of our model to NMNs , we will refer to our model as variational neural module networks ( V-NMN ) for the rest of the paper . Apart from our modeling contribution of a probabilistic latent variable , neuro-symbolic model , our key technical contribution is to show how to train this model in the context of visual question answering . Optimizing such generative models with structured discrete latent variables is an intractable problem to solve in genereal – for this particular case , we formulate training into stages of grounding questions into programs ( question coding ) , learning how to execute them and then with those initializations train the full variational objective . We demonstrate that this stage-wise optimization allows us to successfully learn a probabilistic neural-symbolic model that can answer compositional questions about shapes in an image ( Andreas et al. , 2016a ) . While the V-NMN model is instantiated for the specific problem of visual question answering , we believe the tools and techniques we develop are more generally applicable to probabilistic , structured , discrete , interpretable latent spaces for interpretable neuro-symbolic models for planning , instruction following , grounding referring expressions etc . We benchmark our model on the SHAPES ( Andreas et al. , 2016a ) dataset with compositionally novel questions and show that our model is able to provide interpretable explanations of “ what it is doing ” ( measured by program correctness ) for unseen questions even in the setting where it is given as few as ≈ 20 paired examples of programs and corresponding questions from the dataset , while also getting to high question answering accuracy . We find that the proposed approach outperforms a state of the art neural-symbolic model designed for VQA from Johnson et al . ( 2017 ) , as well as deterministic ablations of our model . 2 A PROBABILISTIC NEURAL-SYMBOLIC MODEL FOR VQA . For a standard neural-symbolic NMN approach to question answering ( Johnson et al. , 2017 ) , we are given an input image i and a question x = ( x1 , · · · , xN ) as a sequence of words xi each encoded using a vocabulary Vx , and must produce answers a by planning and executing a series of modular 1 Note that this model assumes independence of programs from images , which corresponds to the weak sampling assumptions in concept learning ( Tenenbaum , 1999 ) , one can handle question premise , ie . that people might ask a specific set of questions for an image in such a model by reparameterizing the answer variable to include a relevance label . computations ( z = ( z1 , · · · , zN ) ) – i.e . a program of operations ( zi ∈ Vz ) to be executed in order to answer the question . Note that z here is the prefix serialization of the program and programs themselves often have tree-structured topologies . We denote the dataset of ( i , x , a ) triplets as D. Additionally , we are provided programs z for some input questions ( x ∈ S ) , which specify the notion of grounding desired by an end user . Our goal is to provide faithful explanations on unseen questions ( along with correct answers ) consistent with the specified grounding . We accomplish this by modeling z as a latent variable that underlies both question and answer generation given an input image . Intuitively , we expect this will enable us to learn a more meaningful latent representation for questions , as modeling uncertainity should help better handle the potentially underspecified mapping between questions and programs provided by the user . As such , our model factorizes p ( x , z , a|i ) = p ( x|z ) p ( a|i , z ) p ( z ) as in the graphical model shown in Figure 2 , where the prior p ( z ) is learnt by training a neural sequence model on valid strings simulated from known syntax . Conceptually , this factorization captures the notion that many differently-worded questions are reducible to identical underlying semantics ( captured by a program ) , making the program space , in a sense , a compression of the question space . As mentioned in Section 1 the model also captures the independence of questions from answers for an image given programs , which is also an intuitive desiderata for interpretable neural-symbolic models2 . Given the proposed model shown in Figure 2 , we must address two tasks to put the model to use for VQA – learning model parameters , and performing inference for a specific query . Learning . We will always condition on images i but may not observe underlying programs z , so our per-data point training objective is the log-marginal-conditional-likelihood : log p ( x , a | i ) . Since computing this objective involves an intractable summation over latent programs ( log ∑ z p ( xj , z , aj | ij ) ) , we use a variational approximation ( Jordan et al. , 1999 ) . Specifically , let - qφ ( z | x ) denote an amortized inference neural network parameterized by φ , which instantiates a sequence to sequence LSTM ( Hochreiter & Schmidhuber , 1997 ) model - pθz ( a | z , i ) be a NMN parametrized by θz run on image i with program z , - pθ ( x | z ) be a recurrent network ( sequence to sequence model ) parametrized by θ that maps programs to questions , and - p ( z ) be a prior over programs learned from data or based on program syntax directly , parameter- ized as an LSTM ( Hochreiter & Schmidhuber , 1997 ) sequence model Then , for datapoints without program supervision , the objective is : elbo ( a , x|i ) = Ez∼qφ ( z|x ) [ log pθz ( a|z , i ) + log pθ ( x|z ) ] − βKL ( qφ ( z|x ) , p ( z ) ) It is easy to show that elbo ( x , a|i ) ≤ log p ( x , a|i ) for β ≥ 1 ; as such , maximizing this objective is a form of approximate maximum likelihood training for distribution parameters θz , θ , and φ. Additionally , for data points with annotations , we add the likelihood of the program given the question log rφ ( z|x ) to the objective . Note that r shares parameters φ with the inference network q . So our overall objective becomes : elbo ( a , x|i ) + α log r ( z|x ) · [ [ x ∈ S ] ] ( 1 ) where the indicator function [ [ x ∈ S ] ] is 1 if x is in the supervised set and 0 otherwise and α is a 2This view also matches finding from Hu et al . ( 2017 ) who find models linking questions to answer ( via attention ) operating in low-program supervision regimes often have syntactically poor programs despite leading to correct answers – i.e . the program need not be a sufficient statistic anymore to answer the question . positive scaling factor . The goal is to learn the desired association of questions with programs from the second term , and to propagate them to other questions using the first term . Inference . At test time , we can answer questions for images by computing p ( a|i , x ) =∑ z p ( a|i , z ) p ( z|x ) using an approximation pq ( a|i , x ) ≈ Ezi∼q ( z|x ) p ( a|i , zi ) . Challenges . While the elbo term in Equation ( 1 ) appears similar to a standard ELBO for joint latent variable models ( Vedantam et al. , 2018 ; Wang et al. , 2016b ; Suzuki et al. , 2017 ) , there are a few important departures : - Firstly , the latent variable z is not a Gaussian or categorical random variable ( which admit straightforward reparameterizations ( Maddison et al. , 2017 ; Jang et al. , 2016 ) ) but rather a sequence of discrete tokens , which complicates the optimization problem in Equation ( 1 ) . - Secondly , the conditional log-likelihood for a is parameterized by a NMN that is dynamically assembled based on sample programs drawn from q ( z|x ) . Depending upon the uncertainty of the program posterior , program tokens may not be consistently grounded across examples , making learning the appropriate computation for the corresponding neural modules difficult . - Lastly , our observations x are explained by powerful “ autoregressive ” sequence models . Learning meaningful latent variable models that are informative of observations for such models is difficult ( c.f . ( Bowman et al. , 2016 ; Chen et al. , 2016 ; Alemi et al. , 2018 ) ) as powerful decoder models can explain the data using stochasticity from sampling ( referred to as autodecoding in Alemi et al . ( 2018 ) ) . Consequently , it also becomes hard to learn meaningful NMN parameters from the previous step . In the following sections , we present a training regime and adjustments to the objective in Equation ( 1 ) that make optimization more tractable and enable training NMNs with significantly reduced questionaligned program supervision . While instantiated for question answering , it is not difficult to conceive versions of this regime for other problems relevant to probabilistic neural-symbolic models .

Summary: This paper proposes a discrete, structured latent variable model for visual question answering that involves compositional generalization and reasoning. In comparison to the existing approach, this paper well addressed the challenge of learning discrete latent variables in the presence of uncertainty. The results show a significant gain in performance as well as the capability of the model to generalize composition program to unseen data effectively. The qualitative analysis shows that the proposed model not only get the correct answer but also the correct behavior that leads to the answer.