Visual question answering with modules and language modeling

The primary focus in this thesis is to learn modularized representations for the task of Visual Question Answering. Learning such representations holds the potential to generalize to higher order reasoning as is prevalent in human beings. Chapter 1 discusses the literature related to VQA, modular networks and neural structure optimization. In particular, it first details different datasets proposed to study this task. The models for VQA can be categorized into two categories based on the datasets they are suitable for. The first one is open-ended questions about natural images. These questions are mostly about a few objects/persons present in the image and don’t require any significant reasoning capability to answer them. The second category comprises of questions (mostly on synthetic images) which tests the ability of models to perform compositional reasoning. We discuss the different architectural variants of Neural Module Networks (NMN). Finally, we discuss approaches to learn the neural network structures or modules for tasks other than VQA. In Chapter 2, we discuss a way to sparsely execute a CNN model (ResNeXt [110]) and save computations in the process. Here, we used a mixture of experts formulation to execute only the top-K experts in each convolutional block. The most important set of experts are selected based on a gate controller which uses a question-guided attention map followed by fully-connected layers to assign weights to the set of experts. Our experiments show that it is possible to get huge savings in the FLOP count with only a minimal degradation in performance. Chapter 3 is a prologue to Chapter 4. It mentions the key contributions and provides an introduction to the research problem which we try to address in the article. Chapter 4 contains the contents of the article. Here, we are interested in learning the internal structure of the modules for Neural Module Networks (NMN) [3, 37]. We introduce a novel form of module structure which uses elementary arithmetic operations and now the task is to learn the weights of these operations to form the module structure. We cast the problem into a bi-level optimization technique in which the model takes alternating gradient descent steps in architecture and weight spaces. Chapter 5 discusses additional experiments and ablation studies that were done in the context of the previous article. Most works in the literature use a recurrent neural network like LSTM [33] or GRU [13] to model the question features. However, LSTMs can fail to properly encode syntactic features of the question which could be vital to answering some VQA questions [87]. Recently, [76] has shown the utility of language modeling for question-answering. With this motivation, we try to learn a better language model which can be trained in an unsupervised manner. In Chapter 6, we discuss a recursive network for language modeling whose structure aligns with the natural language. More technically, we make use of an unsupervised parsing model (Parsing Reading Predict Network or PPRN [86]) and augment its prediction step with a TreeLSTM [99] model which makes use of the intermediate tree structure given by PRPN model to output a hidden state by utilizing the tree structure. The predict step of PRPN model makes use of a hidden state which is a weighted combination of the TreeLSTM’s hidden state and the one obtained from structured attention. This way it helps the model to do unsupervised parsing and also capture long-term dependencies as the structure now explicitly exists in the model. Our experiments demonstrate that this model leads to improvement on language modeling task over the PRPN baseline on Penn Treebank dataset.

Keyphrases: Visual Question Answering, Visual Reasoning, Modular Networks, Neural Structure Optimization, Language Modeling