Title: VISUAL REASONING BY PROGRESSIVE MODULE NETWORKS

Abstract: Humans learn to solve tasks of increasing complexity by building on top of previously acquired knowledge. Typically, there exists a natural progression in the tasks that we learn -most do not require completely independent solutions, but can be broken down into simpler subtasks. We propose to represent a solver for each task as a neural module that calls existing modules (solvers for simpler tasks) in a functional program-like manner. Lower modules are a black box to the calling module, and communicate only via a query and an output. Thus, a module for a new task learns to query existing modules and composes their outputs in order to produce its own output. Our model effectively combines previous skill-sets, does not suffer from forgetting, and is fully differentiable. We test our model in learning a set of visual reasoning tasks, and demonstrate improved performances in all tasks by learning progressively. By evaluating the reasoning process using human judges, we show that our model is more interpretable than an attention-based baseline.
We demonstrate PMN in learning a set of visual reasoning tasks such as counting, captioning, and Visual Question Answering (VQA). PMN outperform baselines without module composition on all tasks. We further analyze the interpretability of PMN's reasoning process with human judges.
Multi-task learning. The dominant approach to multi-task learning is to have a model that shares parameters in a soft (Duong et al., 2015; Yang & Hospedales, 2017) or hard way (Caruana, 1993) . Soft sharing refers to each task having independent weights that are constrained to be similar (e.g. via 2 regularization (Duong et al., 2015) , trace norm (Yang & Hospedales, 2017)) while hard sharing typically means that all tasks share the base network but have independent layers on the top (Kokkinos, 2017; Misra et al., 2016) . While sharing parameters helps to compute a task-agnostic representation that is not overfit to a specific task, tasks do not directly share information or help each other. Bilen & Vedaldi (2016) propose the Multinet architecture where tasks can interact with each other in addition to shared image features. Multinet solves one task at each time step and appends the encoded output of each task to existing data representation. A similar idea, Progressive Neural Networks (PNNs) (Rusu et al., 2016) use a new neural network for each task, but are designed to prevent catastrophic forgetting as they transfer knowledge from previous tasks by making lateral connections to representations of previously learned tasks. Recently, Wang et al. (2017) propose the VQA-Machine which exploits a set of existing algorithms to solve the VQA problem. Zamir et al. ( 2018) learn a computational taxonomy for task transfer learning on several vision problems. However, the major differences to this work are PMN's compositional modular structure, ability to directly query other modules, and the overall process of learning increasingly complex tasks. Module networks. Pioneering work in modular structure, NMN (Andreas et al., 2016; Hu et al., 2017) addresses VQA where questions have a compositional structure. Given an inventory of small networks, or modules, NMN produces a layout for assembling the modules for any question. PMN is different from NMN as it can be easily extended to new tasks using its compositional structure. It also treats each task as modules which opens up promising ways to train models that can compartmentalize and perform multiple tasks as they progressively improve their abilities. Visual question answering. VQA has seen great progress in recent years: improved multimodal pooling functions (Fukui et al., 2016; Kim et al., 2018 ), multi-hop attention (Yang et al., 2016), driving attention through both bottom-up and top-down schemes (Anderson et al., 2018) , and modeling attention between words and image regions recurrently (Hudson & Manning, 2018) are some of the important advances. There are also attempts to generate programs or sequence of modules automatically that yield a list of interpretable steps (Hu et al., 2017; Johnson et al., 2017b) using policy gradient optimization. Our approach treats visual reasoning as a compositional multi-task problem, and shows that using sub-tasks compositionally can help improve performance and interpretability.

1. INTRODUCTION:

Humans acquire skills and knowledge in a curriculum by building on top of previously acquired knowledge. For example, in school we first learn simple mathematical operations such as addition and multiplication before moving on to solving equations. Similarly, the ability to answer complex visual questions often requires the skills to understand attributes such as color, recognize a variety of objects, and be able to spatially relate them. Just like humans, machines may also benefit by sequentially learning tasks in progressive complexity and composing knowledge along the way.

The process of training a machine learning model to be able to solve multiple tasks, or multi-task learning (MTL), has been widely studied (Long et al., 2017; Ruder, 2017; Ruder et al., 2017; Rusu et al., 2016) . The dominant approach is to have a model that shares parameters (e.g., bottom layers of a CNN) with individualized prediction heads (Caruana, 1993; Long et al., 2017) . By sharing parameters, models are able to learn better task-agnostic data representations. However, the tasks are disconnected as their outputs are not combined to solve tasks of increasing complexity. It is desirable if one task can learn to process the predictions from other tasks thereby reaping the benefits of MTL.

In this paper, we address the problem of MTL where tasks exhibit a natural progression in complexity. We propose Progressive Module Networks (PMN), a framework for multi-task learning by progressively designing modules on top of existing modules. Each module is a neural network that can query modules for lower-level tasks, which in turn may query modules for even simpler tasks. The modules communicate by learning to query other modules and process their outputs, while the internal module processes are a blackbox. This is similar to a computer program that uses available libraries without having to know their internal operations. Parent modules can choose which lower-level modules they want to query via a soft gating mechanism. Examining the queries, replies, and choices a parent module makes, we can understand the reasoning behind the module's output.

PMN is related but different from Andreas et al. (2016) and Hu et al. (2017) . PMN's modules are task-level modules, and they are compositional, i.e. modules build on modules which build on modules. It allows efficient use of data by not needing to re-learn previously acquired knowledge. By learning selective information flow between modules, interpretability arises naturally. Green rectangles denote terminal modules, and yellow rectangles denote compositional modules. Blue arrows and red arrows represent calling and receiving outputs from submodules, respectively. White numbered circles denote computation order. For convenience, assume task levels correspond to the subscripts. Calling M3 invokes a chain of calls (blue arrows) to lower level modules which stop at the terminal modules.

3. PROGRESSIVE MODULE NETWORKS:

Most complex reasoning tasks can be broken down into a series of sequential reasoning steps. We hypothesize that there exists a hierarchy with regards to complexity and order of execution: high level tasks (e.g. counting) are more complex and benefit from leveraging outputs from lower level tasks (e.g. classification). For any task, Progressive Module Networks (PMN) learn a module that requests and uses outputs from lower modules to aid in solving the given task. This process is compositional, i.e., lower-level modules may call modules at an even lower level. Solving a task corresponds to executing a directed acyclic computation graph where each node represents a module (see Fig. 1 ).

PMN has a plug-and-play architecture where modules can be replaced by their improved versions. This opens up promising ways to train intelligent robots that can compartmentalize and perform multiple tasks while progressively learning from experience and improving abilities. PMN also chooses which lower level modules to use through a soft-gating mechanism. A natural consequence of PMN's modularity and gating mechanism is interpretability. While we do not need to know the internal workings of modules, we can examine the queries and replies along with the information about which modules were used to reason about why the parent module produced a certain output.

Formally, given a task n at level i, the task module M n can query other modules M k at level j such that j < i. Each module is designed to solve a particular task (output its best prediction) given an input and environment E. Note that E is accessible to every module and represents a broader set of "sensory" information available to the model. For example, E may contain visual information such as an image, and text in the form of words (i.e., question). PMN has two types of modules: (i) terminal modules execute the simplest tasks that do not require information from other modules (Sec. 3.1); and (ii) compositional modules that learn to efficiently communicate and exploit lower-level modules to solve a task (Sec. 3.2). We describe the tasks studied in this paper in Sec. 3.3 and provide a detailed example of how PMN is implemented and executed for VQA (Sec. 3.4).

3.1. TERMINAL MODULES:

Terminal modules are by definition at the lowest level 0. They are analogous to base cases in a recursive function. Given an input query q, a terminal module M generates an output o = M (q), where M is implemented with a neural network. A typical example of a terminal module is an object classifier that takes as input a visual descriptor q, and predicts the object label o.

3.2. COMPOSITIONAL MODULES:

A compositional module M n makes a sequence of calls to lower level modules which in turn make calls to their children, in a manner similar to depth-first search (see Fig. 1 ). We denote the list of modules that M n is allowed to call by

L n = [M m , . . . , M l ].

Every module in L n has level lower than M n . Since lower modules need not be sufficient in fully solving the new task, we optionally include a terminal module ∆ n that performs "residual" reasoning. Also, many tasks require an attention mechanism to focus on certain parts of data. We denote Ω n as a terminal module that performs such soft-attention. ∆ n and Ω n are optionally inserted to the list L n and treated as any other module.

The compositional aspect of PMN means that modules in L n can have their own hierarchy of calls.

We make L n an ordered list, where calls are being made in a sequential order, starting with the first in the list. This way, information produced by earlier modules can be used when generating the query for the next. For example, if one module is performing object detection, we may want to use its output (bounding box proposals), for querying other modules such as an attribute classifier.

For this work, the list L n , and thus the levels of tasks, are determined by hand. Relaxing this and letting the model learn the task hierarchy itself is a challenging direction that we leave for future work. Also, notice that the number of back-and-forth communications increases exponentially if each module makes use of every lower-level module. Thus, in practice we restrict the list L n to those lower-level modules that may intuitively be needed by the task. We emphasize that M n still (softly) chooses between them, and thus the expert intervention only removes the lower-level modules that are uninformative to the task.

Our compositional module M n runs (pre-determined) T n passes over the list L n . It keeps track of a state variable s t at time step t ≤ T n . This contains useful information obtained by querying other modules. For example, s t can be the hidden state of a Recurrent Neural Network. Each time step corresponds to executing every module in L n and updating the state variable. We describe the module components below, and Algorithm 1 shows how the computation is performed. An example implementation of the components and demonstration of how they are used is detailed in Sec. 3.4. State initializer. Given a query (input) q n , the initial state s 1 is produced using a state initializer I n . Importance function. For each module M k (and ∆ n , Ω n ) in L n , we compute an importance score g k n with G n (s t ). The purpose of g k n is to enable M n to (softly) choose which modules to use. This also enables training all module components with backpropagation. Notice that g k n is input dependent, and thus the module M n can effectively control which lower-level module outputs to use in state s t . Here, G n can be implemented as an MLP followed by either a softmax over submodules, or a sigmoid that outputs a score for each submodule. However, note that the proposed setup can be modified to adopt hard-gating mechanism using a threshold or sampling with reinforcement learning. 

q k = Q n→k (s t , V, Gn(s t ))

produce query for M k 8: Query transmitter and receiver. A query for module M k in L n is produced using a query transmitter, as

o k = Ln[k](q k ) call k th module M k = Ln[k],

q k = Q n→k (s t , V, G n (s t )). The output o k = M k (q k ) received from M k is modified using a receiver function, as v k = R k→n (s t , o k ).

One can think of these functions as translators of the inputs and outputs into the module's own "language". Note that each module has a scratch pad V to store outputs it receives from a list of lower modules L n , i.e., v k is stored to V . Q n→k and R k→n stand for the query transmitter from task n to task k and receiver from task k to task n, respectively.

State update function. After every module in L n is executed, module M n updates its internal state using a state update function U n as s t+1 = U n (s t , V, E, G n (s t )). This completes one time step of the module's computation. Once the state is updated, the scratch pad V is wiped clean and is ready for new outputs. An example can be a simple gated sum of all outputs, i.e., s t+1 = k g k n • v k . Prediction function. After T n steps, the final module output is produced using a prediction function Ψ n as o n = Ψ n (s 1 , . . . , s Tn , q n , E). Recall that E is the environment.

All module functions: state initializer I, importance function G, query transmitter Q, receiver R, state update function U , residual module ∆, attention module Ω, and prediction function Ψ are implemented as neural networks or simple assignment functions (e.g. set q k = v l ). Note that all variables (e.g. o k , q k , v k , s t ) are continuous vectors to allow learning with standard backpropagation. For example, the output of the relationship detection module that predicts an object bounding box is a N dimensional soft-maxed vector (assuming there are total of N boxes or image regions in E).

Training. We train our modules sequentially, from low level to high level tasks, one at a time. The internal weights of the lower level modules are not updated, thus preserving their performance on the original task. The new module only learns to communicate with them via the query transmitter Q and receiver R. We do train the weights of ∆ and Ω. We train I, G, Q, R, U , and Ψ, by allowing gradients to pass through the lower level modules. The loss function depends on the task n.

3.3. PROGRESSIVE MODULE NETWORKS FOR VISUAL REASONING:

We present an example of how PMN can be adopted for several tasks related to visual reasoning. In particular, we consider six tasks: object classification, attribute classification, relationship detection, object counting, image captioning, and visual question answering. Our environment E consists of: (Ren et al., 2015) ; and (ii) language: vector representation of a sentence S (in our example, a question). S is computed through a Gated Recurrent Unit (Cho et al., 2014) by feeding in word embeddings [w 1 , . . . , w T ] at each time step. Below, we discuss each task and a module designed to solve it. We provide detailed implementation and execution process of the VQA module in Sec. 3.4. For other modules, we present a brief overview of what each module does in this section. Further implementation details of all module architectures are in Appendix A.

(i) image regions: N image features X = [X 1 , . . . , X N ], each X i ∈ R d with corresponding bounding box coordinates b = [b 1 , . . . , b N ] extracted from Faster R-CNN

Object and Attribute Classification (level 0). Object classification is concerned with naming the object that appears in the image region, while attribute classification predicts the object's attributes (e.g. color). As these two tasks are fairly simple (not necessarily easy), we place M obj and M att as Bench Compute importance scores. .

Call the attention module. Store the result in V.

Call the relationship module. Store the result in V.

4 Compute the attention map as a summation of weighted by softmax of their importance scores.

Pass the map to object, attribute and residual module. Store the results in V.

Call the counting module. Store the count vector in V.

Call the captioning module. Store the caption vector in V.

7 Update the module state by using outputs of weighted by softmax of their importance scores. terminal modules at level 0. M obj consists of an MLP that takes as input a visual descriptor for a bounding box b i , i.e., q obj = X i , and produces o obj = M obj (q obj ), the penultimate vector prior to classification. Attribute module M att has a similar structure. These are the only modules for which we do not use actual output labels, as we obtained better results for higher level tasks empirically.

Image Captioning (level 1). In image captioning, one needs to produce a natural language description of the image. We design our module M cap as a compositional module that uses information from L cap = [Ω cap , M obj , M att , ∆ cap ]. We implement the state update function as a two-layer GRU network with s t corresponding to the hidden states. Similar to Anderson et al. (2018) , at each time step, the attention module Ω cap attends over image regions X using the hidden state of the first layer. The attention map m is added to the scratch pad V . The query transmitters produce a query (image vector at the attended location) using m to obtain nouns M obj and adjectives M att . The residual module ∆ cap processes other image-related semantic information. The outputs from modules in L cap are projected to a common vector space (same dimensions) by the receivers and stored in the scratch pad. Based on their importance score, the gated sum of the outputs is used to update the state. The natural language sentence o cap is obtained by predicting a word at each time step using a fully connected layer on the hidden state of the second GRU layer.

Relationship Detection (level 1). In this task the model is expected to produce triplets in the form of "subject -relationship -object" (Lu et al., 2016) . We re-purpose this task as one that involves finding the relevant item (region) in an image that is related to a given input through a given relationship. The input to the module is q rel = [b i , r] where b i is a one-hot encoding of the input box and r is a one-hot encoding of the relationship category (e.g. above, behind). The module produces o rel = b out corresponding to the box for the subject/object related to the input b i through r. We place M rel on the first level as it may use object and attribute information that can be useful to infer relationships, i.e., L rel = [M obj , M att , ∆ rel ]. We train the module using the cross-entropy loss.

Object Counting (level 2). Our next task is counting the number of objects in the image. Given a vector representation of a natural language question (e.g. how many cats are in this image?), the goal of this module is to produce a numerical count. The counting task is at a higher-level since it may also require us to understand relationships between objects. For example, "how many cats are on the blue chair?", requires counting cats on top of the blue chair. We thus place M cnt on the second level and provide it access to L cnt = [Ω cnt , M rel ]. The attention module Ω cnt finds relevant objects by using the input question vector. M cnt may also query M rel if the question requires relational reasoning. To answer "how many cats are on the blue chair", we can expect the query transmitter Q cnt→rel to produce a query q rel = [b i , r] for the relationship module M rel that includes the chair bounding box b i and relationship "on top of" r so that M rel outputs boxes that contain cats on the chair. Note that both Ω cnt and M rel produce attention maps on the boxes. The state update function softly chooses a useful attention map by calculating softmax on the importance scores of Ω cnt and M rel . For prediction function Ψ cnt , we adopt the counting algorithm by Zhang et al. (2018) , which builds a graph representation from attention maps to count objects. M cnt returns o cnt which is the count vector corresponding to softmaxed one-hot encoding of the count (with maximum count ∈ Z).

Visual Question Answering (level 3). VQA is our final and most complex task. Given a vector representation of a natural language question, q vqa , the VQA module M vqa uses L vqa = [Ω vqa , M rel , M obj , M att , ∆ vqa , M cnt , M cap ]. Similar to M cnt , M vqa makes use of Ω vqa and M rel to get an attention map. The produced attention map is fed to the downstream modules [M obj , M att , ∆ vqa ] using the query transmitters. M vqa also queries M cnt which produces a count vector. For the last entry M cap in L vqa , the receiver attends over the words of the entire caption produced by M cap to find relevant answers. The received outputs are used depending on the importance scores. Finally, Ψ vqa produces an output vector based on q vqa and all states s t .

3.4 EXAMPLE: M vqa FOR VISUAL QUESTION ANSWERING We give a detailed example of how PMN is implemented for the VQA task. The entire execution process is depicted in Fig. 2 , and the general algorithm is tabulated in Alg. 1.

The input q vqa is a vector representing a natural language question (i.e. the sentence vector S ∈ E). The state variable s t is represented by a tuple (q t vqa , k t-1 ) where q t vqa represents query to ask at time t and k t-1 represents knowledge gathered at time t -1. The state initializer I vqa is composed of a GRU with hidden state dimension 512. The first input to GRU is q vqa , and I vqa sets s 1 = (q 1 vqa , 0) where q 1 vqa is the first hidden state of the GRU and 0 is a zero vector (no knowledge at first). For t in T vqa = 2, M vqa does the following seven operations:

(1) The importance function G vqa is executed. It is implemented as a linear layer R 512 → R 7 (for the seven modules in L vqa ) that takes s t , specifically q t vqa ∈ s t as input.

(2) Q vqa→Ω passes q t vqa to the attention module Ω vqa which attends over the image regions X with q t vqa as the key vector. Ω vqa is implemented as an MLP that computes a dot-product soft-attention similar to Yang et al. (2016) . The returned attention map v Ω is added to the scratch pad V .

(3) Q vqa→rel produces an input tuple [b, r] for M rel . The input object box b is produced by a MLP that does soft attention on image boxes, and the relationship category r is produced through a MLP with q t vqa as input. M rel is called with [b, r] and the returned map v rel is added to V . (4) Q vqa→obj , Q vqa→att , and Q vqa→∆ first compute a joint attention map m as summation of (v Ω , v rel ) weighted by the softmaxed importance scores of (Ω vqa , M rel ), and they pass the sum of visual features X weighted by m to the corresponding modules. ∆ vqa is implemented as an MLP. The receivers project the outputs into 512 dimensional vectors v obj , v att , and v ∆ through a sequence of linear layers, batch norm, and tanh() nonlinearities. They are added to V .

(5) Q vqa→cnt passes q t vqa to M cnt which returns o cnt . R cnt→vqa projects the count vector o cnt into a 512 dimensional vector v cnt through the same sequence of layers as above. v cnt is added to V .

(6) M vqa calls M cap and R cap→vqa receives natural language caption of the image. It converts words in the caption into vectors [w 1 , . . . , w T ] through an embedding layer. The embedding layer is initialized with 300 dimensional GloVe vectors (Pennington et al., 2014) and fine-tuned. It does softmax attention operation over [w 1 , . . . , w T ] through a MLP with q t vqa ∈ s t as the key vector, resulting in word probabilities p 1 , . . . , p T . The sentence representation T i p i • w i is projected into a 512 dimensional vector v cap using the same sequence as v cnt . v cap is added to V .

(7) The state update function U vqa first does softmax operation over the importance scores of (M obj , M att , ∆ vqa , M cnt , M cap ). We define an intermediate knowledge vector k t as the summation of (v obj , v att , δ vqa , v cnt , v cap ) weighted by the softmaxed importance scores. U vqa passes k t as input to the GRU initialized by I vqa , and we get q t+1 vqa the new hidden state of the GRU. The new state s t+1 is set to (q t+1 vqa , k t ). This process allows the GRU to compute new question and state vectors based on what has been asked and seen.

After T vqa steps, the prediction function Ψ vqa computes the final output based on the initial question vector q vqa and all knowledge vectors k t ∈ s t . Here, q vqa and k t are fused with gated-tanh layers and fed through a final classification layer similar to Anderson et al. (2018) , and the logits for all time steps are added. The resulting logit is the final output o vqa that corresponds to an answer in the vocabulary of the VQA task. Note that the exact form of each module can be different. While we leave a more general architecture across tasks as future work, we stress that one of PMN's strengths is that once a module is trained, it can be used as a blackbox by the higher-level modules. Details of other modules' architectures are provided in Appendix A.

4. EXPERIMENTS:

We present experiments demonstrating the impact of progressive learning of modules. We also analyze and evaluate the reasoning process of PMN as it is naturally interpretable. We conduct experiments on three datasets (see Appendix B.1 for details): Visual Genome (VG) (Krishna et al., 2016) , VQA 2.0 (Goyal et al., 2017) , MS-COCO (Lin et al., 2014) . These datasets contain natural images and are thus more complex in visual appearance and language diversity than CLEVR (Johnson et al., 2017a ) that contains synthetic scenes. Neural module networks (Andreas et al., 2016; Hu et al., 2017) show excellent performance on CLEVR but their performance on natural images is quite below the state-of-the-art. For all datasets, we extract bounding boxes and their feature representations using a pretrained model from Anderson et al. (2018) .

4.1. PROGRESSIVE LEARNING OF TASKS AND MODULES:

Object and Attribute Classification. We train these modules with annotated bounding boxes from the VG dataset. We follow Anderson et al. (2018) and use 1,600 and 400 most commonly occurring object and attribute classes, respectively. Each extracted box is associated with the ground truth label of the object with greatest overlap. It is ignored if there are no ground truth boxes with IoU > 0.5. This way, each box is annotated with one object label and zero or more attribute labels. M obj achieves 54.9% top-1 accuracy and 86.1% top-5 accuracy. We report mean average precision (mAP) for attribute classification which is a multi-label classification problem. M att achieves 0.14 mAP and 0.49 weighted mAP. mAP is defined as the mean over all classes, and weighted mAP is weighted by the number of instances for each class. As there are a lot of redundant classes (e.g. car, cars, vehicle) and boxes have sparse attribute annotations, the accuracy may seem artificially low.

Image Captioning. We report results on MS-COCO for image captioning. We use the standard split from the 2014 captioning challenge to avoid data contamination with VQA 2.0 or VG. This split contains 30% less data compared to that proposed in Karpathy & Fei-Fei (2015) that most current works adopt. We report performance using the CIDEr (Vedantam et al., 2015) metric. A baseline (non-compositional module) achieves a strong CIDEr score of 108. Using the object and attribute modules we are able to obtain 109 CIDEr. While this is not a large improvement, we suspect a reason for this is the limited vocabulary. The MS-COCO dataset has a fixed set of 80 object categories and does not benefit by using knowledge from modules that are trained on more diverse data. We believe the benefits of PMN would be clearer on a diverse captioning dataset with many more object classes. Also, including high-level modules such as M vqa would be an interesting direction for future work. Relationship Detection. We use top 20 commonly occurring relationship categories, which are defined by a set of words with similar meaning (e.g. in, inside, standing in). Relationship tuples in the form of "subject -relationship -object" are extracted from Visual Genome (Krishna et al., 2016; Lu et al., 2016) . We train and validate the relationship detection module using 200K/38K train/val tuples that have both subject and object boxes overlapping with the ground truth boxes (IoU > 0.7).

Table 1 shows improvement in performance when using modules. Even though accuracy is relatively low, model errors are reasonable, qualitatively. This is partially attributed to multiple correct answers although there is only one ground truth answer.

Object Counting. We extract questions starting with 'how many' from VQA 2.0 which results in a training set of ∼50K questions. We additionally create ∼89K synthetic questions based on the VG dataset by counting the object boxes and forming 'how many' questions. This synthetic data helps to increase the accuracy by ∼1% for all module variants. Since the number of questions that have relational reasoning and counting (e.g. how many people are sitting on the sofa? how many plates on table ?) is limited, we also sample relational synthetic questions from VG. These questions are used only to improve the parameters of query transmitter Q cnt→rel for the relationship module. Table 2 shows a large improvement (4.6%) of the compositional module over the non-modular baseline. When training for the next task (VQA), unlike other modules whose parameters are fixed, we fine-tune the counting module because counting module expects the same form of input -embedding of natural language question. The performance of counting module depends crucially on the quality of attention map over bounding boxes. By employing more questions from the whole VQA dataset, we obtain a better attention map, and the performance of counting module increases from 50.0% (c.f . Table 2 ) to 55.8% with finetuning (see Appx B.2 for more details).

Visual Question Answering. We present ablation studies on the val set of VQA 2.0 in Table 3 . As seen, PMN strongly benefits from utilizing different modules achieving a performance improvement of 2.6% over the baseline. Note that all results here are without additional questions from the VG data. We also compare performance of PMN for the VQA task with state-of-the-art models in Table 4 . Models are trained on the train split for results on VQA val, while for test-dev and test-std, models are trained on both the train and val splits. Although we start with a much lower baseline performance of 62.05% on the val set (vs. 65.42% (Zhang et al., 2018) , 63.15% (Teney et al., 2018) , 66.04% (Kim et al., 2018) ), PMN's performance is on par with these models. Note that entries with * are parallel works to ours. Also, as Jiang et al. (2018) showed, the performance depends strongly on engineering choices such as learning rate scheduling and ensembling models with different architectures.

Plug-and-play architecture.

The query-answer communication within PMN results in a plugand-play architecture where modules can be replaced by their improved versions. Instead of using generated captions from M cap , when we feed in the ground-truth captions to trained M vqa 5 in Table 3 (with 64.68% acc.), it achieves 65.43%. We also tried training and validating M vqa 5 with ground-truth captions, and this achieved 67.84%. These results shows how PMN can be continually improved.

Three additional experiments on VQA.

(1) To verify that the gain is not from the increased model capacity, we trained a baseline with the number of parameters approximately matching that of the full PMN model. This baseline with more capacity also achieves 62.0%, thus confirming our claim. (2)

We evaluated the impact of the additional data available. We convert the subj-obj-rel triplets used for the relationship detection task to additional QAs (e.g. Q: what is on top of the desk?, A: laptop) and train the M vqa 1 model (Table 3 ). This results in an accuracy of 63.05%, not only lower than M vqa2 (63.64%) that uses the relationship module via PMN, but also lower than M vqa1 at 63.38%. This suggests that while additional data may change the question distribution and reduce performance, PMN is robust and benefits from a separate relationship module. (3) Lastly, we conducted another experiment to show that PMN does make efficient use of the lower level modules. We give equal importance scores to all modules in M vqa5 model (Table 3 ) (thus, fixed computation path), achieving 63.65% accuracy. While this is higher than the baseline at 62.05%, it is lower than M vqa5 at 64.68% which softly chooses which sub-modules to use.

4.2. INTERPRETABILITY ANALYSIS:

Visualizing the model's reasoning process. We present a qualitative analysis of the answering process. In Fig. 2 , M vqa makes query q rel = [b i , r] to M rel where b i corresponds to the blue box 'bird' and r corresponds to 'on top of' relationship. M vqa correctly chooses (i.e. higher importance score) to use M rel rather than its own output produced by Ω vqa since the question requires relational reasoning. With the attended green box obtained from M rel , M vqa mostly uses the object and captioning modules to produce the final answer. More examples are presented in Appendix C. Judging Answering Quality. The modular structure and gating mechanism of PMN makes it easy to interpret the reasoning behind the outputs. We conduct a human evaluation with Amazon Mechanical Turk on 1,600 randomly chosen questions. Each worker is asked to rate the explanation generated by the baseline model and the PMN like a teacher grades student exams in the scale of 0 (very bad), 1 (bad), 2 (satisfactory), 3 (good), 4 (very good). The baseline explanation is composed of the bounding box it attends to and the final answer. For PMN, we form a rule-based natural language explanation based on the prominent modules used. An example is shown in Fig. 3 . Each question is assessed by three human workers. Incorrect reasoning steps are penalized, so if PMN produces wrong reasoning steps, it could get a low score. On the other hand, the baseline model often scores well on simple questions that do not need complex reasoning (e.g. what color is the cat?).

Q: what is behind the men?

• I first find the BLUE box, and then from that, I look at the GREEN box. • The object 'tree' would be useful in answering the question. • In conclusion, I think the answer is trees.

Q: what color is the curvy wire?

• I look at the RED box.

• The object properties white long electrical would be useful in answering the question. • In conclusion, I think the answer is white.

Figure 3 : Example of PMN's reasoning processes. Top: it correctly first find a person and then uses relationship module to find the tree behind him. Bottom: it finds the wire and then use attribute module to correctly infer its attributes -white, long, electrical -and then outputs the correct answer.

We report results in Table 5 , and show more examples in Appendix D. Human evaluators tend to give low scores to wrong answers and high scores to correct answers regardless of explanations, but PMN always scores higher if both PMN and baseline gets a question correct or wrong. Interestingly, a correct answer from PMN gets 1.38 points higher than wrong baseline, but a correct baseline scores only 0.74 higher than a wrong PMN answer. This shows that PMN gets partial marks even when it gets an answer wrong since the reasoning steps are partially correct.

Low Data Regime. PMN benefits from re-using modules and only needs to learn the communication between them. This allows us to achieve good performance even when using a fraction of the training data. Table 6 presents the absolute gain in accuracy PMN achieves. For this experiment, we use L vqa = [Ω vqa , M rel , M obj , M att , ∆ vqa , M cap ] (because of overlapping questions from M cnt ). When the amount of data is really small (1%), PMN does not help because there is not enough data to learn to communicate with lower modules. The maximum gain is obtained when using 10% of data. It shows that PMN can help in situations where there is not a huge amount of training data since it can exploit previously learned knowledge. The gain remains constant at about 2% from then on. 

5. CONCLUSION AND DISCUSSION:

In this work, we proposed Progressive Module Networks (PMN) that train task modules in a compositional manner, by exploiting previously learned lower-level task modules. PMN can produce queries to call other modules and make use of the returned information to solve the current task. Given experts in specific tasks, the parent module only needs to learn how to effectively communicate with them. It can also choose which lower level modules to use. Thus, PMN is data efficient and provides a more interpretable reasoning processes. Also, since there is no need to know about the inner workings of the children modules, it opens up promising ways to train intelligent robots that can compartmentalize and perform multiple tasks as they progressively improve their abilities. Moreover, one task can benefit from unrelated tasks unlike conventional multi-task learning algorithms.

PMN as it stands has few limitations with respect to hand-designed structures and the need for additional supervision. Nevertheless, PMN is an important step towards more interpretable, compositional multi-task models. Some of the questions to be solved in the future include: 1) learning module lists automatically; 2) choosing few modules (hard attention) to reduce overhead; 3) more generic structure of module components across tasks; and 4) joint training of all modules.

The output of M cap , o cap , is a sequence of words produced through Ψ cap which is a linear layer projecting each h t 2 in s t to the output word vocabulary.

A.3 RELATIONSHIP DETECTION (LEVEL 1):

Relationship detection task requires one to produce triplets in the form of "subject -relationship -object" (Lu et al., 2016) . We re-purpose this task as one that involves finding the relevant item (region) in an image that is related to a given input through a given relationship. The input to the module is q rel = [b i , r] where b i is a one-hot encoded input bounding box (whose i-th entry is 1 and others 0) and r is a one-hot encoded relationship category (e.g. above, behind). M rel has L rel = [M obj , M att , ∆ rel ] and goes through T rel = N steps where N is the number of bounding boxes (image regions in the environment). So at time step t, the module looks at the t-th box. M rel uses M obj and M att just as feature extractors for each bounding box. Therefore, it does not have a complex structure.

The state initializer I rel projects r to a 512 dimensional vector with an embedding layer, and the resulting vector is set as the first state s 1 .

For t in T rel = N , M rel does the following three operations:

(1) Q rel→obj and Q rel→att pass the image vector corresponding to the bounding box b t to M obj and M att . R obj→rel and R att→rel are identity functions, i.e., we do not modify the object and attribute vectors. The outputs v obj and v att are added to V .

(2) ∆ rel projects the coordinate of the current box b t to a 512 dimensional vector. This resulting v ∆ is added to V .

(3) U rel concatenates the visual feature X t with v obj , v att , v ∆ from V . The concatenated vector is fed through a MLP resulting in 512 dimensional vector. This corresponds to the new state s t+1 .

After N steps, the prediction function Ψ rel does the following operations: The first state s 1 which contains relationship information is multiplied element-wise with s i+1 (Note: s i+1 corresponds to the input box b i ). Let such a vector be l. It produces an attention map b out over all bounding boxes in b. The inputs to the attention function are s 2 , . . . , s T rel (i.e. all image regions) and the key vector l. o rel = b out is the output of M rel which represents an attention map indicating the bounding box that contains the related object.

A.4 COUNTING (LEVEL 2)

Given a vector representation of a natural language question (e.g. how many cats are in this image?), the goal of this module is to produce a count. The input q cnt = S ∈ E is a vector representing a natural language question. When training M cnt , q cnt is computed through a one layer GRU with hidden size of 512 dimensions. The input to the GRU at each time step is the embedding of each word from the question. Word embeddings are initialized with 300 dimensional GloVe word vectors (Pennington et al., 2014) and fine-tuned thereafter. Similar to visual features obtained through CNN, the question vector is treated as an environment variable. M cnt has L cnt = [Ω cnt , M rel ] and goes through only one time step.

The state initializer I cnt is a simple function that just sets s 1 = q cnt .

For t in T cnt = 1, M cnt does the following four operations:

(1) The importance function G cnt is executed. It is implemented as a linear layer R 512 → R 2 (for the two modules in L cnt ) that takes s t as input.

(2) Q cnt→Ω passes s t to the attention module Ω cnt which attends over the image regions X with s t as the key vector. Ω cnt is implemented as an MLP that computes a dot-product soft-attention similar to Yang et al. (2016) . The returned attention map v Ω is added to the scratch pad V .

(3) Q cnt→rel produces an input tuple [b, r] for M rel . The input object box b is produced by a MLP that does soft attention on image boxes, and the relationship category r is produced through a MLP with s t as input. M rel is called with [b, r] and the returned map v rel is added to V .

(4) U cnt first computes probabilities of using v Ω or v rel by doing a softmax over the importance scores. v Ω and v rel are weighted and summed with the softmax probabilities resulting in the new state s 2 containing the attention map. Thus, the state update function chooses the map from M rel if the given question involves in relational reasoning.

The prediction function Ψ cnt returns a count vector. The count vector is computed through the counting algorithm by Zhang et al. (2018) , which builds a graph representation from attention maps to count objects. The method uses s 2 through a sigmoid and bounding box coordinates b as inputs. Zhang et al. (2018) is a fully differentiable algorithm and the resulting count vector corresponds to one-hot encoding of a number. We let the range of count be 0 to 12 ∈ Z. Please refer to Zhang et al. (2018) for details of the counting algorithm.

A.5 VISUAL QUESTION ANSWERING (LEVEL 3)

The description for the VQA task (Sec. 3.4) is included here again for completeness. The input q vqa is a vector representing a natural language question (i.e. the sentence vector S ∈ E). The state variable s t is represented by a tuple (q t vqa , k t-1 ) where q t vqa represents query to ask at time t and k t-1 represents knowledge gathered at time t -1. The state initializer I vqa is composed of a GRU with hidden state dimension 512. The first input to GRU is q vqa , and I vqa sets s 1 = (q 1 vqa , 0) where q 1 vqa is the first hidden state of the GRU and 0 is a zero vector (no knowledge at first).

For t in T vqa = 2, M vqa does the following seven operations:

(1) The importance function G vqa is executed. It is implemented as a linear layer R 512 → R 7 (for the seven modules in L vqa ) that takes s t , specifically q t vqa ∈ s t as input.

(2) Q vqa→Ω passes q t vqa to the attention module Ω vqa which attends over the image regions X with q t vqa as the key vector. Ω vqa is implemented as an MLP that computes a dot-product soft-attention similar to Yang et al. (2016) . The returned attention map v Ω is added to the scratch pad V .

(3) Q vqa→rel produces an input tuple [b, r] for M rel . The input object box b is produced by a MLP that does soft attention on image boxes, and the relationship category r is produced through a MLP with q t vqa as input. M rel is called with [b, r] and the returned map v rel is added to V . (4) Q vqa→obj , Q vqa→att , and Q vqa→∆ first compute a joint attention map m as summation of (v Ω , v rel ) weighted by the softmaxed importance scores of (Ω vqa , M rel ), and they pass the sum of visual features X weighted by m to the corresponding modules. ∆ vqa is implemented as an MLP. The receivers project the outputs into 512 dimensional vectors v obj , v att , and v ∆ through a sequence of linear layers, batch norm, and tanh() nonlinearities. They are added to V .

(5) Q vqa→cnt passes q t vqa to M cnt which returns o cnt . R cnt→vqa projects the count vector o cnt into a 512 dimensional vector v cnt through the same sequence of layers as above. v cnt is added to V .

(6) M vqa calls M cap and R cap→vqa receives natural language caption of the image. It converts words in the caption into vectors [w 1 , . . . , w T ] through an embedding layer. The embedding layer is initialized with 300 dimensional GloVe vectors (Pennington et al., 2014) and fine-tuned. It does softmax attention operation over [w 1 , . . . , w T ] through a MLP with q t vqa ∈ s t as the key vector, resulting in word probabilities p 1 , . . . , p T . The sentence representation T i p i • w i is projected into a 512 dimensional vector v cap using the same sequence as v cnt . v cap is added to V .

(7) The state update function U vqa first does softmax operation over the importance scores of (M obj , M att , ∆ vqa , M cnt , M cap ). We define an intermediate knowledge vector k t as the summation of (v obj , v att , δ vqa , v cnt , v cap ) weighted by the softmaxed importance scores. U vqa passes k t as input to the GRU initialized by I vqa , and we get q t+1 vqa the new hidden state of the GRU. The new state s t+1 is set to (q t+1 vqa , k t ). This process allows the GRU to compute new question and state vectors based on what has been asked and seen.

After T vqa steps, the prediction function Ψ vqa computes the final output based on the initial question vector q vqa and all knowledge vectors k t ∈ s t . Here, q vqa and k t are fused with gated-tanh layers and fed through a final classification layer similar to Anderson et al. (2018) , and the logits for all time steps are added. The resulting logit is the final output o vqa that corresponds to an answer in the vocabulary of the VQA task. which compute a relationship category and the input image region to be passed to M rel . To be clear, we supervise q rel = [b i , r] to be sent to M rel by reducing cross entropy loss on b i and r.

Visual Question Answering. M vqa is trained using binary cross-entropy loss on o vqa with Adam optimizer with learning rate of 0.0005 with batch size 128 for 7 epochs. We empirically found binary cross-entropy loss to work better than cross-entropy which was also reported by Anderson et al. (2018) . Unlike other modules whose parameters are fixed, we fine-tune only the counting module because counting module expects the same form of input -embedding of natural language question. The performance of counting module depends crucially on the quality of attention map over bounding boxes. By employing more questions from the whole VQA dataset, we obtain a better attention map, and the performance of counting module increases from 50.0% to 55.8% with finetuning. Since M vqa and M cnt expect the same form of input, the weights of attention modules Ω {vqa,cnt} and query transmitters for the relationship module Q {vqa,cnt}→rel are shared.

:



Figures and Tables:

Figure1: An example computation graph for PMN with four tasks. Green rectangles denote terminal modules, and yellow rectangles denote compositional modules. Blue arrows and red arrows represent calling and receiving outputs from submodules, respectively. White numbered circles denote computation order. For convenience, assume task levels correspond to the subscripts. Calling M3 invokes a chain of calls (blue arrows) to lower level modules which stop at the terminal modules.

Figure 2: Example of PMN's module execution trace on the VQA task. Numbers in circles indicate the order of execution. Intensity of gray blocks represents depth of module calls. All variables including queries and outputs stored in V are continuous vectors to allow learning with standard backpropagation (e.g., caption is composed of a sequence of softmaxed W dimensional vectors for vocabulary size W ). For Mcap, words with higher intensity in red are deemed more relevant by R cap vqa . Top: high level view of module execution process. Bottom right: computed importance scores and populated scratch pad. Note that we perform the first softmax operation on (Ωvqa, M rel ) to obtain an attention map and the second on (M obj , Matt, ∆vqa, Mcnt, Mcap) to obtain the answer. Bottom left: visualizing the query Mvqa sends to M rel , and the received output.

Algorithm 1 Computation performed by our Progressive Module Network, for one module Mn

Table content: <table><tr><td colspan="3">1: function Mn(qn)</td><td/><td>The environment E and module list Ln are global variables</td></tr><tr><td>2:</td><td>s 1 = In(qn)</td><td/><td/><td>initialize the state variable</td></tr><tr><td>3:</td><td colspan="3">for t ← 1 to Tn do</td><td>Tn is the maximum time step</td></tr><tr><td>4:</td><td>V = []</td><td/><td/><td>wipe out scratch pad V</td></tr><tr><td>5:</td><td>g 1 n , . . . , g</td><td>|Ln| n</td><td>= Gn(s t )</td><td>compute importance scores</td></tr><tr><td>6:</td><td colspan="3">for k ← 1 to |Ln| do</td><td>Ln is the sequence of lower modules [Mm, ..., M l ]</td></tr><tr><td>7:</td><td/><td/><td/><td/></tr></table>

Performance of M rel

Table content: <table><tr><td>Model</td><td colspan="4">Composition BASE OBJ ATT Object Subject Acc. (%)</td></tr><tr><td>Mrel 0</td><td>-</td><td>-</td><td>51.0</td><td>55.9</td></tr><tr><td>Mrel 1</td><td colspan="3">Mobj Matt 53.4</td><td>57.8</td></tr></table>

Accuracy for Mcnt

Table content: <table><tr><td>Model</td><td colspan="3">Composition BASE OBJ ATT REL</td><td>Acc. (%)</td></tr><tr><td>Mcnt 0</td><td>-</td><td>-</td><td>-</td><td>45.4</td></tr><tr><td>Mcnt 1</td><td colspan="2">Mobj Matt</td><td>-</td><td>47.4</td></tr><tr><td>Mcnt 2</td><td colspan="3">Mobj Matt Mrel 1</td><td>50.0</td></tr></table>

Model ablation for VQA. We report mean±std computed over three runs. Steady increase indicates that information from modules helps, and that PMN makes use of lower modules effectively. The base model Mvqa 0 does not use any lower level modules other than the residual and attention modules.

Table content: <table><tr><td>Model</td><td>BASE</td><td>OBJ</td><td>Composition ATT REL</td><td>CNT</td><td>CAP</td><td>Accuracy (%)</td></tr><tr><td>Mvqa 0 Mvqa 1 Mvqa 2 Mvqa 3 Mvqa 4 Mvqa 5</td><td/><td colspan="4">-M obj Matt -M obj Matt M rel 1 --M obj Matt -M obj Matt M rel 1 Mcnt 2 ---Mcnt 1 M obj Matt M rel 1 Mcnt 2 Mcap 1 -----</td><td>62.05 ±0.11 63.38 ±0.05 63.64 ±0.07 64.06 ±0.05 64.36 ±0.06 64.68 ±0.04</td></tr></table>

Comparing VQA accuracy of PMN with state-of-the-art models. Rows with Ens denote ensemble models. test-dev is development test set and test-std is standard test set for VQA 2.0.

Table content: <table><tr><td>Model</td><td>Ens</td><td colspan="11">VQA 2.0 val Yes/No Number Other All Yes/No Number Other All Yes/No Number Other All VQA 2.0 test-dev VQA 2.0 test-std</td></tr><tr><td>Andreas et al. (2016)</td><td>-</td><td>73.38</td><td colspan="3">33.23 39.93 51.62</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Yang et al. (2016)</td><td>-</td><td>68.89</td><td colspan="3">34.55 43.80 52.20</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Teney et al. (2018)</td><td>-</td><td>80.07</td><td colspan="4">42.87 55.81 63.15 81.82</td><td colspan="4">44.21 56.05 65.32 82.20</td><td colspan="2">43.90 56.26 65.67</td></tr><tr><td>Teney et al. (2018)</td><td/><td>-</td><td>-</td><td>-</td><td>-</td><td>86.08</td><td colspan="4">48.99 60.80 69.87 86.60</td><td colspan="2">48.64 61.15 70.34</td></tr><tr><td>Yu et al. (2018)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>84.27</td><td colspan="3">49.56 59.89 68.76</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Yu et al. (2018)</td><td/><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>86.65</td><td colspan="2">51.13 61.75 70.92</td></tr><tr><td>Zhang et al. (2018)</td><td>-</td><td>-</td><td>49.36</td><td>-</td><td colspan="2">65.42 83.14</td><td colspan="4">51.62 58.97 68.09 83.56</td><td colspan="2">51.39 59.11 68.41</td></tr><tr><td>Kim et al. (2018)*</td><td>-</td><td>-</td><td>-</td><td>-</td><td colspan="2">66.04 85.43</td><td colspan="4">54.04 60.52 70.04 85.82</td><td colspan="2">53.71 60.69 70.35</td></tr><tr><td>Kim et al. (2018)*</td><td/><td>-</td><td>-</td><td>-</td><td>-</td><td>86.68</td><td colspan="4">54.94 62.08 71.40 87.22</td><td colspan="2">54.37 62.45 71.84</td></tr><tr><td>Jiang et al. (2018)*</td><td/><td>-</td><td>-</td><td>-</td><td>-</td><td>87.82</td><td colspan="4">51.54 63.41 72.12 87.82</td><td colspan="2">51.59 63.43 72.25</td></tr><tr><td>baseline Mvqa 0</td><td>-</td><td>80.28</td><td colspan="3">43.06 53.21 62.05</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PMN Mvqa 5</td><td>-</td><td>82.48</td><td colspan="4">48.15 55.53 64.68 84.07</td><td colspan="3">52.12 57.99 68.07</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PMN Mvqa 5</td><td/><td>-</td><td>-</td><td>-</td><td>-</td><td>85.74</td><td colspan="4">54.39 60.60 70.25 86.34</td><td colspan="2">54.26 60.80 70.68</td></tr></table>

Average human judgments from 0 to 4. indicates that model got final answer right, and for wrong.

Table content: <table><tr><td>Correct? PMN Baseline</td><td># Q</td><td colspan="2">Human Rating PMN Baseline</td></tr><tr><td/><td colspan="2">715 3.13</td><td>2.86</td></tr><tr><td/><td colspan="2">584 2.78</td><td>1.40</td></tr><tr><td/><td colspan="2">162 1.73</td><td>2.47</td></tr><tr><td/><td colspan="2">139 1.95</td><td>1.66</td></tr><tr><td>All images</td><td colspan="2">1600 2.54</td><td>2.24</td></tr></table>

Absolute gain in accuracy when using a fraction of the training data.

Table content: <table><tr><td>Fraction of VQA training data (in %)</td><td>1</td><td>5</td><td>10</td><td>25</td><td>50 100</td></tr><tr><td>Absolute accuracy gain (in %)</td><td colspan="5">-0.49 2.21 4.01 2.66 1.79 2.04</td></tr></table>

Peer review (iclr2019):
 [Summary] This paper presents a multi-task learning approach for VQA that represent a solver for each task as a neural module that calls existing modules in a program manner. The authors manually design the task hierarchy and propose a progressive module network to recursive calls the lower modules and gather the information by soft-attention. The final prediction uses all the states and question to infer the final answers. The authors verify the effectiveness of the proposed method on the performance of different tasks and modules. Experiment on VQA shows the proposed model benefits from utilizing different modules. The authors also qualitatively show the model*s reasoning process and human study on judging answering quality. [Strength] 1. The proposed method is novel and explores to use the existing modules as a black box for visual question answering. This is different from most existing work. 2: By examing different modules, the proposed method is more interpretable compare to canonical methods. 3: The experiment results are good, especially for the counting problem. [Weakness] 1. The title of the paper is "visual reasoning by progressive module networks." The title may be a little overstated since the major task is focused on visual question answering (VQA). 2. Annotation is not clear in this paper. For example, on page 3, Query transmitter and receiver, "the output o_k = M_k(q_k) received from M_k is modified using receiver function as v_k = R_{k->n}(s^t, o_k). " There are multiple new variables in this paragraph, without specifying the dimension and meaning for each attribute, it*s really hard to understand. On page 4, State update function, what is the meaning of variable "Epsilon" in the equation? From the supplementary, it seems Epsilon means the environment? 3. On the object counting task, the query transmitter needs to produce a query for a relationship module. The authors mentioned that this is softly calculated by softmax on the importance score. Since q_rel require one hot vector as input, how to sample the q_rel given the importance score and how backprob the gradient in this case? 4. The cider score of image captioning is 109 compared to the baseline 108. The explanation is the COCO dataset has a fixed set of 80 object categories and does not benefit from training the diverse data. Since the input visual feature is the same, the only difference is the proposed model has additional label embedding as input. My assumption is the visual feature already contains the label information for image captioning. 5. On relational detection task, is there a way to compare with the STOA method on some specific data split? This will leads to much more convincing results. 6. Similar as above question, on the object counting task, is there a way to compare with previous counting methods? 7. In Table 4, the accuracy of number on Zhang et.al is 49.39, which is higher than other methods, while on test-dev, the accuracy is 51.62, which is lower than others. Is the number right?