Title: TRANSFER LEARNING VIA UNSUPERVISED TASK DISCOVERY FOR VISUAL QUESTION ANSWERING

Abstract: We study how to leverage off-the-shelf visual and linguistic data to cope with out-ofvocabulary answers in visual question answering. Existing large-scale visual data with annotations such as image class labels, bounding boxes and region descriptions are good sources for learning rich and diverse visual concepts. However, it is not straightforward how the visual concepts should be captured and transferred to visual question answering models due to missing link between question dependent answering models and visual data without question or task specification. We tackle this problem in two steps: 1) learning a task conditional visual classifier based on unsupervised task discovery and 2) transferring and adapting the task conditional visual classifier to visual question answering models. Specifically, we employ linguistic knowledge sources such as structured lexical database (e.g. Wordnet) and visual descriptions for unsupervised task discovery, and adapt a learned task conditional visual classifier to answering unit in a visual question answering model. We empirically show that the proposed algorithm generalizes to unseen answers successfully using the knowledge transferred from the visual data.

1. INTRODUCTION:

People see and understand a visual scene from diverse perspectives. For example, from a single image of a chair, people effortlessly recognize diverse visual concepts such as color, material, style, usage, and so on. These diverse perspectives may be associated with different questions in natural language for which people find appropriate answers. Recently visual question answering (VQA) (Antol et al., 2015) is proposed as an effort to learn deep neural network models with capability to perform diverse visual recognition tasks defined adaptively by questions.

Approaches to VQA rely on a large-scale dataset of image, question and answer triples, and train a classifier taking an image and a question as inputs and producing an answer. Despite recent remarkable progress (Yang et al., 2016; Fukui et al., 2016; Anderson et al., 2018) , this direction has a critical limitation that image, question and answer triples in datasets are the only source for learning visual concepts. Such drawback may result in lack of scalability because the triplets may be collected artificially by human annotators with limited quality control and have weak diversity in visual concepts. In fact, VQA datasets (Goyal et al., 2017; Agrawal et al., 2018) suffer from inherent bias, which hinders learning true visual concepts from the datasets. On the contrary, people answer a question based on visual concepts learned from diverse sources such as personal experience, books, pictures, and videos, which are not necessarily associated with target questions. Even for machines, there exist more natural and scalable sources for learning visual concepts: image class labels, bounding boxes and region descriptions. Such information is already available in large-scale (Deng et al., 2009; Krasin et al., 2017; Krishna et al., 2017) and can scale further with reasonable cost (Papadopoulos et al., 2016; 2017) . This observation brings a natural question; can we learn visual concepts without question annotations and transfer them for VQA?

To address this question, we introduce VQA with out-of-vocabulary answers, which is illustrated in Figure 1 . External visual data provide a set of labels A and only a subset of these labels B ⊂ A appears in VQA training set as answers. The goal of this setting is to handle out-of-vocabulary answers a ∈ A -B successfully by exploiting visual concepts learned from external visual data. To address this problem, this paper studies how to learn visual concepts without questions and how to transfer the learned concepts to VQA models. To learn transferable visual concepts, we train a task conditional visual classifier, whose task is defined by a task specification vector. The classifier is used as an answering unit where a task specification vector is inferred from a question. To train the task conditional visual classifier without task annotations, we propose an unsupervised task discovery technique based on linguistic knowledge sources such as structured lexical databases, e.g., Wordnet (Fellbaum, 1998) , and region descriptions. We present that the proposed transfer learning helps generalization in VQA with out-of-vocabulary answers.

The main contribution of our paper is three-fold:

• We present a novel transfer learning algorithm for visual question answering based on a task conditional visual classifier.

• We propose an unsupervised task discovery technique for learning task conditional visual classifiers without explicit task annotations.

• We demonstrate that the proposed method enables to answer with out-of-vocabulary answers based on knowledge transfer from visual data without question annotations.

The rest of the paper is organized as follows. Section 2 discusses prior works related to our approach. We describe the overall transfer learning framework in Section 3. Learning visual concepts by unsupervised task discovery is described in Section 4. Section 5 analyzes experimental results and Section 6 makes our conclusion.

2. RELATED WORKS:

Standard VQA evaluation assumes identically distributed train and test set (Malinowski & Fritz, 2014; Antol et al., 2015; Zhu et al., 2016) . As this evaluation setting turns out to be vulnerable to models exploiting biases in training set (Goyal et al., 2017) , several alternatives have been proposed. One approach is to reduce observed biases either by balancing answers for individual questions (Goyal et al., 2017) or by providing different biases to train and test sets intentionally (Agrawal et al., 2018) .

Another approach is to construct compositional generalization split (Johnson et al., 2017; Agrawal et al., 2017) whose question and answer pairs in test set are formed by novel compositions of visual concepts and question types appearing in the training set. This split is constructed by repurposing an existing VQA dataset (Agrawal et al., 2017) or by constructing a synthetic dataset (Johnson et al., 2017) . The problem setting studied in this paper is similar to Teney & Hengel (2016) in the sense that out-of-vocabulary answers are used for testing, but unlike the prior work, we formulate the problem as a transfer learning where out-of-vocabulary answers are learned from external visual data.

External data are often employed in VQA for better generalization. Convolutional neural networks (Krizhevsky et al., 2012; He et al., 2016) pretrained on ImageNet (Deng et al., 2009 ) is a widely accepted standard for diverse VQA models (Yang et al., 2016; Fukui et al., 2016) . As an alternative, object detector (Ren et al., 2015) trained on Visual Genome (Krishna et al., 2017) is employed to extract pretrained visual features (Anderson et al., 2018) . Pretrained language models such as word embeddings (Pennington et al., 2014) or sentence embeddings (Kiros et al., 2015) are frequently used (Right) Pretrained task conditional visual classifier is transferred to VQA with the learned parameters and adapts input representations v φvqa (I, q) and τ ηvqa (q) without fine-tuning.

to initialize parameters of question encoders (Noh et al., 2016; Fukui et al., 2016; Teney et al., 2018) .

Exploiting information retrieval from knowledge base (Auer et al., 2007; Bollacker et al., 2008) or external vision algorithms (Wang et al., 2017b) to provide additional inputs to VQA models was investigated in (Wu et al., 2016; Wang et al., 2017a; b) . Sharing aligned image-word representations between VQA models and image classifiers was proposed in (Gupta et al., 2017) to exploit external visual data. However, transfer learning from external data to cope with out-of-vocabulary answers in VQA has hardly been investigated so far.

Transfer learning from external data to cope with out-of-vocabulary words is actively studied in novel object captioning (Anne Hendricks et al., 2016; Venugopalan et al., 2017; Yao et al., 2017; Lu et al., 2018) . Anne Hendricks et al. (2016) and Venugopalan et al. (2017) decompose image captioning into visual classification and language modeling and exploit unpaired visual and linguistic data as additional resources to train visual classifier and language model respectively. Recent approaches incorporate pointer networks (Vinyals et al., 2015) and learn to point an index of word candidates (Yao et al., 2017) or an associated region (Lu et al., 2018) , where the word candidates are detected by a multi-label classifier (Yao et al., 2017) or an object detector (Lu et al., 2018) trained with external visual data. These algorithms are not directly applicable to our problem setting because they focus on predicting object words without task specification while the task conditional visual recognition is required for VQA.

3. ALGORITHM OVERVIEW:

The main objective of this paper is to learn visual concepts using off-the-shelf visual data and transfer the concepts to VQA models. To adapt the learned visual concepts to diverse questions about a visual scene, we should learn not only a name of a visual concept but also a type of the concept. For example, a question about a dog image can be about name of the animal as well as its visual attributes such as breed and color. We introduce a task conditional visual classifier for the purpose, in which a type or granularity of the concept is specified by a task specification vector and the final answer is an output of the specified recognition task. We pretrain this task conditional visual classifier using visual data without question or task specifications, and adapt it to VQA models by transferring the learned parameters. Figure 2 illustrates overview of the proposed algorithm.

3.1. TASK CONDITIONAL VISUAL CLASSIFIER:

Task conditional visual classifier is a function taking a visual feature v ∈ R d and a task specification vector τ ∈ R k and producing a probability distribution of answers a ∈ [0, 1] l . It is formulated as a neural network with parameter θ, and models a conditional distribution p θ (a|v, τ ). The inputs v and τ are typically constructed by external feature encoders v φ (•) and τ η (•), respectively.

In the proposed transfer learning scenario, a task conditional visual classifier is pretrained with off-the-shelf visual data and transferred to VQA. In the pretraining stage, the model parameter θ and the parameters for external feature encoders φ and η are jointly learned by back-propagation as described in Equation 2. This stage allows a task conditional visual classifier to learn diverse visual recognition tasks p τ (a|v) by varying task specification vector τ . Transfer learning to VQA is achieved by reusing parameter θ and adapting another external feature encoders v φvqa (•) and τ ηvqa (•) to task conditional visual classifier. We first describe transfer learning to VQA in Section 3.2 while pretraining a task conditional visual classifier with off-the-self visual data by unsupervised task discovery is described in Section 4.

3.2. TRANSFER LEARNING FOR VISUAL QUESTION ANSWERING:

As illustrated in Figure 2 , the proposed VQA model contains a task conditional visual classifier p θ (a|v, τ ). The pretrained visual concepts are transferred to VQA by defining the function with the learned parameters θ. Then, learning a VQA model is now formulated as learning input representations v and τ for p θ (a|v, τ ), which is given by

φ * vqa , η * vqa = arg max φvqa,ηvqa

E p vqa (a,I,q) log p θ (a|v φvqa (I, q), τ ηvqa (q)),

where v φvqa (I, q) is an encoded visual feature with an image I and a question q using an attention mechanism with parameter φ vqa , and τ ηvqa (q) is a task specification vector encoded with a question q using an encoder parameterized by η vqa . The p vqa (a, I, q) is a training data distribution of VQA dataset. We learn φ vqa and η vqa by stochastic gradient descent with maximum likelihood objective while the parameter for pretrained task conditional visual classifier θ remains fixed.

Matching visual features:

To reuse pretrained visual classifier in VQA without fine-tuning, visual features v should not be changed. This is fulfilled in recent approaches for VQA that do not fine-tune pretrained visual feature extractors. In this setting, all we need to do is using an identical visual feature extractor for both pretraining and VQA. Specifically, we use attention mechanism (Kim et al., 2016) on a pretrained bottom-up attention features (Anderson et al., 2018) , where attention based on bounding boxes is used for pretraining and question-based attention is employed for VQA.

Weakly supervised task regression Utilizing a pretrained task conditional visual classifier to perform visual recognition specified by a question q requires to infer an task specification vector τ * q . This requirement introduces a learning problem-task regression-that optimizes an encoder τ ηvqa (q) to correctly predict τ * q . Instead of directly minimizing E(τ * q , τ ηvqa (q)) with additional supervision, we exploit VQA data as a source of weak supervision. We propose to optimize a mapping from a visual feature to an answer using an indirect loss denoted by E(p τ * q (a|v), p θ (a|v, τ ηvqa (q))); the resulting training objective is identical to Equation 1.

Out-of-vocabulary answering:

We learn VQA by adapting input representation while fixing the pretrained task conditional visual classifier p θ (a|v, τ ). This strategy allows a model to focus on learning to infer a visual recognition task τ ηvqa (q) from questions, which does not require data for all possible answers. Once the task specification vector τ is inferred, the learned task conditional visual classifier p θ (a|v, τ ) can answer pretrained visual concepts including out-of-vocabulary answers.

4. UNSUPERVISED TASK DISCOVERY:

Learning a task conditional visual classifier with the off-the-shelf visual data (Krishna et al., 2017) is not straightforward due to missing annotation for task specifications. To address this issue, we propose unsupervised task discovery, which exploits a modeled task distribution to sample a task specification instead of collecting additional data. The motivation of this approach is that output domain of a visual recognition task reflects our understanding about hierarchy of concepts or knowledge of the world. For example, classification problems about electronic devices and holdable objects both restricts possible outputs to the corresponding word groups, which are defined by our prior knowledge. This knowledges is often accessible with linguistic knowledge sources, which is used for modeling a task distribution. We describe an algorithm to learn task conditional visual classifier with modeled task distribution in Section 4.1 and discuss how linguistic knowledge sources are used to model a task distribution in Section 4.2. 

EQUATION

where v φpre (r) is a visual feature encoded by a region of interest r given by a whole image or a bounding box annotation, τ ηpre (t) is a task specification vector regressed from a task specification t, and {θ, φ pre , η pre } are model parameters. This objective requires a joint distribution p D (a, r, t), which is modeled by dense annotations of (a, r, t) triples. However, this approach limits the utility of existing large scale visual annotations, which often consist of (a, r) pairs only.

We decompose the joint distribution into two components-one is for visual annotation p V (a, r) and the other is for task conditioned on visual annotation p T (t|a, r)-which is formally given by p D (a, r, t) = p T (t|a, r)p V (a, r).

(3)

Note that this formulation facilitates utilizing existing visual annotations p V (a, r) by appropriately modeling p T (t|a, r). With the decomposed data distribution, the joint distribution for (a, r, t) is approximated by sampling (a, r) from p V (a, r) followed by sampling t from p T (t|a, r). Modeling p T (t|a, r) with linguistic knowledge sources is described next.

4.2. LEVERAGING LINGUISTIC KNOWLEDGE SOURCES:

A visual recognition task defines a mapping from visual inputs to a set of answers, where a range of the mapping is a finite set of answers, which is a subset of a common category set. Especially, when there is no ambiguity about which entity in a visual scene is referenced1 , a visual recognition task is uniquely defined by specifying the range of a mapping. This intuition leads to a simple task modeling approach by treating a task as a set of answers. We exploit linguistic knowledge sources to extract the answer set. Specifically, we consider the following two sources: 1) a structured lexical database called Wordnet (Fellbaum, 1998) and 2) visual descriptions that are provided with a visual data.

Wordnet Wordnet (Fellbaum, 1998 ) is a lexical database represented with a directed acyclic graph of disambiguated word entities, called synsets. The graph represents a hierarchical structure of Wordnet, where the parents of a node correspond to hypernyms of the word in the child. We make a simple assumption that a set of words sharing common ancestors in the graph can construct an answer set in our concept hierarchy. A sample subgraph with extracted answer sets is illustrated in Figure 3 . Given a list of answer sets extracted from Wordnet, a task specification t s is a name of the answer set W ts in the list. A task distribution is modeled by a distribution of answer set p T (t s |a) given an answer a, where a task specification t s is independent of region annotation r. We assign zero probability for answer sets that do not contain the target answer {t s |a ∈ W ts } and uniform probability to all answer sets that contain the answer {t s |a ∈ W ts }, where W ts denotes an answer set named t s . Word embedding is used to encode t s into a task specification vector τ η s pre (t s ). 

5. EXPERIMENTS:

We evaluate how effectively the proposed framework leverages the external data without questions to answer out-of-vocabulary words in visual question answering. We compare the proposed method with baselines equipped with idea from zero-shot image classification (Frome et al., 2013) and novel object captioning (Anne Hendricks et al., 2016; Venugopalan et al., 2017) , which are related to the proposed problem. We also analyze the impact of the external data used for pretraining and visualize the mapping between questions and task specifications learned by weakly supervised task regression.

5.1. DATASETS:

Pretraining We learn visual concepts about most frequently observed 3,000 objects and 1,000 attributes in Visual Genome dataset (Krishna et al., 2017) . We construct external visual data with region bounding box annotations, which are provided with region descriptions. Then, we extract visual words-answer candidates-from region descriptions and construct visual data pairs (r, a). We also construct blanked description (d) from the region description by replacing visual words with <blank>. Note that distribution p(d|a, r) is either zero or one depending on whether the description d is from the region r or not. We use 1,169,708 regions from 80,602 images for training data construction. To use Wordnet (Fellbaum, 1998) , we map visual words to synset using synset annotations from visual genome dataset, and words that are not covered by the annotation are mapped using Textblob (Loria, 2018) python library. Dataset construction We repurpose VQA v2 dataset to construct a train/test split as illustrated in Figure 1 . We use training and validation set of VQA v2. To ensure that every out-of-vocabulary answer appears during pretraining, we select out-of-vocabulary answers from the pretrained visual words. Among 3,813 visual words observed in VQA dataset, we randomly select 954 answers as out-of-vocabulary answers. Since we focus on transferability of visual words, answers about yes/no and numbers are not considered in our evaluation. Based on the selected out-of-vocabulary answers, we split questions into 462,788 training, 51,421 validation, 5,176 test-validation and 20,802 test splits. The training and validation splits do not contain out-of-vocabulary answers while testvalidation and test splits consist of out-of-vocabulary answers only. We plan to make our splits publicly available. Evaluation is performed on the test split using the standard VQA protocol with 10 ground-truth answers annotated for each question (Antol et al., 2015) . The VQA score is given by 100 • min 1 3 10 i=1 1(gt i , â), 1 , where â is a predicted answer, gt i is the i-th ground truth answer and 1(•) is an indicator function.

5.2. BASELINES:

Since leveraging external visual data for visual question answering with out-of-vocabulary answers has hardly been explored, there is no proper evaluation benchmark and we employ the following baselines to compare with the proposed model:

• Answer embedding employs idea from zero-shot image classification (Frome et al., 2013) that learns mapping from visual features to pretrained answer embedding. We use GloVe (Pennington et al., 2014) to embed each answer. • Separable classifier adopts idea from novel object captioning (Anne Hendricks et al., 2016; Venugopalan et al., 2017) that learns visual and language classifier separately and combines them by element-wise sum of logits for joint inference. This baseline and the proposed model are trained with the same data.

5.3. RESULTS:



Model comparisons:

Figure 4 illustrates model comparison results with standard VQA, answer embedding and separable classifier. For this experiment, we perform VQA adaptation with 4 different random seed and plot mean and standard deviation of VQA accuracy. The standard VQA model achieves 0 VQA score because there is no clue for inferring out-of-vocabulary answers. Answer embedding baseline generalizes slightly better by exploiting similarity of answer words in the embedding space. However, we observe very marginal performance improvement because mapping visual features onto answer embedding space is learned with insufficient information available only in the subset of answers. Using off-the-shelf visual data and task specifications from linguistic knowledge sources dramatically improves performance both for the separable classifier baseline and the proposed model. However, independent consideration of visual data and task specifications as in separable classifier has a critical limitation to model joint interaction between task specification and visual features. Especially, this baseline illustrates substantially lower performance on attribute answers, which have significant variations depending on tasks. Note that the bias in the VQA training set cannot be exploited in the proposed evaluation setting, as the evaluation is performed with out-of-distribution answers only. Table 1 : Learned mapping between a question and a task specification. We retrieved questions for each answer sets based on the similarity score between task specification vectors. Results show that appropriate task specifications are regressed from each questions. No explicit supervision is used for mapping between questions and task specifications. Data comparisons Figure 5 illustrates the effect of different linguistic sources on the proposed model. Visual description and Wordnet for the proposed model show complementary characteristics and additional improvement is achieved by their combination. To study the complementary characteristic of visual description and Wordnet, we visualize average VQA score for 50 answers in Figure 6 .

As shown in answers goose and pigeon, Wordnet has advantage on the questions about fine-grained classification such as species of a bird. This is because this categorization is explicitly modeled in the word hierarchy in Wordnet. On the other hand, visual description is more effective on the questions related to type of interactions, for example, holding.

Qualitative results Figure 7 shows prediction of proposed model and the baselines, where Sep-cls denotes separable classifier. The proposed model correctly predicts out-of-vocabulary answers for questions asking diverse visual concepts such as places, styles, species of bird and holiday names.

Weakly supervised task regression Given that answer sets extracted from Wordnet models diverse visual recognition tasks, matching these answer sets to relevant questions is critical for categorization of VQA data and model interpretation. As we learn VQA models by task regression, this matching can be performed by comparing the encoded task specification vector from a question τ ηvqa (q) and the task specification vector of an answer set τ ηpre (t s ). For each τ ηpre (t s ), we sorted questions in a descending order of dot product similarity between τ ηpre (t s ) and τ ηvqa (q). In the sorted question list, top 2 questions whose length fits in a table is visualized in Table 1 . The proposed model is trained with Wordnet data for this experiment. Note that we perform the task regression without explicit supervision on matching between a question and a task specification.

6. CONCLUSION:

We present a transfer learning approach for visual question answering (VQA) with out-of-vocabulary answers. We pretrain a task conditional visual classifier with off-the-shelf visual and linguistic data based on unsupervised task discovery. The pretrained task conditional visual classifier is transferred to VQA adaptively. The experimental results show that exploiting off-the-shelf visual and linguistic data boosts performance in the proposed setting and jointly training the task conditional visual classifier is important to model interaction between visual features and task specifications.

A WEAKLY SUPERVISED TASK REGRESSION:

This section describes how task regression is performed by Equation 2 in the main paper.

Let q be a question from a VQA dataset and τ * q be a true task specification defined by a question. Our intuition is that a task is defined by a mapping from visual inputs to a set of answers, which is represented by a conditional distribution p τ * q (a|v). As the task is defined by a conditional distribution, the objective of a task regression becomes to find a task specification vector τ ηvqa (q) that approximates this conditional distribution using a pretrained task conditional visual classifier p θ (a|v, τ ηvqa (q)). We can formulate this objective as a maximum log-likelihood, which is formally written as follows.

EQUATION

where p(v|q) is conditional distribution of visual features v given a question q.

As we are interested in learning η vqa , which is a parameter of a question encoder working over all question q, we need to optimized the objective expected over the distribution of q, which is formally written as follows.

E p(q) E p(v|q) E τ * q (a|v) [log p θ (a|v, τ ηvqa (q))] = E p(q) E p(v φvqa (I,q)|q) E τ * q (a|v φvqa (I,q)) [log p θ (a|v φvqa (I, q), τ ηvqa (q))] = E p(q) E p(I|q) E τ * q (a|I,q) [log p θ (a|v φvqa (I, q), τ ηvqa (q))] = E p vqa (a,I,q) [log p θ (a|v φvqa (I, q), τ ηvqa (q))],

(5) where, p vqa (a, I, q) = τ * q (a|I, q)p(I|q)p(q) and a visual feature v is altered to v φvqa (I, q) because the visual feature is inferred by a visual encoder from an image I. This derivation relates the task regression to standard VQA objective in Equation 1 in the main paper.

B COMBINING KNOWLEDGE LEARNED BY VQA:

While we focus on learning visual concepts from external visual data, VQA dataset is still a valuable source of learning diverse knowledges. Especially, some answers in the VQA dataset are not visual words and require visual reasoning. For example, yes and no are one of the most frequent answers in the VQA dataset Antol et al. (2015) but it is not straightforward to learn these answers only with the external visual data. Therefore, we consider combining knowledge learned from VQA and from external visual data.

We construct a split of the VQA dataset consisting of 405,228 training, 37,031 validation, 43,171 test-validation and 172 ,681 test questions. The training and validation set does not contain any outof-vocabulary answers and test-validation and test set contains out-of-vocabulary answers. Contrary to the split used in the main paper, this split also contains training answers in the test-validation and test set and the training answers includes logical answers, numbers and visual words. The list of out-of-vocabulary answers are identical to the main paper. Among 172,681 test questions, 103,013 questions could be correctly answered only with the training answers.

To combine knowledge from VQA and external visual data, we construct two task conditional visual classifier and fine-tune one classifier to learn newly appearing answers from VQA training set and fix one classifier to answer known visual answers including out-of-vocabulary answers. After training, we simply combine two logits by element-wise sum and pick answer with largest score for prediction.

The result is illustrated in Figure 8 . We compare the proposed model with the standard VQA model and the answer embedding baseline. Each models are trained with 3 different random seeds and their mean and standard deviation are plotted. Overall, the proposed model performs the best. While the standard VQA model achieved the best performance for training answers, it cannot predict any out-of-vocabulary answers. The answer embedding baseline achieves some generalization to out-of-vocabulary answers, but constraints in the answer embedding degrades its performance on training answers. Table 2 : Learned mapping between a question and a task specification. We retrieved questions for each answer sets based on the similarity score between task specification vectors. Results show that appropriate task specifications are regressed from each questions. No explicit supervision is used for mapping between questions and task specifications. 

C ADDITIONAL TASK REGRESSION RESULTS:

Table 2 shows examples of task regressed questions corresponding to each answers sets, which are represented as the synset of common hypernyms. Similarity is computed by dot product between task specification vector of the answer set embedding τ ηpre (t s ) and the task specification vector regressed by a question τ ηvqa (q). We manually select top ranked questions that fit within the table in terms of the number of characters. 

D ADDITIONAL QUALITATIVE RESULTS:

To illustrate that the proposed model could answer diverse questions with out-of-vocabulary answers, we present additional qualitative results. Figure 9 illustrates that the proposed model correctly predict out-of-vocabulary answers to diverse visual recognition tasks. Figure 10 illustrates that the proposed model performs question and image dependent answering and could predict out-of-vocabulary answers depending on both image and question.

:

In the proposed VQA model, ambiguity of reference is usually resolved by an attention model.

Figures and Tables:

Figure 1: VQA with out-of-vocabulary answers. Given a set of labels in visual data A and a set of answers in VQA training set B, we evaluate the model on VQA test set with answers a ∈ A -B. External visual data provide a set of bounding box labels and visual descriptions.

Figure 2: Overview of the proposed algorithm. (Left) Unsupervised task discovery learns a task conditional visual classifier by leveraging off-the-shelf visual data without task specification t.It also defines a task distribution p T (t|a, r) using linguistic knowledge sources, where stochastic sampling associates a task specification t with a visual annotation (a, r). (Center) A visual annotation with a task specification, denoted by (a, r, t), is employed to pretrain a task conditional visual classifier. (Right) Pretrained task conditional visual classifier is transferred to VQA with the learned parameters and adapts input representations v φvqa (I, q) and τ ηvqa (q) without fine-tuning.

Figure 3: Illustration of Wordnet and extracted answer set. (Left) A subgraph of the Wordnet(Fellbaum, 1998). Complex hierarchy of words reveals the diverse categorization of each words. (Right) A set of words sharing common parents in the tree is grouped as a single answer set. Diverse grouping of words reveals diverse level of understanding about the world. 4.1 PRETRAINING WITH DECOMPOSED DATA DISTRIBUTION Learning task conditional visual classifier is naturally formulated as maximizing expected log likelihood as θ * , φ * pre , η * pre = arg max θ,φpre,ηpre

Figure 4: Model comparisons. Exploiting external data with unsupervised task discovery boosts performance of the proposed model and separable classifier significantly. However, separable classifier showed limited performance gain on attribute answers, which have significant variations depending on tasks.

Figure 5: Data comparisons. Using visual description and Wordnet shows different generalization characteristics and combining them brings additional improvement.

Figure 6: Complementary characteristics of visual description and Wordnet. Wordnet shows advantage on answers related to specific categorizations such as species of a bird (e.g. goose and pigeon) and visual description is more effective on answers about interactions (e.g. holding).

Figure 7: Out-of-vocabulary answers with diverse types of concepts. Green and red color denote correct and wrong answers respectively. Asterisk(*) denotes answers appearing in the training set. Answers without asterisks are out-of-vocabulary answers.

Figure 8: Combining knowledge from VQA and external visual data. Evaluation results on a test set containing both out-of-vocabulary answers and trained answers. The proposed model showed relatively lower performance on trained answers but significantly better performance on out-of-vocabulary answers. In total, the proposed model showed the best performance.

Figure 9: Out-of-vocabulary answers with diverse types of concepts. Green and red color denotes correct and wrong answers respectively. Asterisk(*) denotes answers appearing in the training set. Answers without asterisks are out-of-vocabulary answers. The proposed model correctly predict out-of-vocabulary answers for diverse visual recognition tasks.

name ts answer set Wt s appliance.n.02 freezer, hair dryer, refrigerator, fridge, oven, dishwasher, .. opening.n.10 manhole, rear window, exit, nozzle, car window, coin slot, .. food.n.02 potatoes, french fries, chicken, melted cheese, tomatoes, .. move.v.03 twisted, ruffled, curving, curved, wheeled, rolled, coiled, ..

Table content: <table/>

Answer set ts Questions organic process.n.01 what are the giraffes doing? / what are the animals doing? athletic game.n.01 what type of sport ball is shown? / what type of sport are the men participating in? furniture.n.01 what piece of furniture are the cats sitting on? / what furniture is the cat sitting on? fruit.n.01 what type of fruit is the animal eating? / what type of fruit juice is on the counter? time period.n.01 what kind of season is it? / what type of season is it? tool.n.01 what utensil is in the person 's hand? / what utensil is laying next to the bread? hair.n.01 what hairstyle does the surfer have ? / what type of hairstyle does this man have ?

Table content: <table/>

Answer set ts Questions animal.n.01 what type of animals are near the road? / what species of animals are these? building material.n.01 what kind of material is the flooring made from? / what is the type of flooring made of? meat.n.01 what kind of meat is next to the broccoli? / what kind of meat is next to the veggies? liquid.n.01 what kind of soda are the people drinking? / what type of soda are they drinking ? plant.n.02 what kind of flower is in the tall vase? / what kind of plant leaves are on the plate? plant organ.n.01 what type of fruit is the animal eating? / what kind of fruit is the kid eating? structure.n.01 what type of structure are they in? / what kind of building structure is she in? furniture.n.01 what piece of furniture are the cats sitting on? / what furniture is the cat sitting on? fruit is the animal eating? / what type of fruit juice is on the counter? building.n.01 what kind of building structure is she in? / what type of building is he standing in? hair.n.01 what hairstyle does the surfer have ? / what type of hairstyle does this man have ? communication.n.02 what type of language is on the buildings? / what type of language is on the signs? body of water.n.01 what type of body of water is the man on? / what type of body of water is in photo? tool.n.01 what utensil is in the person 's hand? / what utensil is laying next to the bread? time period.n.01 what kind of season is it? / what type of season is it? appliance.n.02 what kind of appliance is the cat standing in? / what appliance is she standing next to? public transport.n.01 what type of transportation is passing by? / what type of vehicle is shown in the sign? fabric.n.01 what type of fabric are the bears made of? / what type of fabric is the chair made of? tree.n.01 what kind of trees are under all that snow? / what type of trees are the tall ones? beverage.n.01 what kind of soda is on the desk? / what kind of soda is in the bottle? home appliance.n.01 what kind of appliance is the cat standing in? / what appliance is she standing next to? consumer goods.n.01 what clothing item is this person wearing? / what clothing item is the girl wearing? cutlery.n.02 what type of utensil is on the tray? / what utensil is under the fork? edible fruit.n.01 what type of fruit is the animal eating? / what kind of fruit is the purple fruit? shape.n.02 what shape are most of the windows? / what shape is the tall structure to the right? meal.n.01 what meal of the day are these designed for? / what meal are these typically eaten for? sport.n.01 which sport are they doing? / what type of sport ball is shown?

Table content: <table><tr><td>slope.n.01</td><td>is the man going uphill or downhill? / is the bus parked uphill or downhill?</td></tr><tr><td>color.n.01</td><td>what color is the trash bag? / what color is the garbage bag?</td></tr><tr><td>consumption.n.01</td><td>what are the cows doing? / what are the animals doing?</td></tr><tr><td>artifact.n.01</td><td>what object is the cat laying under? / what type of structure are they in?</td></tr><tr><td>fruit.n.01</td><td>what type of</td></tr></table>

Peer review (iclr2019):
 ========== Strengths: ========== - Out-of-vocabulary VQA is an interesting and timely task. - The presented approach leverages region annotations (Visual Genome attributes and objects) and descriptions (COCO Captions) to learn a task-conditioned visual classifier. This seems fairly intuitive and an efficient reuse of existing data to learn visual groundings for out-of-vocabulary words. - Overall the qualitative and quantitative results support the proposed method*s effectiveness. - I really appreciate the extended results in the supplement. - The paper is generally written well with clear descriptions and useful figures. Thanks! - Thank you for using multiple random seeds and reporting variance. ========== Concerns: ========== [A] How is the initial answer space of the task-conditioned classifier set? The dataset construction paragraph is somewhat vague about what a *pretrained visual word* is. Is it the 3000 objects and 1000 attributes from Visual Genome referenced in the pretraining paragraph? [B] The task construction for visual descriptions could be clearer. From my understanding, you select a caption and randomly remove a word, replacing it with the <blank> symbol. This word becomes the answer and the RNN encoded blanked-caption is the task. How is the word selected? [C] Minor point: I think the VQA score listed in 5.1 is incomplete. If I understand correctly, the calculation should be run over all 10 choose 9 sets of answers and the mean reported. [D] Small request: Could the authors add a list of answers in train and test in the supplement? Aside from academic curiosity, I would also like to get a sense for how many of these answer refer to novel concepts / entities / relationships as opposed to being synonyms for existing ones in train. Any attempt to quantify this would also be appreciated! ========== Overview: ========== I*m overall positive on this paper. I think the problem is interesting and the approach leverages existing source of visual grounding in a exciting way. The results in the main paper and supplement paint a convincing picture of the method*s efficacy. There are a couple of things that I would like more clarity on in the submission.