Title: HADAMARD PRODUCT FOR LOW-RANK BILINEAR POOLING

Abstract: Bilinear models provide rich representations compared with linear models. They have been applied in various visual tasks, such as object recognition, segmentation, and visual question-answering, to get state-of-the-art performances taking advantage of the expanded representations. However, bilinear representations tend to be high-dimensional, limiting the applicability to computationally complex tasks. We propose low-rank bilinear pooling using Hadamard product for an efficient attention mechanism of multimodal learning. We show that our model outperforms compact bilinear pooling in visual question-answering tasks with the state-of-the-art results on the VQA dataset, having a better parsimonious property.

1. INTRODUCTION:

Bilinear models (Tenenbaum & Freeman, 2000) provide richer representations than linear models. To exploit this advantage, fully-connected layers in neural networks can be replaced with bilinear pooling. The outer product of two vectors (or Kroneker product for matrices) is involved in bilinear pooling, as a result of this, all pairwise interactions among given features are considered. Recently, a successful application of this technique is used for fine-grained visual recognition (Lin et al., 2015) . However, bilinear pooling produces a high-dimensional feature of quadratic expansion, which may constrain a model structure and computational resources. For example, an outer product of two feature vectors, both of which have 1K-dimensionality, produces a million-dimensional feature vector. Therefore, for classification problems, the choice of the number of target classes is severely constrained, because the number of parameters for a standard linear classifier is determined by multiplication of the size of the high-dimensional feature vector and the number of target classes.

Compact bilinear pooling (Gao et al., 2016) reduces the quadratic expansion of dimensionality by two orders of magnitude, retaining the performance of the full bilinear pooling. This approximation uses sampling-based computation, Tensor Sketch Projection (Charikar et al., 2002; Pham & Pagh, 2013) , which utilizes an useful property that Ψ(x ⊗ y, h, s) = Ψ(x, h, s) * Ψ(y, h, s), which means the projection of outer product of two vectors is the convolution of two projected vectors. Here, Ψ is the proposed projection function, and, h and s are randomly sampled parameters by the algorithm.

Nevertheless, compact bilinear pooling embraces two shortcomings. One comes from the sampling approach. Compact bilinear pooling relies on a favorable property, E[ Ψ(x, h, s), Ψ(y, h, s) ] = x, y , which provides a basis to use projected features instead of original features. Yet, calculating the exact expectation is computationally intractable, so, the random parameters, h and s are fixed during training and evaluation. This practical choice leads to the second. The projected dimension of compact bilinear pooling should be large enough to minimize the bias from the fixed parameters. Practical choices are 10K and 16K for 512 and 4096-dimensional inputs, respectively (Gao et al., 2016; Fukui et al., 2016) . Though, these compacted dimensions are reduced ones by two orders of magnitude compared with full bilinear pooling, such high-dimensional features could be a bottleneck for computationally complex models.

We propose low-rank bilinear pooling using Hadamard product (element-wise multiplication), which is commonly used in various scientific computing frameworks as one of tensor operations. The proposed method factors a three-dimensional weight tensor for bilinear pooling into three twodimensional weight matrices, which enforces the rank of the weight tensor to be low-rank. As a result, two input feature vectors linearly projected by two weight matrices, respectively, are computed by Hadamard product, then, followed by a linear projection using the third weight matrix. For example, the projected vector z is represented by

W T z (W T x x • W T y y)

, where • denotes Hadamard product.

We also explore to add non-linearity using non-linear activation functions into the low-rank bilinear pooling, and shortcut connections inspired by deep residual learning (He et al., 2016) . Then, we show that it becomes a simple baseline model (Antol et al., 2015) or one-learning block of Multimodal Residual Networks (Kim et al., 2016b) as a low-rank bilinear model, yet, this interpretation has not be done.

Our contributions are as follows: First, we propose low-rank bilinear pooling to approximate full bilinear pooling to substitute compact bilinear pooling. Second, Multimodal Low-rank Bilinear Attention Networks (MLB) having an efficient attention mechanism using low-rank bilinear pooling is proposed for visual question-answering tasks. MLB achieves a new state-of-the-art performance, and has a better parsimonious property. Finally, ablation studies to explore alternative choices, e.g. network depth, non-linear functions, and shortcut connections, are conducted.

2. LOW-RANK BILINEAR MODEL:

Bilinear models use a quadratic expansion of linear transformation considering every pair of features.

f i = N j=1 M k=1 w ijk x j y k + b i = x T W i y + b i (1)

where x and y are input vectors, W i ∈ R N ×M is a weight matrix for the output f i , and b i is a bias for the output f i . Notice that the number of parameters is L × (N × M + 1) including a bias vector b, where L is the number of output features. Pirsiavash et al. (2009) suggest a low-rank bilinear method to reduce the rank of the weight matrix W i to have less number of parameters for regularization. They rewrite the weight matrix as

W i = U i V T i where U i ∈ R N ×d and V i ∈ R M ×d

, which imposes a restriction on the rank of W i to be at most d ≤ min(N, M ).

Based on this idea, f i can be rewritten as follows:

f i = x T W i y + b i = x T U i V T i y + b i = 1 T (U T i x • V T i y) + b i (2)

where 1 ∈ R d denotes a column vector of ones, and • denotes Hadamard product. Still, we need two third-order tensors, U and V, for a feature vector f , whose elements are {f i }. To reduce the order of the weight tensors by one, we replace 1 with P ∈ R d×c and b i with b ∈ R c , then, redefine as U ∈ R N ×d and V ∈ R M ×d to get a projected feature vector f ∈ R c . Then, we get:

EQUATION

where d and c are hyperparameters to decide the dimension of joint embeddings and the output dimension of low-rank bilinear models, respectively.

3. LOW-RANK BILINEAR POOLING:

A low-rank bilinear model in Equation 3 can be implemented using two linear mappings without biases for embedding two input vectors, Hadamard product to learn joint representations in a multiplicative way, and a linear mapping with a bias to project the joint representations into an output vector for a given output dimension. Then, we use this structure as a pooling method for deep neural networks. Now, we discuss possible variations of low-rank bilinear pooling based on this model inspired by studies of neural networks.

3.1. FULL MODEL:

In Equation 3, linear projections, U and V , can have their own bias vectors. As a result, linear models for each input vectors, x and y, are integrated in an additive form, called as full model for linear regression in statistics:

EQUATION

Here,

U T = diag(b y ) • U T , V T = diag(b x ) • V T , and b = b + P T (b x • b y ).

3.2. NONLINEAR ACTIVATION:

Applying non-linear activation functions may help to increase representative capacity of model. The first candidate is to apply non-linear activation functions right after linear mappings for input vectors.

f = P T σ(U T x) • σ(V T y) + b

(5) where σ denotes an arbitrary non-linear activation function, which maps any real values into a finite interval, e.g. sigmoid or tanh. If two inputs come from different modalities, statistics of two inputs may be quite different from each other, which may result an interference. Since the gradient with respect to each input is directly dependent on the other input in Hadamard product of two inputs.

Additional applying an activation function after the Hadamard product is not appropriate, since activation functions doubly appear in calculating gradients. However, applying the activation function only after the Hadamard product would be alternative choice (We explore this option in Section 5) as follows:

f = P T σ U T x • V T y + b.

(6) Note that using the activation function in low-rank bilinear pooling can be found in an implementation of simple baseline for the VQA dataset (Antol et al., 2015) without an interpretation of low-rank bilinear pooling. However, notably, Wu et al. (2016c) studied learning behavior of multiplicative integration in RNNs with discussions and empirical evidences.

3.3. SHORTCUT CONNECTION:

When we apply two previous techniques, full model and non-linear activation, linear models of two inputs are nested by the non-linear activation functions. To avoid this unfortunate situation, we add shortcut connections as explored in residual learning (He et al., 2016) .

f = P T σ(U T x) • σ(V T y) + h x (x) + h y (y) + b

(7) where h x and h y are shortcut mappings. For linear projection, the shortcut mappings are linear mappings. Notice that this formulation is a generalized form of the one-block layered MRN (Kim et al., 2016b) . Though, the shortcut connections are not used in our proposed model, as explained in Section 6.

4. MULTIMODAL LOW-RANK BILINEAR ATTENTION NETWORKS:

In this section, we apply low-rank bilinear pooling to propose an efficient attention mechanism for visual question-answering tasks, based on the interpretation of previous section. We assumed that inputs are a question embedding vector q and a set of visual feature vectors F over S × S lattice space.

4.1. LOW-RANK BILINEAR POOLING IN ATTENTION MECHANISM:

Attention mechanism uses an attention probability distribution α over S × S lattice space. Here, using low-rank bilinear pooling, α is defined as

EQUATION

where

α ∈ R G×S 2 , P α ∈ R d×G , σ is a hyperbolic tangent function, U q ∈ R N ×d , q ∈ R N , 1 ∈ R S 2 , V F ∈ R M ×d , and F ∈ R S 2 ×M . If G > 1,

multiple glimpses are explicitly expressed as in Fukui et al. (2016) , conceptually similar to Jaderberg et al. (2015) . And, the softmax function applies to each row vector of α. The bias terms are omitted for simplicity.

Attended visual feature v is a linear combination of F i with coefficients α g,i . Each attention probability distribution α g is for a glimpse g.

For G > 1, v is the concatenation of resulting vectors vg as v = G g=1 S 2 s=1 α g,s F s (9)

where denotes concatenation of vectors. The posterior probability distribution is an output of a softmax function, whose input is the result of another low-rank bilinear pooling of q and v as

EQUATION

where â denotes a predicted answer, Ω is a set of candidate answers and Θ is an aggregation of entire model parameters.

5. EXPERIMENTS:

In this section, we conduct six experiments to select the proposed model, Multimodal Low-rank Bilinear Attention Networks (MLB). Each experiment controls other factors except one factor to assess the effect on accuracies. Based on MRN (Kim et al., 2016b) , we start our assessments with an initial option of G = 1 and shortcut connections of MRN, called as Multimodal Attention Residual Networks (MARN). Notice that we use one embeddings for each visual feature for better performance, based on our preliminary experiment (not shown). We attribute this choice to the attention mechanism for visual features, which provides more capacity to learn visual features. We use the same hyper-parameters of MRN (Kim et al., 2016b) , without any explicit mention of this.

The VQA dataset (Antol et al., 2015) is used as a primary dataset, and, for data augmentation, question-answering annotations of Visual Genome (Krishna et al., 2016) However, we speculate that an introduction of attention mechanism makes deep networks hard to optimize. Therefore, we explore the number of learning blocks of MARN, which have an attention mechanism using low-rank bilinear pooling. Fukui et al. (2016) show that the attention mechanism of two glimpses was an optimal choice. In a similar way, we assess one, two, and four-glimpse models. Non-Linearity We assess three options applying non-linearity on low-rank bilinear pooling, vanilla, before Hadamard product as in Equation 5, and after Hadamard product as in Equation 6.

Number of Glimpses:

Answer Sampling VQA (Antol et al., 2015) dataset has ten answers from unique persons for each question, while Visual Genome (Krishna et al., 2016) dataset has a single answer for each question.

Since difficult or ambiguous questions may have divided answers, the probabilistic sampling from the distribution of answers can be utilized to optimize for the multiple answers. An instance2 can be found in Fukui et al. (2016) . We simplify the procedure as follows:

EQUATION

where |a i | denotes the number of unique answer a i in a set of multiple answers, a 0 denotes a mode, which is the most frequent answer, and a 1 denotes the secondly most frequent answer. We define the divided answers as having at least three answers which are the secondly frequent one, for the evaluation metric of VQA (Antol et al., 2015) , The rate of the divided answers is approximately 16.40%, and only 0.23% of questions have more than two divided answers in VQA dataset. We assume that it eases the difficulty of convergence without severe degradation of performance.

EQUATION

)

Shortcut Connection:

The contribution of shortcut connections for residual learning is explored based on the observation of the competitive performance of single-block layered model. Since the usefulness of shortcut connections is linked to the network depth (He et al., 2016) .

Data Augmentation:

The data augmentation with Visual Genome (Krishna et al., 2016) question answer annotations is explored. Visual Genome (Krishna et al., 2016) originally provides 1.7 Million visual question answer annotations. After aligning to VQA, the valid number of question-answering pairs for training is 837,298, which is for distinct 99,280 images.

6. RESULTS:

The six experiments are conducted sequentially. Each experiment determines experimental variables one by one. Refer to Table 1 , which has six sectors divided by mid-rules.

6.1. SIX EXPERIMENT RESULTS:

Number of Learning Blocks Though, MRN (Kim et al., 2016b) has the three-block layered architecture, MARN shows the best performance with two-block layered models (63.92%). For the multiple glimpse models in the next experiment, we choose one-block layered model for its simplicity to extend, and competitive performance (63.79%).

Compared with the results of Fukui et al. (2016) , four-glimpse MARN (64.61%) is better than other comparative models. However, for a parsimonious choice, two-glimpse MARN (64.53%) is chosen for later experiments. We speculate that multiple glimpses are one of key factors for the competitive performance of MCB (Fukui et al., 2016) , based on a large margin in accuracy, compared with one-glimpse MARN (63.79%).

Non-Linearity:

The results confirm that activation functions are useful to improve performances. Surprisingly, there is no empirical difference between two options, before-Hadamard product and after-Hadamard product. This result may build a bridge to relate with studies on multiplicative integration with recurrent neural networks (Wu et al., 2016c) .

Answer Sampling Sampled answers (64.80%) result better performance than mode answers (64.53%). It confirms that the distribution of answers from annotators can be used to improve the performance. However, the number of multiple answers is usually limited due to the cost of data collection.

Shortcut Connection Though, MRN (Kim et al., 2016b) effectively uses shortcut connections to improve model performance, one-block layered MARN shows better performance without the shortcut connection. In other words, the residual learning is not used in our proposed model, MLB.

It seems that there is a trade-off between introducing attention mechanism and residual learning. We leave a careful study on this trade-off for future work.

Data Augmentation Data augmentation using Visual Genome (Krishna et al., 2016) question answer annotations significantly improves the performance by 0.76% in accuracy for VQA test-dev split. Especially, the accuracy of others (ETC)-type answers is notably improved from the data augmentation.

6.2. COMPARISON WITH STATE-OF-THE-ART:

The comparison with other single models on VQA test-standard is shown in Table 2 . The overall accuracy of our model is approximately 1.9% above the next best model (Noh & Han, 2016) on the Open-Ended task of VQA. The major improvements are from yes-or-no (Y/N) and others (ETC)type answers. In Table 3 , we also report the accuracy of our ensemble model to compare with other ensemble models on VQA test-standard, which won 1st to 5th places in VQA Challenge 20163 . We beat the previous state-of-the-art with a margin of 0.42%. 

7. RELATED WORKS:

MRN (Kim et al., 2016b) proposes multimodal residual learning with Hadamard product of low-rank bilinear pooling. However, their utilization of low-rank bilinear pooling is limited to joint residual mapping function for multimodal residual learning. Higher-order Boltzmann Machines (Memisevic & Hinton, 2007; 2010) use Hadamard product to capture the interactions of input, output, and hidden representations for energy function. Wu et al. (2016c) propose the recurrent neural networks using Hadamard product to integrate multiplicative interactions among hidden representations in the model. For details of these related works, please refer to Appendix D.

Yet, compact bilinear pooling or multimodal compact bilinear pooling (Gao et al., 2016; Fukui et al., 2016) is worth to discuss and carefully compare with our method.

7.1. COMPACT BILINEAR POOLING:

Compact bilinear pooling (Gao et al., 2016) approximates full bilinear pooling using a samplingbased computation, Tensor Sketch Projection (Charikar et al., 2002; Pham & Pagh, 2013) :

EQUATION

= FFT -1 (FFT(Ψ(x, h, s) • FFT(Ψ(y, h, s)) (16) where ⊗ denotes outer product, * denotes convolution, Ψ(v, h, s) i := j:hj =i s j • v j , FFT denotes Fast Fourier Transform, d denotes an output dimension, x, y, h, s ∈ R n , x and y are inputs, and h and s are random variables. h i is sampled from {1, ..., d}, and s i is sampled from {-1, 1}, then, both random variables are fixed for further usage. Even if the dimensions of x and y are different from each other, it can be used for multimodal learning (Fukui et al., 2016) .

Similarly to Equation 1, compact bilinear pooling can be described as follows:

f i = x T W i y (17) where W ijk = s ijk w ijk if s ijk is sampled from {-1, 1},

w ijk is sampled from {P i1 , P i2 , . . . , P id }, and the compact bilinear pooling is followed by a fully connected layer P ∈ R |Ω|×d . Then, this method can be formulated as a hashing trick (Weinberger et al., 2009; Chen et al., 2015) to share randomly chosen bilinear weights using d parameters for a output value, in a way that a single parameter is shared by N M/d bilinear terms in expectation, with the variance of N M (d -1)/d 2 (See Appendix B).

In comparison with our method, their method approximates a three-dimensional weight tensor in bilinear pooling with a two-dimensional matrix P, which is larger than the concatenation of three two-dimensional matrices for low-rank bilinear pooling. The ratio of the number of parameters for a single output to the total number of parameters for |Ω| outputs is d/d|Ω| = 1/|Ω| (Fukui et al., 2016) , vs. d(N + M + 1)/d(N + M + |Ω|) = (N + M + 1)/(N + M + |Ω|) ≈ 2/3 (ours), since our method uses a three-way factorization. Hence, more parameters are allocated to each bilinear approximation than compact bilinear pooling does, effectively managing overall parameters guided by back-propagation algorithm.

MCB (Fukui et al., 2016) , which uses compact bilinear pooling for multimodal tasks, needs to set the dimension of output d to 16K, to reduce the bias induced by the fixed random variables h and s. As a result, the majority of model parameters (16K × 3K = 48M) are concentrated on the last fully connected layer, which makes a fan-out structure. So, the total number of parameters of MCB is highly sensitive to the number of classes, which is approximately 69.2M for MCB+att, and 70.5M for MCB+att+GloVe. Yet, the total number of parameters of our proposed model (MLB) is 51.9M, which is more robust to the number of classes having d = 1.2K, which has a similar role in model architecture.

8. CONCLUSIONS:

We suggest a low-rank bilinear pooling method to replace compact bilinear pooling, which has a fan-out structure, and needs complex computations. Low-rank bilinear pooling has a flexible structure using linear mapping and Hadamard product, and a better parsimonious property, compared with compact bilinear pooling. We achieve new state-of-the-art results on the VQA dataset using a similar architecture of Fukui et al. (2016) , replacing compact bilinear pooling with low-rank bilinear pooling. We believe our method could be applicable to other bilinear learning tasks. In this section, the algorithm of multimodal compact bilinear pooling (MCB) (Gao et al., 2016; Fukui et al., 2016) is described as a kind of hashing tick (Chen et al., 2015) .

x ∈ R nx and y ∈ R ny are the given inputs, Φ(x, y) ∈ R d is the output. Random variables hx ∈ N nx and hy ∈ N ny are uniformly sampled from {1, . . . , d}, and sx ∈ Z nx and sy ∈ Z ny are uniformly sampled from {-1, 1}. Then, Count Sketch projection function Ψ (Charikar et al., 2002) projects x and y to intermediate representations Ψ(x, hx, sx) ∈ R d and Ψ(y, hy, sy) ∈ R d , which is defined as:

EQUATION

Notice that both h and s remain as constants after initialization (Fukui et al., 2016) .

The probability of hxj = i and hyj = i for the given j is 1/d 2 . Hence, the expected number of bilinear terms in Ψ(x, hx, sx)iΨ(y, hy, sy)i is (nxny)/d 2 . Since, the output Φ(x, y) is a result of circular convolution of Ψ(x, hx, sx) and Ψ(y, hy, sy), the expected number of bilinear terms in Φ(x, y)i is (nxny)/d. Likewise, the probability of that a bilinear term is allocated in Φ(x, y)i is 1/d. The probability distribution of the number of bilinear terms in Φ(x, y)i follows a multinomial distribution, whose mean is (nxny)/d and variance is (nxny)(d -1)/d 2 .

Linear projection after the multimodal compact bilinear pooling provides weights on the bilinear terms, in a way that a shared weight is assigned to Φ(x, y)i, which has (nxny)/d bilinear terms in expectation, though each bilinear term can have a different sign induced by both sx and sy.

HashedNets (Chen et al., 2015) propose a method to compress neural networks using a low-cost hashing function (Weinberger et al., 2009) , which is the same function of Ψ(v, h, s). They randomly group a portion of connections in neural networks to share a single weight. We speculate that multimodal compact bilinear pooling uses the hashing tick to reduce the number of full bilinear weights with the rate of d/(nxny). However, this approximation is limited to two-way interaction, compared with three-way factorization in our method.

C REPLACEMENT OF LOW-RANK BILINEAR POOLING:

For the explicit comparison with compact bilinear pooling, we explicitly substitute compact bilinear pooling for low-rank bilinear pooling to control everything else, which means that the rest of the model architecture is exactly the same.

According to Fukui et al. (2016) , we use MCB followed by Signed Square Root, L2-Normalization, Dropout (p=0.1), and linear projection from 16,000-dimension to the target dimension. Also, Dropout (p=0.3) for a question embedding vector. Note that an overall architecture for multimodal learning of both is the same. Experimental details are referenced from the implementation4 of Fukui et al. (2016) .

For test-dev split, our version of MCB gets 61.48% for overall accuracy (yes/no: 82.48%, number: 37.06%, and other: 49.07%) vs. 65.08% (ours, MLB in Table 1 ). Additionally, if the nonlinearity in getting attention distributions is increased as the original MCB does using ReLU, we get 62.11% for overall accuracy (yes/no: 82.55%, number: 37.18%, and other: 50.30%), which is still the below of our performance5 .

We do not see it as a decisive evidence of the better performance of MLB, but as a reference (the comparison of test-dev results may be also unfair.), since an optimal architecture and hyperparameters may be required for each method.

D RELATED WORKS D.1 MULTIMODAL RESIDUAL NETWORKS:

MRN (Kim et al., 2016b) is an implicit attentional model using multimodal residual learning with Hadamard product which does not have any explicit attention mechanism.

EQUATION

HL(q, v) = W q q + L l=1

EQUATION

where W * are parameter matrices, L is the number of learning blocks, H0 = q, W q = Π L l=1 W (l)

q , and W F (l) = Π L m=l+1 W

(m)

q . Notice that these equations can be generalized by Equation 7. However, an explicit attention mechanism allows the use of lower-level visual features than fully-connected layers, and, more importantly, spatially selective learning. Recent state-of-the-art methods use a variant of an explicit attention mechanism in their models (Lu et al., 2016; Noh & Han, 2016; Fukui et al., 2016) . Note that shortcut connections of MRN are not used in the proposed Multimodal Low-rank Bilinear (MLB) model. Since, it does not have any performance gain due to not stacking multiple layers in MLB. We leave the study of residual learning for MLB for future work, which may leverage the excellency of bilinear models as suggested in Wu et al. (2016a) .

D.2 HIGHER-ORDER BOLTZMANN MACHINES:

A similar model can be found in a study of Higher-Order Boltzmann Machines (Memisevic & Hinton, 2007; 2010) . They suggest a factoring method for the three-way energy function to capture correlations among input, output, and hidden representations.

-E(y, h; x) =

f i xiw x if j yjw y jf k h k w h kf + k w h k h k + j w y j yj = x T W x • y T W y • h T W h 1 + h T w h + y T w y (21)

Setting aside of bias terms, the I × J × K parameter tensor of unfactored Higher-Order Boltzmann Machines is replaced with three matrices, W x ∈ R I×F , W y ∈ R J×F , and W h ∈ R K×F .

D.3 MULTIPLICATIVE INTEGRATION WITH RECURRENT NEURAL NETWORKS:

Most of recurrent neural networks, including vanilla RNNs, Long Short Term Memory networks (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (Cho et al., 2014) , share a common expression as follows:

EQUATION

)

where φ is a non-linear function, W ∈ R d×n , x ∈ R n , U ∈ R d×m , h ∈ R m , and b ∈ R d is a bias vector. Note that, usually, x is an input state vector and h is an hidden state vector in recurrent neural networks. Wu et al. (2016c) propose a new design to replace the additive expression with a multiplicative expression using Hadamard product as

EQUATION

Moreover, a general formulation of this multiplicative integration can be described as

EQUATION

which is reminiscent of full model in Section 3.1.

:

https://github.com/jnhwkim/MulLowBiVQA

https://github.com/akirafukui/vqa-mcb/blob/5fea8/train/multi_att_2_ glove/vqa_data_provider_layer.py#L130

http://visualqa.org/challenge.html

https://github.com/akirafukui/vqa-mcb

Our version of MCB definition can be found in https://github.com/jnhwkim/MulLowBiVQA/ blob/master/netdef/MCB.lua

Figures and Tables:

Figure1: A schematic diagram of MLB. Replicate module copies an question embedding vector to match with S 2 visual feature vectors. Conv modules indicate 1 × 1 convolution to transform a given channel space, which is computationally equivalent to linear projection for channels.

are used. Validation is performed on the VQA test-dev split, and model comparison is based on the results of the VQA test-standard split. For the comprehensive reviews of VQA tasks, please refer toWu et al. (2016a)   andKafle & Kanan (2016a). The details about preprocessing, question and vision embedding, and hyperparameters used in our experiments are described in Appendix A. The source code for the experiments is available in Github repository 1 .

Table content: <table><tr><td>Number of Learning Blocks Kim et al. (2016b) argue that three-block layered MRN shows the</td></tr><tr><td>best performance among one to four-block layered models, taking advantage of residual learning.</td></tr></table>

The accuracies of our experimental model, Multimodal Attention Residual Networks (MARN), with respect to the number of learning blocks (L#), the number of glimpse (G#), the position of activation functions (tanh), answer sampling, shortcut connections, and data augmentation using Visual Genome dataset, for VQA test-dev split and Open-Ended task. Note that our proposed model, Multimodal Low-rank Bilinear Attention Networks (MLB) have no shortcut connections, compared with MARN. MODEL: model name, SIZE: number of parameters, ALL: overall accuracy in percentage, Y/N: yes/no, NUM: numbers, and ETC: others. SinceFukui et al. (2016) only report the accuracy of the ensemble model on the test-standard, the test-dev results of their single models are included in the last sector. Some figures have different precisions which are rounded. * indicates the selected model for each experiment.

Table content: <table><tr><td>MODEL</td><td>SIZE ALL</td><td colspan="3">Y/N NUM ETC</td></tr><tr><td colspan="5">MRN-L3 65.0M 61.68 82.28 38.82 49.25</td></tr><tr><td colspan="5">MARN-L3 65.5M 62.37 82.31 38.06 50.83</td></tr><tr><td colspan="5">MARN-L2 56.3M 63.92 82.88 37.98 53.59</td></tr><tr><td colspan="5">* MARN-L1 47.0M 63.79 82.73 37.92 53.46</td></tr><tr><td colspan="5">MARN-L1-G1 47.0M 63.79 82.73 37.92 53.46</td></tr><tr><td colspan="5">* MARN-L1-G2 57.7M 64.53 83.41 37.82 54.43</td></tr><tr><td colspan="5">MARN-L1-G4 78.9M 64.61 83.72 37.86 54.33</td></tr><tr><td colspan="5">No Tanh 57.7M 63.58 83.18 37.23 52.79</td></tr><tr><td colspan="5">* Before-Product 57.7M 64.53 83.41 37.82 54.43</td></tr><tr><td colspan="5">After-Product 57.7M 64.53 83.53 37.06 54.50</td></tr><tr><td colspan="5">Mode Answer 57.7M 64.53 83.41 37.82 54.43</td></tr><tr><td colspan="5">* Sampled Answer 57.7M 64.80 83.59 38.38 54.73</td></tr><tr><td colspan="5">Shortcut 57.7M 64.80 83.59 38.38 54.73</td></tr><tr><td colspan="5">* No Shortcut 51.9M 65.08 84.14 38.21 54.87</td></tr><tr><td colspan="5">MLB 51.9M 65.08 84.14 38.21 54.87</td></tr><tr><td colspan="5">MLB+VG 51.9M 65.84 83.87 37.87 56.76</td></tr><tr><td colspan="2">MCB+Att (Fukui et al., 2016) 69.2M 64.2</td><td>82.2</td><td>37.7</td><td>54.8</td></tr><tr><td colspan="2">MCB+Att+GloVe (Fukui et al., 2016) 70.5M 64.7</td><td>82.5</td><td>37.6</td><td>55.6</td></tr><tr><td colspan="2">MCB+Att+Glove+VG (Fukui et al., 2016) 70.5M 65.4</td><td>82.3</td><td>37.2</td><td>57.4</td></tr></table>

The VQA test-standard results to compare with state-of-the-art. Notice that these results are trained by provided VQA train and validation splits, without any data augmentation.

Table content: <table><tr><td>Open-Ended</td><td>MC</td></tr></table>

The VQA test-standard results for ensemble models to compare with state-of-the-art. For unpublished entries, their team names are used instead of their model names. Some of their figures are updated after the challenge.

Table content: <table><tr><td/><td/><td>Open-Ended</td><td>MC</td></tr><tr><td>MODEL</td><td>ALL</td><td colspan="2">Y/N NUM ETC ALL</td></tr><tr><td>RAU (Noh &amp; Han, 2016)</td><td colspan="3">64.12 83.33 38.02 53.37 67.34</td></tr><tr><td>MRN (Kim et al., 2016b)</td><td colspan="3">63.18 83.16 39.14 51.33 67.54</td></tr><tr><td>DLAIT (not published)</td><td colspan="3">64.83 83.23 40.80 54.32 68.30</td></tr><tr><td colspan="4">Naver Labs (not published) 64.79 83.31 38.70 54.79 69.26</td></tr><tr><td>MCB (Fukui et al., 2016)</td><td colspan="3">66.47 83.24 39.47 58.00 70.10</td></tr><tr><td>MLB (ours)</td><td colspan="3">66.89 84.61 39.07 57.79 70.29</td></tr><tr><td colspan="4">Human (Antol et al., 2015) 83.30 95.77 83.39 72.67 91.54</td></tr></table>

The effect of joint embedding size d.

Table content: <table><tr><td/><td>Open-Ended</td></tr><tr><td>d SIZE ALL</td><td>Y/N NUM ETC</td></tr><tr><td colspan="2">800 45.0M 64.89 84.08 38.15 54.55</td></tr><tr><td colspan="2">1000 48.4M 65.06 84.18 38.01 54.85</td></tr><tr><td colspan="2">1200 51.9M 65.08 84.14 38.21 54.87</td></tr><tr><td colspan="2">1400 55.4M 64.94 84.13 38.00 54.64</td></tr><tr><td colspan="2">1600 58.8M 65.02 84.15 37.79 54.85</td></tr><tr><td>Q</td><td>V</td></tr><tr><td>Linear</td><td>Conv</td></tr><tr><td>Tanh</td><td>Tanh</td></tr><tr><td>Replicate</td><td/></tr><tr><td>Linear</td><td>Conv</td></tr><tr><td>Tanh</td><td>Softmax</td></tr><tr><td>Linear</td><td>Linear</td></tr><tr><td>Softmax</td><td>Tanh</td></tr><tr><td>A</td><td/></tr></table>

The individual models used in our ensemble model in Table3.

Table content: <table><tr><td/><td/><td/><td>Open-Ended</td></tr><tr><td colspan="3">MODEL GLIMPSE ALL</td><td>Y/N NUM ETC</td></tr><tr><td>MLB</td><td>2</td><td colspan="2">64.89 84.13 37.85 54.57</td></tr><tr><td>MLB</td><td>2</td><td colspan="2">65.08 84.14 38.21 54.87</td></tr><tr><td>MLB</td><td>4</td><td colspan="2">65.01 84.09 37.66 54.88</td></tr><tr><td>MLB-VG</td><td>2</td><td colspan="2">65.76 83.64 37.57 56.86</td></tr><tr><td>MLB-VG</td><td>2</td><td colspan="2">65.84 83.87 37.87 56.76</td></tr><tr><td>MLB-VG</td><td>3</td><td colspan="2">66.05 83.88 38.13 57.13</td></tr><tr><td>MLB-VG</td><td>4</td><td colspan="2">66.09 83.59 38.32 57.42</td></tr><tr><td>Ensemble</td><td>-</td><td colspan="2">66.77 84.54 39.21 57.81</td></tr></table>

Peer review (iclr2017):
This work proposes to approximate the bilinear pooling (outer product) with a formulation which uses the Hadamard Product (element-wise product). This formulation is evaluated on the visual question answering (VQA) task together with several other model variants. Strength: 1. The paper discusses how the Hadamard product can be used to approximate the full outer product. 2. The paper provides an extensive experimental evaluation of other model aspect for VQA. 3. The full model archives a slight improvement over prior state-of-the-art on the challenging and large scale VQA challenge. Weaknesses: 1. Novelty: The paper presents only a new “interpretation” of the Hadamard product which has previously been widely used for pooling, including for VQA. 2. Experimental evaluation: 2.1. An experimental direct comparison with MCB missing. Although the evaluated model is similar to Fukui et al. several other changes have been made, including question encoding (GRU vs. LSTM), normalization (tanh vs. L2 vs. none). The small difference in performance (0.44% om Table 1) could easily be attributed to these differences. 2.2. An experimental comparison to the full outer product (e.g. for a lower dimension) is missing. It remains unclear how good the proposed approximation for the full outer product is. While a comparison to MCB is presented this seems insufficient as MCB is a very different model. 2.3. One of the most important hyper parameters for the Hadamard Product seems to be the dimension of the lower dimensional embedding d. What effect does changing this have? 2.4. Comparison with other pooling strategies, e.g. elementwise sum instead of elementwise product. 3. No theoretical analysis or properties of the approximation are presented. 4. The paper seems to be general at the beginning, but the claim of the benefit of the Hadamard product is only shown experimentally on the VQA dataset. 5. Related work: The comparison to the related works in the appendix should at least be mentioned in the main paper, even if the details are the supplemental. Minor - It is not clear why the Lu et al, 2015 is cited rather than the published paper from Antol et al. - Sect 2, first sentence: “every pairs” -> “every pair” Summary: While the paper provides a new best performance and an interesting interpretation of Hadamard product, to be a strong paper, either a more theoretical analysis of the properties of this approximation is required or a corresponding experimental evaluation. It is a bit unfortunate that most of the experimental evaluation is not about the main claim of the paper (the Hadamard product) but of unrelated aspects which are important to achieve high performance in the VQA challenge. To be more convincing I would like to see the following experiments - Comparison with Outer product in the identical model - Comparison with MCB in the identical model - Comparison with elementwise sum instead of elementwise product - One of the most important hyper parameters for the Hadamard Product seems to be the dimension of the lower dimensional embedding d. What effect does changing this have?