Title: Hadamard Product for Low-rank Bilinear Pooling
Authors: Jin-Hwa Kim
Kyoung-Woon On
Woosang Lim
Jeonghee Kim
Jung-Woo Ha
Byoung-Tak Zhang
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning
Multi-modal learning

Abstract: Bilinear models provide rich representations compared with linear models. They have been applied in various visual tasks, such as object recognition, segmentation, and visual question-answering, to get state-of-the-art performances taking advantage of the expanded representations. However, bilinear representations tend to be high-dimensional, limiting the applicability to computationally complex tasks. We propose low-rank bilinear pooling using Hadamard product for an efficient attention mechanism of multimodal learning. We show that our model outperforms compact bilinear pooling in visual question-answering tasks with the state-of-the-art results on the VQA dataset, having a better parsimonious property.

TL;DR: A new state-of-the-art on the VQA (real image) dataset using an attention mechanism of low-rank bilinear pooling

Review discussion:
Hi,
It would be great if you could clarify your contribution.
You list three in the introduction.
"First, we propose low-rank bilinear pooling to approximate full bilinear pooling to substitute compact bilinear pooling.
Second, Multimodal Low-rank Bilinear Attention Networks (MLB) having an efficient attention mechanism using low-rank bilinear pooling is proposed for visual question-answering tasks. MLB achieves a new state-of-the-art performance, and has a better parsimonious property.
Finally, ablation studies to explore alternative choices, e.g. network depth, non-linear functions, and shortcut connections, are conducted."
However, as the first, element wise multiplication/Hadamard product, you propose as low-rank bilinear pooling has previously widely been used to combine multiple modalities.
E.g. Antol et al. 2015 for VQA, or Xu et al's  "show attend and tell" for image captioning using similar to this work, attention.
The second is using attention for VQA, which is also a widely used technique, e.g. Fukui et al 2016 and others.
Also in combination with attention, hadamard has been used in Xu et al's  "show attend and tell".
I appreciate the extensive experimental evaluation, which is basically the third contribution.
Thanks.
Thanks for sharing your time to review. We sincerely appreciate that.
# Contribution 1: “First, we propose low-rank bilinear pooling to approximate full bilinear pooling to substitute compact bilinear pooling.”
To enable bilinear approximation in deep neural networks, we propose low-rank bilinear pooling, which is effectively and efficiently replaceable with compact bilinear pooling (which is used for multimodal learning in Fukui et al., 2016).
The recent rise of multiplicative methods (Hadamard product) in deep neural networks makes us rethink this computation as an intermediate module of pooling in neural networks. In this paper, we explicitly discuss the role of “Hadamard product” in deep neural networks as low-rank bilinear approximation.
We cited “Antol et al. 2015 for VQA” as Lu et al., 2015 (it’s Github repository) in our paper. Xu et al. (2015) use multilayer perceptron as an attention model, to optimize with variational lower bound on the marginal log-likelihood of word sequence given image features.
# Contribution 2: “Second, Multimodal Low-rank Bilinear Attention Networks (MLB) having an efficient attention mechanism using low-rank bilinear pooling is proposed for visual question-answering tasks. MLB achieves a new state-of-the-art performance, and has a better parsimonious property.”
Yes, there are many attention models for VQA.
Though some of them already used Hadamard product for multimodal representation learning, we argue that this advantage is not merely from Hadamard itself, but from the interpretation of low-rank bilinear pooling. This interpretation leads to, technically, appropriate embeddings for each modality and applying nonlinear activation function. With this, we could propose a novel attention model, MLB. We include a detailed comparison with Fukui et al. (2016) about stability of model structure and use of parameters in Section 7.1.
By the way, Xu et al. (2015) didn’t use Hadamard product in attention model, only in LSTM. If we missed something, please let us know.
Moreover, we include FAQ via personal communications.
# What is different from the element-wise multiplication model of Fukui et al. (2016)?
Since they do not describe the details of the comparative experiment, we don't know why they couldn't get a better result. If they only substitute MCB with element-wise multiplication, appropriate embeddings for both modalities are missed. This condition leads to weak bilinear approximation as our paper noted. Nonlinear activation function for the pooling may get another advantage for the optimization.
# For the nonlinear funtion, why is tanh?
In the prelimary experiments, tanh was the best. We believe tanh is a robust choice for learning except image classification tasks using CNNs (in that case, ReLU).
Also, you can think about activation functions in various RNNs.
Dear Authors,
The paper stresses that low-rank bilinear pooling outperforms compact bilinear pooling.
However, the paper does not explicitly compare the performance of low-rank bilinear pooling vs. compact bilinear pooling, keeping the rest of the model architecture exactly same. It would be great if you could provide experimental results for the same model architecture with the above two types of pooling methods.
Thanks.
Thanks!
This work proposes to approximate the bilinear pooling (outer product) with a formulation which uses the Hadamard Product (element-wise product).
This formulation is evaluated on the visual question answering (VQA) task together with several other model variants.
Strength:
1.	The paper discusses how the Hadamard product can be used to approximate the full outer product.
2.	The paper provides an extensive experimental evaluation of other model aspect for VQA.
3.	The full model archives a slight improvement over prior state-of-the-art on the challenging and large scale VQA challenge.
Weaknesses:
1.	Novelty: The paper presents only a new “interpretation” of the Hadamard product which has previously been widely used for pooling, including for VQA.
2.	Experimental evaluation:
2.1.	An experimental direct comparison with MCB missing.
Although the evaluated model is similar to Fukui et al. several other changes have been made, including question encoding (GRU vs. LSTM), normalization (tanh vs. L2 vs. none). The small difference in performance (0.44% om Table 1) could easily be attributed to these differences.
2.2.	An experimental comparison to the full outer product (e.g. for a lower dimension) is missing. It remains unclear how good the proposed approximation for the full outer product is.
While a comparison to MCB is presented this seems insufficient as MCB is a very different model.
2.3.	One of the most important hyper parameters for the Hadamard Product seems to be the dimension of the lower dimensional embedding d. What effect does changing this have?
2.4.	Comparison with other pooling strategies, e.g. elementwise sum instead of elementwise product.
3.	No theoretical analysis or properties of the approximation are presented.
4.	The paper seems to be general at the beginning, but the claim of the benefit of the Hadamard product is only shown experimentally on the VQA dataset.
5.	Related work: The comparison to the related works in the appendix should at least be mentioned in the main paper, even if the details are the supplemental.
Minor
-	It is not clear why the Lu et al, 2015 is cited rather than the published paper from Antol et al.
-	Sect 2, first sentence: “every pairs” -> “every pair”
Summary:
While the paper provides a new best performance and an interesting interpretation of Hadamard product, to be a strong paper, either a more theoretical analysis of the properties of this approximation is required or a corresponding experimental evaluation. It is a bit unfortunate that most of the experimental evaluation is not about the main claim of the paper (the Hadamard product) but of unrelated aspects which are important to achieve high performance in the VQA challenge.
To be more convincing I would like to see the following experiments
-	Comparison with Outer product in the identical model
-	Comparison with MCB in the identical model
-	Comparison with elementwise sum instead of elementwise product
-	One of the most important hyper parameters for the Hadamard Product seems to be the dimension of the lower dimensional embedding d. What effect does changing this have?
2.1 Please refer to our recent comment.
2.2 Equivalent full outer product cannot be done even for modern GPUs. Note that Fukui et al. (2016) did an experiment with fairly small dimensions, which shows that their approximation is better for generalization.
2.4 With the idea of residual learning, element-wise sum can be interpreted as additive complementary learning with two different models.
However, in multimodal learning, two complementary models compete against with each other to fit the desired function, not considering interactive characteristics of multimodality.
5. You're right, we’ll update that.
Minor. Early version of Antol et al. (2015) didn’t include Lu et al. (2015)’s model. We’ll update the cite. Thanks!
Hi,
I think the additional clarification and experiments speak for accepting the paper and I thus increased my score to 7.
@authors: please also include the direct comparisons to MCB you made here in the forum also to the pdf, thanks.
We will update the pdf soon.
Results on the VQA task are good for this simple model, the ablation study of table 1 gives some insights as to what is important.
Missing are some explanations about the language embedding and the importance in deciding embedding dimension and final output dimension, equivalent to deciding the projected dimension in	the compact bilinear model. Since the main contribution of the
paper seems to be slightly better performance with fairly large reduction in parameters vs. compact bilinear something should be said about choice of those hyper parameters. If you increase embedded and output dimensions to equalize parameters to the compact bilinear model are further gains possible?  How is the question encoded? Is word order preserved in this encoding, the compact bilinear model compared to in table 1 mentions glove, the proposed model is using this as well? The meaning of visual attention in this model along with the number of glimpses should be tied to the sentence embedding, so now we are looking at particular spatial components when that part of the sentence is encoded, then we stack according to your equation 9?
Explanation about the question embedding can be found in Appendix A.1.1, and please refer to our recent comment on that. The output dimension (the number of candidate answers) and joint embedding size is fixed according to Kim et al., (2016b). Our preliminary study shows that the choice of the number of joint embedding size is difficult, since the variance of performance is relatively high, but around 1,000 is fairly good.
For the multiple-glimpse attention mechanism, we get two different attention probability distributions over 14x14 grids using attention mechanism, and concatenating the two different weighted-sum visual features for each distribution, respectively.
Thank your for sharing your time to review, and we humbly appreciate your efforts.
For all reviewers:
For question embedding, we use GRU which have been used in the previous VQA models (Malinowski et al., 2016; Noh et al., 2016a; Kim et al., 2016b).
However, there was conflicted results; Malinowski et al. (2016) show that LSTM is slightly better than GRU, and Noh et al. (2016b) revert their previous choice of GRU (Noh et al., 2016a) into 2-layer LSTMs (via personal communication). The 2-layer LSTMs can be also found in Fukui et al. (2016). We cautiously argue that the choice of RNN for question embedding is up to overall architectural design, which is unlike to solely contribute to the performance gain.
We did not use the concatenation of GloVe word embeddings (Fukui et al, 2016). We are trying to be as simple as possible for model design. This is another reason that we did not take an advantage of question embedding techniques.
For an experiment of the explicit comparison with compact bilinear pooling, we explicitly substitute *compact bilinear pooling* for *low-rank bilinear pooling*. According to Fukui et al. (2016), we use MCB followed by Signed Square Root, L2-Normalization, Dropout(rate=0.1), and Linear projection from 16,000-dimension to the target dimension (one for attention parameters, one for output).
Also, Dropout(rate=0.3) for a question embedding vector. Experiment details are from Fukui et al. (2016)’s implementation from Github repository (https://github.com/akirafukui/vqa-mcb). For test-dev split, our version of MCB (Fukui et al., 2016) model get 61.48% for overall accuracy (yes/no : 82.48%, number : 37.06%, other : 49.07%) vs 65.08% (ours, MLB). The accuracy is not competent as Fukui et al. (2016)’s, especially for other-type. We attribute this result to other hyper-parameters, which should be adjusted to accordingly. Note that this is an explicit comparison with compact bilinear pooling using the same architecture to control other factors, though it maybe not satisfactory to justify the claim.
Rather, we emphasis on the parsimonious property of MLB, which reduces the total number of parameters around 25%, achieving the new state-of-the-art result with reasonable margin.  (We are doing additional experiments to response to your requests, and to double-check if there is any missing critical hyper-parameter in the explicit comparison experiment.)
For an experiment of the explicit comparison with compact bilinear pooling, we explicitly substitute *compact bilinear pooling* for *low-rank bilinear pooling*. According to Fukui et al. (2016), we use MCB followed by Signed Square Root, L2-Normalization, Dropout(rate=0.1), and Linear projection from 16,000-dimension to the target dimension (one for attention parameters, one for output).
Also, Dropout(rate=0.3) for a question embedding vector. Experiment details are from Fukui et al. (2016)’s implementation from Github repository (https://github.com/akirafukui/vqa-mcb). For test-dev split, our version of MCB (Fukui et al., 2016) model get 61.48% for overall accuracy (yes/no : 82.48%, number : 37.06%, other : 49.07%) vs 65.08% (ours, MLB).
Sincerely, we're sorry for your inconvenience.
Could you please clarify the following ---
1. Table 1 in the paper reports 64.7 as the accuracy (overall) for MCB (with Attention and Glove) on test-dev split.
So, why are the accuracies of MLB compared with 61.48?
2. What does 65.08 accuracy refer to -- does it refer to the accuracy of the MCB model with compact bilinear pooling swapped with low-rank bilinear pooling?
1. We do not use GloVe technique, because we think that this is an adhoc to improve the accuracy.
Hence, MCB+Att (64.2) is more appropriate to compare with our model.
However, question embedding method is different from each other: MCB uses stacked two LSTMs, whereas MLB uses single GRU. 61.48 comes from our version of MCB, which constrains the other conditions, keeping the rest of the model architecture exactly same, upon the request.
Though we do not see it as a decisive evidence of better performance, but we think that an optimal architecture may be required for a proposed method.
2.
Except the question embedding above (GRU). In multimodal learning, we swapped compact bilinear pooling with low-rank bilinear pooling.
Overall architecture of multimodal learning part is the same.
Thanks for carrying out the experiments. It is good to know that the explicit comparison of low-rank and compact bilinear pooling shows that low-rank bilinear pooling outperforms compact bilinear pooling. I am changing the review score to "Good Paper, Accept".
That's our duty!
Summary: The paper presents low-rank bilinear pooling that uses Hadamard product (commonly known as element-wise multiplication). The paper implements low-rank bilinear pooling on an existing model (Kim et al., 2016b) and builds a model for Visual Question Answering (VQA) that outperforms the current state-of-art by 0.42%. The paper presents various ablation studies of the new VQA model they built.
Strengths:
1. The paper presents new insights into element-wise multiplication operation which has been previously used in VQA literature (such as Antol et al., ICCV 2015) without insights on why it should work.
2. The paper presents a new model for the task of VQA that beats the current state-of-art by 0.42%.
However, I have concerns about the statistical significance of the performance (see weaknesses below).
3. The various design choices made in model development have been experimentally verified.
Weaknesses/Suggestions:
1.
When authors explicitly (keeping rest of the model architecture same) compared low-rank bilinear pooling with compact bilinear pooling, they found that low-rank bilinear pooling performs worse.
Hence, it could not be experimentally verified that low-rank bilinear pooling is better in performance than compact bilinear pooling (at least for the task of VQA).
2. The authors argue that low-rank bilinear pooling uses 25% less parameters than compact bilinear pooling.
So, could the authors please explain how does the reduction in number of parameters help experimentally? Does the training time of the model reduce significantly? Can we train the model with less data?
3. One of the contributions of the paper is that the proposed model outperforms the current state-of-art on VQA by 0.42%.
However, I am skeptical that the performance of the proposed model is statistically significantly better than the current state-of-art.
4. I would like the authors to explicitly mention the differences between MRN, MARN and MLB. It is not very clear from reading the paper.
5. In the caption for Table 1, fix the following: “have not” -> “have no”
Review Summary: I like the insights about low-rank bilinear pooling using Hadamard product (element-wise multiplication) presented in the paper.
However, it could not be justified that low-rank bilinear pooling leads to better performance than compact biliear pooling. It does lead to reduction in number of parameters but it is not clear how much that helps experimentally.
So, to be more convinced I would like the authors to provide experimental justification of why low-rank bilinear pooling is better than other forms of pooling.
1. Maybe you misunderstand our recent comment. We found that compact bilinear pooling performs worse, not low-rank bilinear pooling, keeping rest of the model architecture of ours. (We updated the previous comment to avoid confusing). If it affects your rating, please reconsider it.
2. Major parameter reduction comes from the difference between the last fully-connected layer as described in Section 7.1. Fukui et al. (2016) should use 16,000-dimension in the middle of networks for their method, compact bilinear pooling.
3. The VQA dataset has 360K questions for training/validation, and 240K for test.
Moreover, for test-standard, we should upload our prediction result to the designated evaluation server with up to five submissions (though we used only two submissions for our paper, one for single model, and the other one for ensemble model). The accuracy variance is empirically around 0.05%.
Hence, we believe 0.42% gain is a significant improvement.
Notice that we attribute a relative huge improvement of the ensemble model from Fukui et al. (2016) in Table 3 to both answer sampling (in our experiment, +0.27%) and data augmentation using visual-genome dataset (in Fukui et al. (2016) +0.70%), which is not used in previous methods. We also use both methods in our ensemble model to compare with Fukui et al. (2016).
4. MRN is from Kim et al. (2016b), MARN is an experimental model which have an attention mechanism in MRN, and MLB, our proposed model, is a simpler model of MARN, which have no shortcut connection.
1.
Regarding parameter reduction: It would be great if you could elaborate on how does parameter reduction help with the training of the model. I understand the cause of the reduction, but I am looking for what effect does it have experimentally on model training -- does it make the model training faster, does it require less data for training the model?
2.
Regarding statistical significance -- statistical significance should be computed by bootstrapping samples multiple times and comparing the min and max accuracies for the two models and see if they overlap  at certain confidence level. This can't be done for VQA test set because the per question accuracies are not available, but can be done for VQA val set. I am concerned about statistical significance, because as shown during the VQA challenge analyses (http://www.visualqa.org/static/slides/vqa_challenge_overview_2016.pptx), 0.42% is not statistically significant generally.
1. A good point for general discussion with regard to parameter reduction. The study of model parsimony in deep learning is especially important for constrained environments.
However, we more focus on to set a strong baseline model with parsimony, simple structure, and better performance for further studies making rooms (gpu memory budget, etc.) for improvements, especially for VQA 2.0, not a dead end study.
2. We think that is a rigorous approach to compare with proposed models.
But, your suggestion embraces a critical weakness: the comparison using a *test-dev* split, which can be used for hyperparameter tuning, is not technically rigorous enough to compare with. Notice that our model comparison is performed for a *test-standard* split in Table 2 & 3.
1. Theoretical analysis on multimodal compact bilinear pooling (MCB) is updated in Section 7.1 and Appendix B.
In short, multimodal compact bilinear pooling reduces the number of parameters of two-way interactions in a three-dimensional tensor using a hashing function [1]. Each parameter is shared by NM/d terms in expectation, with the variance of NM(d-1)d^2 (N, M: input sizes, d: pooled representation size). The total number of bilinear terms is NM. Since our method is three-way factorization instead of two-way, the parsimony of our model is better than MCB, and our model can afford to allocate more parameters in the two-way interactions.
2. In Section 5 Experiments, a description is updated to indicate the details of hyperparameters including preprocessing, question & vision embedding in Appendix A.
3. The effect of joint embedding size d (800 to 1600) is explored in Table 5 (Appendix A).
4. We publish our source code in Github repository (https://github.com/jnhwkim/MulLowBiVQA), and mention it in Section 5.
5. In Section 7, a summarization of related works is updated, and point to Appendix C for details.
6. Correct minor typos and citation reflecting reviewers’ comments.
[1] Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2015). Compressing Neural Networks with the Hashing Trick. In 32nd International Conference on Machine Learning (pp. 2285–2294).
1. Appendix C is inserted to discuss about "the explicit comparison with MCB". We elaborate the previous conversations in the forum.
2. In Appendix A, the batch size is 100, not 200. Apology for your unnecessary confusion. The Github repository is also updated.
3. We update Acknowledgments.

Meta-review:
role: PC
data: comment: The program committee appreciates the authors' response to concerns raised in the reviews. While there are some concerns with the paper that the authors are strongly encouraged to address for the final version of the paper, overall, the work has contributions that are worth presenting at ICLR.

Decision: Accept (Poster)