Title: Area Attention
Authors: Yang Li
Lukasz Kaiser
Samy Bengio
Si Si
Venue: ICLR 2019
Keywords: Deep Learning
attentional mechanisms
neural machine translation
image captioning

Abstract: Existing attention mechanisms, are mostly item-based in that a model is trained to attend to individual items in a collection (the memory) where each item has a predefined, fixed granularity, e.g., a character or a word. Intuitively, an area in the memory consisting of multiple items can be worth attending to as a whole. We propose area attention: a way to attend to an area of the memory, where each area contains a group of items that are either spatially adjacent when the memory has a 2-dimensional structure, such as images, or temporally adjacent for 1-dimensional memory, such as natural language sentences. Importantly, the size of an area, i.e., the number of items in an area or the level of aggregation, is dynamically determined via learning, which can vary depending on the learned coherence of the adjacent items. By giving the model the option to attend to an area of items, instead of only individual items, a model can attend to information with varying granularity. Area attention can work along multi-head attention for attending to multiple areas in the memory. We evaluate area attention on two tasks: neural machine translation (both character and token-level) and image captioning, and improve upon strong (state-of-the-art) baselines in all the cases. These improvements are obtainable with a basic form of area attention that is parameter free. In addition to proposing the novel concept of area attention, we contribute an efficient way for computing it by leveraging the technique of summed area tables.

TL;DR: The paper presents a novel approach for attentional mechanisms that can benefit a range of tasks such as machine translation and image captioning.

Review discussion:
I prefer the idea of using some statistics (such as variances) of multiple items for attention.
This direction may lead to better attention units for future works.
I do not fully understand the argument, "Attention mechanisms are designed to focus on a single item in the entire memory".
In my understanding, the attention formulation has no mathematical bias to focus on a single item.
I have been working on the enterprise NMT for years, and observed many cases where the attention weights concentrate in a few (not a single), tokens.
Do you have any comments?
Could you show some concise examples that we really need to attend multiple (adjacent) items to boost the performance?
For example, in char-based machine translation case, we can mimic the area attention with the wordpiece + token-wise NMT.
For the image case, the adjacent area looks like a "super pixel".
It is unfortunate to observe that the gains on BLEU and perplexity are limited.
Since the authors do not provide any statistical tests, or a confidence interval of the scores,
I cannot be sure these gains are truly significant.
From my experiences +1.0 BLEU score is often insignificant in NMT experiments (BLEU variance is high in general).
Summary
+ A new variant of attention, allowing attention to asses statistics of multiple items (such as variances) is interesting
- Claims are not so much convincing for the need of attending multiple adjacent items.
- Gains in experiments are limited.
Thank you very much for your feedback. We have revised the paper substantially and here address each of your concerns.
Re: “Attention mechanisms are designed to focus on a single item”
This is indeed miscommunication in our original submission. What we intend to convey is that regular attention mechanisms are designed to focus on individual items, i.e., the granularity of attention is each item. Such granularity is predetermined and fixed, e.g., a character or a word-piece.
In contrast, area attention allows a model to attend to information with varying granularity by dynamically grouping adjacent items.
For example, it can be a group of adjacent regions on an image that forms an object or a "super pixel", or multiple word pieces that form a phrase. The difference with regular attention is that we do not have to decide what the proper unit is.
Rather, we let the model pick on the right level of aggregation of items or raw features. Such granularity or aggregation is acquired through learning. We have clarified this point throughout the paper.
# the gains on BLEU and perplexity are limited.
We reported BLEU scores on all the translation tasks including those previously with perplexity. We also added COCO40 Official test results for the image captioning tasks.
Overall, for translation tasks, while the margin is not as large as we hoped, the accuracy gain with area attention is quite consistent throughout most conditions. Particularly on Token-level EN-DE translation, area attention achieved BLEU 29.68 that improved upon the state of the art results with a big margin. For image captioning, the accuracy gain is significant for both in-domain and out-of-domain tests.
We have improved the paper in many ways. Please see the complete list of changes we made in the Summary of Changes and the revised paper for details.
I thank the authors for their efforts in revising paper.
I think I understand the philosophy behind the area attention.  It is possibly a right direction for development of attention tech.
As the other reviewers pointed out, experimental results are remain weak to fully support the necessity of the proposed framework.
Thus I do not change the review score.
While accuracy gains are not always significant, the improvements from area attention are pretty consistently seen across most conditions.
While the baseline models were well tuned by previous work, we didn’t particularly tune each model to better work with area attention. We believe some hyperparameter tuning would result in better accuracy.
For example, we acquired some new results (29.74 for En-De and 41.5 for En-Fr) recently by just tuning on the maximum area size. Tuning other hyperparameters such as attention dropout ratio can help realize area attention’s full potential.
[Summary]
Paper “AREA ATTENTION” extends the current attention models from word level to “area level”, i.e., the combination of adjacent words.
Specifically, every $r_i$ adjacent words are first merged into a new item;
next a key and the value for this item is calculated based on Eqn.(3 or 7) and Eqn. (4), and then the conventional attention models are applied to these new items. The authors work on (char level) NMT and image captioning to verify the algorithm.
[Details]
1.	In the abstract, “… Using an area of items, instead of a single, we hope attention mechanisms can better capture the nature of the task …”, can you provide an example to show why “an area of items” can “better capture the nature of the task”?
In particular, you need to show why the conventional attention mechanism fails.
2.	In this new proposed framework, how should we define the query for each area including multiple items like words?
For example, in Figure 1, what is the query for $n$-item areas where $n=1,2,3$.
3.	Two different kinds of keys are proposed in Eqn. (3) and Eqn. (7). Any comparison between them?
4.	I am not convinced by the experimental results.
(4a) On WMT’14 En-to-Fr and En-to-De, we know that “transformer_big” can achieve better results than the three settings shown in Table 1 & 2. The results of using transformer_big are not reported.
Besides, it is not necessary to use the “tiny” setting for En-to-{De, Fr} translation considering the data size.
(4b) It is widely adopted to use token-level neural machine translation. It is not convincing to work on char-level NMT only.
Also, please provide the results using transformer_big setting.
(4c) There are no BLEU scores for the LSTM setting. Note that comment (4b) and (4c) are also pointed by anonymous readers.
(4d) It is really strange for me to “trained on COCO and tested on Flickr” (See the title of Table 4). It is not a common practice in image captioning literature.
Even if in (Soricut et al., 2018), the authors report the results of training of COCO and test on COCO (the Table 5).
Therefore, the results are not convincing. You should train on COCO and test on COCO too.
e.	What if we use different area size? I do not find the study in this paper.
[Pros & Cons]
(+) A new attempt of the attention model that tries to build the attention beyond unigrams.
(-) Experiments are not convincing.
(-) The motivation is not strong.
Thank you for your detailed comments. We have carefully addressed all your questions in the revision. Please see the Summary of Changes and the revised paper for details. We here respond to each of your questions.
Re: "1. can you provide an example to show why “an area of items” can “better capture the nature of the task”?
In particular, you need to show why the conventional attention mechanism fails."
We miscommunicated our motivation in the original paper. Our motivation is to allow a model to attend to information with varying granularity. Conventional attention mechanisms attend to items with a predefined and fixed granularity, e.g., a word piece of a character in translation or a cell in a fixed grid for image captioning. With area attention, the unit of attention can be a combination of a group of adjacent items.
For example, multiple cells in an image grid can make a car that can be more efficient for the word “car” in the caption to attend to. We have clarified our motivation throughout the paper.
Re: "2. In this new proposed framework, how should we define the query for each area including multiple items like words?
For example, in Figure 1, what is the query for $n$-item areas where $n=1,2,3$."
There is no special requirement on queries. A query can be the same as in a regular attention.
For example, a word in a target sentence can be a translation of a multi-word phrase (an area) in a source sentence.
Re: "3. Two different kinds of keys are proposed in Eqn. (3) and Eqn. (7). Any comparison between them?"
We added a comparison of these two forms of keys in Token-Level Translation Tasks (see Table 1 & 2).
Overall, keys with feature combination (Eqn.5-9) generally performs better than the basic form Eqn.3, although the basic form already outperforms the baselines in most conditions.
Re: "(4a) (4b)"
We added Token-Level Translation tasks in the revised paper (see Section 4.1). We also added a comparison with “Transformer_Big”. Again, area attention consistently improves these models over all the conditions.
In particular, area attention achieved BLEU 29.68 on EN-DE that improved upon the previous result of BLEU 28.4 reported for Transformer Big in the original Transformer paper. For EN-FR, area attention also outperformed the baseline, although it didn’t match the previous result on EN-FR. We think it’s due to the use of a different batch size and training steps.
Re: "(4c) There are no BLEU scores for the LSTM setting. Note that comment (4b) and (4c) are also pointed by anonymous readers."
We reported the BLEU scores for LSTM as well. All the models are tested for both token and character-level translation tasks.
Re: "(4d) It is really strange for me to “trained on COCO and tested on Flickr” (See the title of Table 4). It is not a common practice in image captioning literature.
Even if in (Soricut et al., 2018), the authors report the results of training of COCO and test on COCO (the Table 5).
Therefore, the results are not convincing. You should train on COCO and test on COCO too."
We added COCO40 test results (from the official COCO challenge website) in the revised paper (see Table 5). Area attention significantly improved the original model on both CIDEr and ROUGE-L metrics.
Re: "What if we use different area size? I do not find the study in this paper."
This is a great question. We reported the effect of different area sizes for image captioning tasks, e.g., 2x2 versus 3x3. We explored which layers in Transformer can benefit from area attention and found area attention helps more when it’s used in lower layers. We speculate that this is because each position is well equipped with contextual information due to self attention in latent layers and area attention might not be able to help much.
In contrast, the lower layers can benefit a lot from area attention. We also found a large area size does not necessarily improve accuracy. It is worth tuning the max area size as a hyper parameter to achieve even better results for specific problems.
Thanks a lot for your kind response and most of my questions are answered. This version is more solid than the initial submission.
However, I still have several concerns for this work:
[Motivation]
I am not fully convinced by the current motivation. (1) Without area attention, what is the phenomenon/problem of the current neural machine translation and image captioning? How many cases are suffering from the absence of area attention? Can these problems be solved by carefully tuning the network? (2) After introducing area attention, how the problems are solved? The area attention weights should be visualized. Currently, I do not find the visualization, statistics or examples in this paper.
[Significance]
The improvement on NMT is too minor under the transformer_big setting. In Table 1, the BLEU score of En-De of regular attention is 29.43, and area attention improves it to 29.68; on En-Fr, the improvement is 0.18, which is not significant. Since the ``area attention’’ model needs more parameters, I am not sure whether the improvement is brought by more parameters.
Besides, the improvement of en-fr under transformer_base setting is also small (39.10 v.s. 39.28).
Therefore, we cannot conclude that area attention really works for NMT.
Re: Motivation
We agree that attention visualization would be a good way to show, which we should add to the paper. We feel the need to support areas or ranges has been well demonstrated in a collection of previous works that are suggested by Reviewer1 and discussed by us in the Related Work section. Area attention we proposed here provides a simple and effective solution for enabling areas. We will add concrete examples to the paper.
Re: Significance
On one hand, we agree that Transformer_big is a strong baseline and our gain is not always significant.
On the other hand, we feel area attention provides a simple solution that can be easily applied to many tasks and models that lead to better accuracy. We want to point out that while the baseline models (Transformer_Base & Big) were extensively tuned for the NMT tasks, we did not particularly tune model conditions for area attention. We simply used all the hyperparameters that were tuned for the baselines for area attention experiments. In our recent experiments, by tuning on the maximum area size, we found area attention achieved even better accuracy, e.
g. , 29. 74 for En-De and 41. 5 for En-Fr, which enable a larger gain over the baselines. There are other hyperparameters to tune such as attention dropout ratio, which can be crucial for area attention to realize its potential. Please note that the basic form of area attention (Eq. 3) requires no additional parameters and the feature combination version of area attention (Eq. 5-9) uses very few additional parameters: less than 0. 03% for Transformer Big.
I have several concerns about this paper.
[originality]
Some important related studies are missing.
# Related studies about the perspective of “area”.
The consecutive position in sequence is often referred to as “span” in NLP filed, which is identical to what the authors call “area” in this paper.
Then, the idea of utilizing spans currently becomes a very popular in NLP field. We can find several papers,
e.g.,
Wenhui Wang, Baobao Chang, “Graph-based dependency parsing with bidirectional lstm”, ACL-2016.
Mitchell Stern, Jacob Andreas, Dan Klein, “A Minimal Span-Based Neural Constituency Parser”, ACL-2017.
Kenton Lee, Luheng He, Mike Lewis, Luke Zettlemoyer, “End-to-end Neural Coreference Resolution”, EMNLP-2017.
Nikita Kitaev, Dan Klein, “Constituency Parsing with a Self-Attentive Encoder”, ACL-2018.
Similarly, there are several related studies in image processing field,
e,g.,
Marco Pedersoli, Thomas Lucas, Cordelia Schmid, Jakob Verbeek, “Areas of Attention for image captioning”, ICCV-2017
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo, “Image Captioning with Semantic Attention”, CVPR-2016.
# Related studies about the perspective of “structured attention”.
Several papers about structured attention have already been proposed,
e.g.,
Yoon Kim, Carl Denton, Luong Hoang, Alexander M. Rush. “Structured Attention Networks”, ICLR-2017.
Vlad Niculae, Mathieu Blondel. “A Regularized Framework for Sparse and Structured Neural Attention”, NIPS-2017.
I think the authors should explain the relations between their method and the methods proposed in the above listed papers.
[significance]
# Concern about experimental settings
The experimental setting for NMT looks unnormal in the community.
Currently, most of papers use sentences split in subword units rather than character units. I cannot find a reason to select the character units. I think the authors should report the effectiveness of the proposed method on the widely-used settings.
# computational cost
The authors should report the actual calculation speed by comparing with the baseline method and the proposed method.
In Sec. 2.2, the authors provided the computational cost.
I feel that the cost of O(|M|A) is still enough large and that can unacceptably damage the actual calculation speed of the proposed method.
Overall, the proposed method itself seems to be novel and interesting.
However, in my opinion, writing and organization of this paper should be much improved as a conference paper. I feel like the current status of this paper is still ongoing to write.
Thus, it is a bit hard for me to strongly recommend this paper to be accepted.
Thank you so much for the thorough feedback that really strengthens the paper. We have conducted additional experiments and revised the paper to address these points. We here respond each point in your review.
Re: originality
Thanks for bringing up a great collection of previous works, which helped us better position our work in the literature. We have added the Related Work section to clarify the relationship of area attention with each previous work you suggested.
Re: Concern about experimental settings
We added token-level translation experiments as requested. The results are summarized in Table 1 & Table 2. Area attention consistently outperformed regular attention on token-level translation on all the conditions as well. There is particularly a significant accuracy gain on EN-DE tasks.
Re: computational cost
We reported the actual calculation speed of area attention in comparison to regular attention for two major model configurations (Transformer Base and Big) in section 4.1.1.
Briefly, for Transformer Base model, on 8 NVIDIA P100 GPUs, each training step took 0.4 seconds for Regular Attention, 0.5 seconds for the basic form of Area Attention (Eq.3 & Eq.4), 0.8 seconds for Area Attention using multiple features (Eq.9 & Eq.4).
We have made a number of other improvements to the paper. Please see the summary of changes and the revised paper for details.
Authors properly addressed all of my questions.
I agree that the paper was substantially improved.
I understand the authors' effort.
However, unfortunately, token-level translation experiments did not much support the usefulness of the area attention.
In conclusion, it is a bit hard to "strongly" recommend this paper to be accepted in my opinion.
Thank you for your encouraging remarks. We agree for some conditions our accuracy gain is not as significant.
However, the accuracy improvement has been very consistent across most model conditions when area attention is used. We currently simply used the hyperparameters tuned for the baseline models. We believe some tuning of hyperparameters for area attention can result in better accuracy. In our recent experiments, area attention achieved 29.74 for En-De and 41.5 for En-Fr (a larger margin than previously reported in the revision) by simply tuning on the maximum area size. Tuning on other hyperparameters such as attention dropout ratio can further realize its potential.
We thank the anonymous reviewers and readers for their thoughtful feedback and encouraging remarks. We have substantially improved the paper by making the following changes in the revision.
1. Motivation & Goals
Clarified the purpose of area attention throughout the paper. Area attention allows a model to attend to information with learned, varying granularity and levels of aggregation, which is in contrast to existing attention mechanisms focus on items with predetermined fixed granularity. Please read the revision for details.
2. Related Work
Added a Related Work section to clarify the relationship of area attention with each previous work suggested by Reviewer1 and readers.
3. Presentation
Increased the clarity of the writing throughout the paper, particularly improved the Pseudo code.
4. Experiments
* Token-Level Translation
Added a section for token-level translation experiments as requested by the reviewers and readers. The results are summarized in Table 1 & 2. Area attention consistently outperformed regular attention on token-level translation across all the conditions.
In particular, it improved upon the state-of-art result on EN-DE with a significant margin.
* Actual Cost
Reported the actual calculation speed of area attention in comparison to regular attention for two major model configurations (Transformer Base and Big) in section 4.1.1.
* Comparing Area Attention Keys
Added a comparison of the two forms of area attention keys: the parameter free version (Eq.3) and the feature combination version (Eq.5-9) on all the token-level translation tasks (see Table 1 & 2).
* BLEU for LSTM
Reported BLEU scores for LSTM translation tasks as well (see Table 2 & 4).
* Transformer Big
Added experiments with Transformer Big where area attention has also shown consistent improvements.
* Tests on COCO
Added COCO40 official tests for in-domain image captioning (see Table 5). Area attention outperformed the benchmark model with a significant margin.

Meta-review:
role: AC
data: metareview: although the idea is a straightforward extension of the usual (flat) attention mechanism (which is positive), it does show some improvement in a series of experiments done in this submission. the reviewers however found the experimental results to be rather weak and believe that there may be other problems in which the proposed attention mechanism could be better utilized, despite the authors' effort at improving the result further during the rebuttal period. this may be due to a less-than-desirable form the initial submission was in, and when the new version with perhaps a new set of more convincing experiments is reviewed elsewhere, it may be received with a more positive attitude from the reviewers.

Decision: Reject