QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

Abstract:  Current end-to-end machine reading and question answering (Q\&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs. We propose a new Q\&A architecture called QANet, which does not require recurrent networks:  Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models. The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model. 
      On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score on the test set, which is significantly better than the best published F1 score of 81.8.

Reviews and author responses:

Thank you for your paper! We really liked your approach to accelerate inference and training times in QA. 
I have one question regarding the comparison with BiDAF. On the article, you mention that you batched the training examples by paragraph length in your model, but it is not clear whether you did the same for BiDAF (the implementation on GitHub offers the flags --cluster and --len_opt for that).
That is an important consideration because that change alone has a significant impact on training and inference times. In fact, by batching the inputs in that way we have fully trained BiDAF in 6h30 (12 epochs) on an NVidia Titan X, which is more than two times faster than your reported time of 15h on a more powerful P100.
Could you please clarify this point in the article?

The performance of the model on SQuAD dataset is impressive. In addition to the performance on the test set, we are also interested in the sample complexity of the proposed model. Currently, the SQuAD dataset splits the collection of passages into a training set, a development set, and a test set in a ratio of 80%:10%:10% where the test set is not released. Given the released training and dev set, we are wondering what would happen if we split the data in a different ratio, for example, 50% for training and the rest 50% for dev. We will really appreciate it if the authors can report the model performance (on training/dev respectively) under this scenario.  


I am happy with the rebuttal. I think this paper has good enough contributions to get published.

I have revised my scores accordingly.

Summary:

This paper proposes a non-recurrent model for reading comprehension which used only convolutions and attention. The goal is to avoid recurrent which is sequential and hence a bottleneck during both training and inference. Authors also propose a paraphrasing based data augmentation method which helps in improving the performance. Proposed method performs better than existing models in SQuAD dataset while being much faster in training and inference.

My Comments:

The proposed model is convincing and the paper is well written.

1. Why don’t you report your model performance without data augmentation in Table 1? Is it because it does not achieve SOTA? The proposed data augmentation is a general one and it can be used to improve the performance of other models as well. So it does not make sense to compare your model + data augmentation against other models without data augmentation. I think it is ok to have some deterioration in the performance as you have a good speedup when compared to other models.

2. Can you mention your leaderboard test accuracy in the rebuttal?

3. The paper can be significantly strengthened by adding at least one more reading comprehension dataset. That will show the generality of the proposed architecture. Given the sufficient time for rebuttal, I am willing to increase my score if authors report results in an additional dataset in the revision.

4. Are you willing to release your code to reproduce the results?


Minor comments:

1. You mention 4X to 9X for inference speedup in abstract and then 4X to 10X speedup in Intro. Please be consistent.
2. In the first contribution bullet point, “that exclusive built upon” should be “that is exclusively built upon”.


This paper proposes two contributions: first, applying CNNs+self-attention modules instead of LSTMs, which could result in significant speedup and good RC performance; second, enhancing the RC model training with passage paraphrases generated by a neural paraphrasing model, which could improve the RC performance marginally.

Firstly, I suggest the authors rewrite the end of the introduction. The current version tends to mix everything together and makes the misleading claim. When I read the paper, I thought the speeding up mechanism could give both speed up and performance boost, and lead to the 82.2 F1. But it turns out that the above improvements are achieved with at least three different ideas: (1) the CNN+self-attention module; (2) the entire model architecture design; and (3) the data augmentation method. 

Secondly, none of the above three ideas are well evaluated in terms of both speedup and RC performance, and I will comment in details as follows:

(1) The CNN+self-attention was mainly borrowing the idea from (Vaswani et al., 2017a) from NMT to RC. The novelty is limited but it is a good idea to speed up the RC models. However, as the authors hoped to claim that this module could contribute to both speedup and RC performance, it will be necessary to show the RC performance of the same model architecture, but replacing the CNNs with LSTMs. Only if the proposed architecture still gives better results, the claims in the introduction can be considered correct.

(2) I feel that the model design is the main reason for the good overall RC performance. However, in the paper there is no motivation about why the architecture was designed like this. Moreover, the whole model architecture is only evaluated on the SQuAD dataset. As a result, it is not convincing that the system design has good generalization. If in (1) it is observed that using LSTMs in the model instead of CNNs could give on par or better results, it will be necessary to test the proposed model architecture on multiple datasets, as well as conducting more ablation tests about the model architecture itself.

(3) I like the idea of data augmentation with paraphrasing. Currently, the improvement is only marginal, but there seems many other things to play with. For example, training NMT models with larger parallel corpora; training NMT models with different language pairs with English as the pivot; and better strategies to select the generated passages for data augmentation.

I am looking forward to the test performance of this work on SQuAD.

This paper presents a reading comprehension model using convolutions and attention. This model does not use any recurrent operation but it is not per se simpler than a recurrent model. Furthermore, the authors proposed an interesting idea to augment additional training data by paraphrasing based on off-the-shelf neural machine translation.  On SQuAD dataset, their results show some small improvements using the proposed augmentation technique. Their best results, however, do not outperform the best results reported on the leader board.

Overall, this is an interesting study on SQuAD dataset. I would like to see results on more datasets and more discussion on the data augmentation technique. At the moment, the description in section 3 is fuzzy in my opinion. Interesting information could be:
- how is the performance of the NMT system? 
- how many new data points are finally added into the training data set?
- what do ‘data aug’ x 2 or x 3 exactly mean?


We have just added a new result on the adversarial SQuAD dataset [1] . In terms of robustness to the adversarial examples, our model is on par with the state-of-the-art model. Please see the Section 4.1.5 of the latest version for more details.

ps: This addition is also partly motivated by Reviewer 1's promise to increase our score (to above 7). Until now, we have added 2 more benchmarks: TriviaQA & Adversarial SQuAD.

[1] Jia Robin, Percy Liang. Adversarial Examples for Evaluating Reading Comprehension Systems. In EMNLP 2017.

Thanks!

As we have included the result on the triviaQA dataset as well, we hope the reviewer can reconsider the score, as promised in the original review. Thanks again for your suggestion to help us improve the paper!

Dear Area Chair,

We have submitted the rebuttal and revision. Our rebuttal contains a general one to address the common concerns of the reviewers, and three separated ones to answer the individual questions for each reviewer.

Thanks! 

We believe there are misunderstandings we have addressed below. We also have also included more experimental results. 

Q: The reviewer said “I suggest the authors rewrite the end of the introduction. The current version tends to mix everything together and makes the misleading claim.”
A: Thank you for the suggestions! We have revised the introduction to make our contributions clearer. Note that even though self-attention has already been used extensively in Vaswani et al, the combination of convolutions and self-attention is novel, and is significantly better than self-attention alone and gives 2.7 F1 gain in our experiments. The use of convolutions also allows us to take advantage of common regularization methods in ConvNets such as stochastic depth (layer dropout), which gives an additional gain of 0.2 F1 in our experiments.
We would like to point out that in this paper the use of CNN + self-attention is to speed-up the model during training and inference.  The speed-up leads to faster experimentation and allows us to train on more augmented data, contributing to the strong result on SQuAD.

Q: The reviewer comments “I feel that the model design is the main reason for the good overall RC performance. However, in the paper there is no motivation about why the architecture was designed like this.”
A: At a high level, our architecture is the standard “embedding -> embedding encoder -> attention -> modeling encoder -> output” architecture, shared by many neural reading comprehension models. Thus, we do NOT claim any novelty in the overall architecture. Traditionally, the encoder components are bidirectional LSTMs. Our motivation is to speed-up the architecture by replacing the bidirectional LSTMs with convolution+self-attention for the encoders for both embedding and modeling components. The context passages are over one hundred words long in SQuAD, so the parallel nature of CNN architectures leads to a significant speed boost for both training and inference. Replacing bidirectional LSTMs with convolution+self-attention is our main novelty.

Q: The reviewer comments “it will be necessary to show the RC performance of the same model architecture, but replacing the CNNs with LSTMs. Only if the proposed architecture still gives better results, the claims in the introduction can be considered correct.”
A: We think the reviewer might have misunderstood our claim. As mentioned above, we do NOT claim any novelty in the overall architecture, as it is a common reading comprehension model. We will make this point clearer in the revision. Our contribution, as we have emphasized several times, is to replace the LSTM encoders with convolution+self-attention, without changing the remaining components. We find the resulting model both fast and accurate. In fact, if we switch back to LSTM encoders, it will become BiDAF [1] or DCN [2], which are both slower (see our speedup experiments) and less accurate (see the leaderboard: https://rajpurkar.github.io/SQuAD-explorer/) than ours.

[1] Bidirectional Attention Flow for Machine Comprehension. In ICLR 2017.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, Hannaneh Hajishirzi.
[2] Dynamic Coattention Networks For Question Answering. ICLR 2017.
Caiming Xiong, Victor Zhong, Richard Socher.

Q: Results on one more dataset.
A: We have conducted experiments on another Q&A dataset, TriviaQA, to verify that the effectiveness and efficiency of our model is general. In a nutshell, again, our model is 4x to 16x times faster than the RNN counterparts, while outperforming the state-of-the-art single-paragraph-reading model by more than 3.0 in both F1 and EM. Please see the revision. 

Q: More result on data augmentation.
A: Thanks for the suggestions! We indeed put more experiments in the revision and here are some interesting findings: 
Translating to more languages can lead to more diverse augmented data, which further result in better generalization. Currently we try both French and German.
The sampling ratio of (original : English-French-English : English-German-English) during training matters. The best empirical ratio is 3:1:1.

Q: Leaderboard result.
A: We submitted our best model for test set evaluation on SQuAD, on Dec 20, 2017. Our single model (named “FRC”) is ranked 3rd among all single models in terms of F1 with F1/EM=84.6/76.2 (https://rajpurkar.github.io/SQuAD-explorer/). The performance gain is because we add more regularization to the model. Note that the two single models ranked above us have NOT been published yet: “BiDAF + Self Attention + ELMo” & “AttentionReader+”.

From the reviewer’s comments, it is not immediately clear to us the reviewer’s rationale for rejection. What we only know is the reviewer wants to know more about the data augmentation approach. It would be great if the reviewer can elaborate more on the rejection rationale. 

We believe our work is significant in the following aspects:
(a) Our work is novel: we introduced a new architecture for reading comprehension and a data augmentation technique that yields non-trivial gain on a strong SQuAD model. Note that even though self-attention has already been used extensively in Vaswani et al, the combination of convolutions and self-attention is novel, and is significantly better than self-attention alone and gives 2.7 F1 gain in our experiments. The use of convolutions also allows us to take advantage of common regularization methods in ConvNets such as stochastic depth (layer dropout), which gives an additional gain of 0.2 F1 in our experiments.

(b) Our model is accurate: we are currently ranked 3rd by F1 score on the SQuAD leaderboard among single models (note: the two single models ranked above us are not published yet).

(c) Our model is fast: we achieve a speed-up of up to 13x and 9x in training and inference respectively on SQuAD.
 
As stated above, we are disappointed with the low scores that our paper has received. Concurrent to our submission, there are two other papers on SQuAD, FusionNet[1] and DCN+ [2], which only tested on SQuAD and obtained much lower F1 scores (83.9 and 83.0 respectively) compared to ours (84.6). Their papers, however, received the averaged review scores of 6.33 and 7 respectively, which are much higher than our averaged review score of 5.33. As such, we encourage the reviewers to reconsider their scores.

[1] https://openreview.net/forum?id=BJIgi_eCZ&noteId=BJIgi_eCZ
[2] https://openreview.net/forum?id=H1meywxRW

More detailed comments:
Q: Regarding “simplicity”.
A: Thanks for raising this point. By simplicity, we mean we do not use hand-crafted features such as POS tagging ([3]), nor multiple reading pass ([4]). We have made this point clear in the revision and tried not using “simple” to avoid confusion.

[3] Reading Wikipedia to Answer Open-Domain Questions. In ACL 2017.
Danqi Chen, Adam Fisch, Jason Weston, Antoine Bordes. 
[4] Reasonet: Learning to stop reading in machine comprehension. In KDD 2017.
Yelong Shen, Po-Sen Huang, Jianfeng Gao, Weizhu Chen.

Q: Leaderboard result.
A: We submitted our best model for test set evaluation on SQuAD, on Dec 20, 2017. Our single model (named “FRC”) is ranked 3rd among all single models in terms of F1 with F1/EM=84.6/76.2 (https://rajpurkar.github.io/SQuAD-explorer/). The performance gain is because we add more regularization to the model. Note that the two single models ranked above us have NOT been published yet: “BiDAF + Self Attention + ELMo” & “AttentionReader+”. 

Q: Results on one more dataset.
A: We have conducted experiments on another Q&A dataset, TriviaQA, to verify that the effectiveness and efficiency of our model is general. In a nutshell, again, our model is 4x to 16x times faster than the RNN counterparts, while outperforming the state-of-the-art single-paragraph-reading model by more than 3.0 in both F1 and EM. Please see the revision. 

Q: Section 3 and more discussion on the data augmentation.
A: We have revised the paper to give more details regarding our method and results with data augmentation. Here, we highlight a few major details that the reviewers asked, as well as several new findings:
a) Performance of NMT systems:
English-German (newstest2015): 27.6 (to German) and 29.9 (to English)
English-French (newstest2014): 36.7 (to French) and 35.9 (to English)
b) Note that in our Table, “x2” means the total amount of the final training data is twice as large as the original data, i.e. the added amount is the same as the original. We have clarified this as well in the revision.
c) New finding: translating to more languages can lead to more diverse augmented data, which further result in better generalization. Currently we try both English-French and English-German.
d) New finding: we have shown in the revised experiment section that different ratios (original : English-French-English : English-German-English) would have different effects on the final performance. Empirically,  when the ratio is 3:1:1, we get the best result. We interpret this phenomenon as: the translation might bring noise to the augmented data that we should lay more weight to the original clean data.



We thank the reviewer for your acknowledgement to our contributions! We address the comments below.

Q: The reviewer asks “Why don’t you report your model performance without data augmentation in Table 1?” 
A: We thank the reviewer for the suggestion! We have added this result in the revision. In summary, without data augmentation, our model gets 82.7 F1 on dev set, while with data augmentation, we get 83.8 F1 on dev. We only submitted the model with augmented data, and get 84.6 F1 on test set, which outperforms most of the existing models and is the best among all the published results, as of Dec 20, 2017. 

Q: The reviewer asks “Can you mention your leaderboard test accuracy in the rebuttal?”
A: We submitted our best model for test set evaluation on SQuAD, on Dec 20, 2017. Our single model (named “FRC”) is ranked 3rd among all single models in terms of F1 with F1/EM=84.6/76.2 (https://rajpurkar.github.io/SQuAD-explorer/). The performance gain is because we add more regularization to the model. Note that the two single models ranked above us have NOT been published yet: “BiDAF + Self Attention + ELMo” & “AttentionReader+”. 

Q: Results on one more dataset.
A: We have conducted experiments on another Q&A dataset, TriviaQA, to verify that the effectiveness and efficiency of our model is general. In a nutshell, again, our model is 4x to 16x times faster than the RNN counterparts, while outperforming the state-of-the-art single-paragraph-reading model by more than 3.0 in both F1 and EM. Please see the revision. 

Q: The reviewer asks “Are you willing to release your code to reproduce the results?”
A: Yes, we will release the code after the paper gets accepted.

Q: Minor comments.
A: Thank you. We addressed all of them in the latest revision. 


We thank reviewers for comments and feedback to our paper, which have helped us improve the paper. However, we are disappointed with the low scores that our paper has received. Concurrent to our submission, there are two other papers on SQuAD, FusionNet[1] and DCN+ [2], which only tested on SQuAD and obtained much lower F1 scores (83.9 and 83.0 respectively) compared to ours (84.6). Their papers, however, received the averaged review scores of 6.33 and 7 respectively, which are much higher than our averaged review score of 5.33. As such, we encourage the reviewers to reconsider their scores.

[1] https://openreview.net/forum?id=BJIgi_eCZ&noteId=BJIgi_eCZ
[2] https://openreview.net/forum?id=H1meywxRW

We answer here some key questions by the reviewers:

1. Novelty
A major concern amongst the reviewers novelty of this paper because it’s similar to Vaswani et al. We stress here that our model is indeed novel: Note that even though self-attention has already been used extensively in Vaswani et al, the combination of convolutions and self-attention is novel, and is significantly better than self-attention alone and gives 2.7 F1 gain in our experiments. Our good accuracy is coupled with very good speedup gains. The speedup gains of up to 13x per training iteration and 9x during inference on SQuAD is not small. This significant gain makes our model most promising for larger datasets.

2. Test set result on SQuAD leaderboard
We submitted our best model for test set evaluation on SQuAD, on Dec 20, 2017. Our single model (named “FRC”) is ranked 3rd among all single models in terms of F1 with F1/EM=84.6/76.2 (https://rajpurkar.github.io/SQuAD-explorer/). The performance gain is because we add more regularization to the model. Note that the two single models ranked above us have NOT been published yet: “BiDAF + Self Attention + ELMo” & “AttentionReader+”. 

3. Results on an additional benchmark (TriviaQA)  
We have conducted experiments on another Q&A dataset, TriviaQA, to verify that the effectiveness and efficiency of our model is general. In a nutshell, again, our model is 4x to 16x times faster than the RNN counterparts, while outperforming the state-of-the-art single-paragraph-reading model by more than 3.0 in both F1 and EM. Please see the revision. 


Authors, 

Please post a rebuttal for this work. Discussion period ends Jan 5th. 

Thanks for your interest and the questions! Here are the answers:

1.  The number of heads is 8, which is consistent throughout the layers. The attention key depth is 128, so the per head depth is 128/8=16.

2. It should be "the kernel sizes are 7 and 5". 

We will clarify those in the revision. Thanks!

Thank you for your work. It seems the paper lacks some of the implementation details and sometimes includes ambiguous statements.
1. What is the number of heads used for the multi-head self attention, and is the number consistent throughout the layers? And is the attention key depth per head also 128? I feel that the encoder layer detail is lacking.
2. Subsection 2.2 in 2. Embedding Encoder Layer, the paper states that kernel size of 7 is used for embedding encoder. However, later on subsection 4.2 Basic Setup describes "the kernel sizes are 5 and 7" respectively. Could you please clarify this?

Meta-review: This work replaces the RNN layer of square with a self-attention and convolution, achieving a big speed up and performance gains, particularly with data augmentation. The work is mostly clear presented, one reviewer found it "well-written" although there was a complaint the work did not clear separate out the novel aspects. In terms of results the work is clearly of high quality, producing top numbers on the shared task. There were some initial complaints of only using the SQuAD dataset, but the authors have now included additional results that diversify the experiments. Perhaps the largest concern is novelty. The idea of non-RNN self-attention is now widely known, and there are several systems that are applying it. Reviewers felt that while this system does it well, it is maybe less novel or significant than other possible work. 

Decision: accepted-poster-papers