Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:

This paper proposes a non-recurrent model for reading comprehension which used only convolutions and attention. The goal is to avoid recurrent which is sequential and hence a bottleneck during both training and inference. Authors also propose a paraphrasing based data augmentation method which helps in improving the performance. Proposed method performs better than existing models in SQuAD dataset while being much faster in training and inference.

My Comments:

The proposed model is convincing and the paper is well written.

1. Why don’t you report your model performance without data augmentation in Table 1? Is it because it does not achieve SOTA? The proposed data augmentation is a general one and it can be used to improve the performance of other models as well. So it does not make sense to compare your model + data augmentation against other models without data augmentation. I think it is ok to have some deterioration in the performance as you have a good speedup when compared to other models.

2. Can you mention your leaderboard test accuracy in the rebuttal?

3. The paper can be significantly strengthened by adding at least one more reading comprehension dataset. That will show the generality of the proposed architecture. Given the sufficient time for rebuttal, I am willing to increase my score if authors report results in an additional dataset in the revision.

4. Are you willing to release your code to reproduce the results?


Minor comments:

1. You mention 4X to 9X for inference speedup in abstract and then 4X to 10X speedup in Intro. Please be consistent.
2. In the first contribution bullet point, “that exclusive built upon” should be “that is exclusively built upon”.

review_title: Good paper. Results in an additional dataset needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a reading comprehension model using convolutions and attention. This model does not use any recurrent operation but it is not per se simpler than a recurrent model. Furthermore, the authors proposed an interesting idea to augment additional training data by paraphrasing based on off-the-shelf neural machine translation.  On SQuAD dataset, their results show some small improvements using the proposed augmentation technique. Their best results, however, do not outperform the best results reported on the leader board.

Overall, this is an interesting study on SQuAD dataset. I would like to see results on more datasets and more discussion on the data augmentation technique. At the moment, the description in section 3 is fuzzy in my opinion. Interesting information could be:
- how is the performance of the NMT system? 
- how many new data points are finally added into the training data set?
- what do ‘data aug’ x 2 or x 3 exactly mean?

review_title: Interesting augmentation method

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes two contributions: first, applying CNNs+self-attention modules instead of LSTMs, which could result in significant speedup and good RC performance; second, enhancing the RC model training with passage paraphrases generated by a neural paraphrasing model, which could improve the RC performance marginally.

Firstly, I suggest the authors rewrite the end of the introduction. The current version tends to mix everything together and makes the misleading claim. When I read the paper, I thought the speeding up mechanism could give both speed up and performance boost, and lead to the 82.2 F1. But it turns out that the above improvements are achieved with at least three different ideas: (1) the CNN+self-attention module; (2) the entire model architecture design; and (3) the data augmentation method. 

Secondly, none of the above three ideas are well evaluated in terms of both speedup and RC performance, and I will comment in details as follows:

(1) The CNN+self-attention was mainly borrowing the idea from (Vaswani et al., 2017a) from NMT to RC. The novelty is limited but it is a good idea to speed up the RC models. However, as the authors hoped to claim that this module could contribute to both speedup and RC performance, it will be necessary to show the RC performance of the same model architecture, but replacing the CNNs with LSTMs. Only if the proposed architecture still gives better results, the claims in the introduction can be considered correct.

(2) I feel that the model design is the main reason for the good overall RC performance. However, in the paper there is no motivation about why the architecture was designed like this. Moreover, the whole model architecture is only evaluated on the SQuAD dataset. As a result, it is not convincing that the system design has good generalization. If in (1) it is observed that using LSTMs in the model instead of CNNs could give on par or better results, it will be necessary to test the proposed model architecture on multiple datasets, as well as conducting more ablation tests about the model architecture itself.

(3) I like the idea of data augmentation with paraphrasing. Currently, the improvement is only marginal, but there seems many other things to play with. For example, training NMT models with larger parallel corpora; training NMT models with different language pairs with English as the pivot; and better strategies to select the generated passages for data augmentation.

I am looking forward to the test performance of this work on SQuAD.
review_title: Good ideas, but not so good evaluation.

Meta-review: This work replaces the RNN layer of square with a self-attention and convolution, achieving a big speed up and performance gains, particularly with data augmentation. The work is mostly clear presented, one reviewer found it "well-written" although there was a complaint the work did not clear separate out the novel aspects. In terms of results the work is clearly of high quality, producing top numbers on the shared task. There were some initial complaints of only using the SQuAD dataset, but the authors have now included additional results that diversify the experiments. Perhaps the largest concern is novelty. The idea of non-RNN self-attention is now widely known, and there are several systems that are applying it. Reviewers felt that while this system does it well, it is maybe less novel or significant than other possible work. 
Decision: Accept (Poster)