Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes Block Skim Transformer (BST), a variation of Transformer-based architectures for QA where a newly proposed module predicts the relevance of each 'block' of context text and masks less relevant blocks out. The authors show 1.6x speedup during inference across 5 QA datasets, with a small increase in accuracy.

While I find the motivation and the high level idea of using Attention weights to decide block relevance interesting, there are too many gaps in both the idea and the writeup at this point.  Below I list a few of them:

* The way the block relevance part of the model is trained, it should really only work for factoid or single-hop QA tasks. This is because a block is marked relevant only if it contains answer words. On the other hand, in multi-hop or mulit-fact QA tasks (which is where the focus on the QA community currently is), this simple relevance criteria just doesn't work. E.g., in a question like, "Where was the 44th president of the USA born?", any information about WHO the 44th president is would not be marked as relevant, yet it's critical to answer the question.  The same applies to so-called "comparison" questions in HotpotQA.

* Many important details are missing, which makes it difficult to assess the setup and the value of the findings. E.g., HotpotQA's so-called distractor setting (which is the easier one) has 10 paragraphs as context. How are these even fed into your baseline and BST models, when you have a max token limit of 512? Similarly for TriviaQA, where there are many (six, I think) evidence documents from Wikipedia.

* The technique, by design, needs all context in the bottom 1/3 or 1/4 layers of the Transformer architecture.  This means it cannot handle any longer context than the baseline; everything must fit into the 512 token limit, which is arguably the biggest bottleneck of most of the current Transformer architectures. In other words, the paper doesn't address this problem, which is more important than obtaining a 1.6x speedup.

* Eqn (2) is confusing: what are a and b? Start and end of a block? If so, this computation is the sum of all self-attention within the block from a to b. The text mentions using the attention between the question and the block for assessing relevance. The precise equation for that cross-attention would be more valuable here.

* Eqn (6) is confusing: I was expecting a quadratic term, like N^2 in the numerator and k^2 N^2 in the right hand side of the denominator. What is "Layer" in this equation?  Also, please check your simplification, it doesn't seem correct (note, e.g., that as k increases, your speedup expression on the right also increases, but it should decrease). The correct term seems something like L / (l + (L-l) k^2)  =  L / (k^2 L + (1-k^2) l).

* The speedup of 1.6x, while valuable, doesn't seem like a make-or-break decision when deciding to use a Transformer model or not at inference time.  I was hoping for a larger gain when keeping only a fraction of the context.

* What about training time -- is there any speed up there?  Do I understand correctly the answer is "no" because, I think, you keep all blocks during training time in case the relevance decision is incorrect?

* Inference experiments are motivated by a "mobile" use case setting, where a CPU is used with batch size one.  Is it correct to assume there isn't any significant speedup on a GPU with a larger batch, e.g., when computing Dev/Test numbers for the 5 QA datasets considered?

The writing has many typos and small mistakes, and would benefit from through proof reading.

A convincing clarification of the above concerns could change my mind, but at this point I don't think the paper is ready for publication.
review_title: Interesting idea but unconvincing execution

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a block skimming approach for QA. 

The main idea is to use a light-weight module that uses the attention weights to predict which blocks of input are not useful for answering a question and use this information to avoid processing these blocks in further layers. Empirical evaluation shows that this approach can reduce computation and provide as much as 1.6X speedup with no drop in accuracy and even further speedups for more small drops in accuracy. Furthermore, using the block skimming component i.e. recognizing which blocks are useful works as a regularizer and can even increase QA performance by modest amounts. 

The main strengths of the work are: 

1. The main idea is interesting and well-motivated. We want good solutions for getting large models to run efficiently.
2. The evaluation on multiple datasets and across three types of transformer based models show that skimming can provide modest but consistent gains and can provide speedups.

The main weaknesses are: 

1. Efficiency is a key point of this method. Yet, there are no external efficiency related baselines compared (e.g. DistillBERT). I buy the argument that this method can be seen as orthogonal to the other methods but it will be useful at the very least to show how the speed-ups fare when compared to other methods. The only comparison we have is with Albert but we don’t have any speed-up comparisons with it either.  
2. The gains with using skimming as a regularization method is limited. 


Other Questions and Suggestions:


1. Given the limited improvements in effectiveness, it will be worth reporting averages of a few runs (three-five) for at least  some of the rows in Table 1. 

2. It will be useful to know the speed-ups for some of the block sizes. Is it possible to add these numbers in Table 2? 

3. DeFormer [1], a paper that appeared in ACL 2020, is a related work that simply removes attention computation across question and passage blocks in lower layers. This idea here is more general than the DeFormer work but is close enough to warrant a discussion in the paper at the very least.

4. Can you provide some intuitions for the specific design choices for the Block Skimming Module? 

5. It will be interesting to show if you are able to run BERT-large in the skim mode faster than  BERT-base in the original model. This will highlight the fact that skimming method can be used to improve

6. What kinds of blocks get ignored by the model? A post-hoc manual analysis would have been nice.

review_title: Nice idea for improving QA efficiency by skimming some blocks.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces Block Skim Transformer (BST), a variant of Transformer that skips tokens for self-attention so that it can pay less attention to less import tokens and achieve speedup. Although previous work studied the skimming mechanism for RNNs, this is the first paper that uses such an idea for transformers.

The model is pretty straight-forward - a block skim module in each transformer layer has a token-wise classifier that identifies which token to skip. One thing to note is that this model is specific to question answering (as the title indicates), because the supervision for tokens to skip and not to skip are pre-determined by identifying whether the token is part of the answer or not.

Strengths of the paper:
1) The problem is well-motivated with extensive discussion of related work.
2) The model is straight-forward and the description is easy to follow.
3) The paper includes extensive experiments using six different types of base pretrained models and five QA datasets, and comprehensive ablations.
4) The model achieves impressive speedup, e.g., without more than 0.5% performance degradation, the model achieves x1.4 speedup and x1.8 speedup with BERT base and BERT large, respectively.

Weakness of the paper:
The most important concern I have is that the model requires pre-defined supervision for tokens in order to train BST module. For instance, this paper trains the model to skim all the tokens that are not the answer token (which is the reason that it is specific to question answering). This means that BST module is trained for an end task (finding the answer tokens), rather than being trained to identify less important tokens for the end task. This is also a main difference from previous work that incorporates skimming modules for RNN, as most of them treat blocks to skip as latent variables and hence are easy to generalize to any tasks. This makes the model significantly less novel, and this fact was not apparent in Section 1 and 2, before reading the actual description of the model.
review_title: Well-written paper with comprehensive experiments & impressive speedup, however, the model requires pre-defined supervision for tokens for skimming

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Summary: This paper presents the "Block Skim Transformer" for extractive question answering tasks. The key idea in this model is using a classifier, on the self-attention distributions of a particular layer, to classify whether a large spans of non-contiguous text (blocks) contain the answer. If a block is rejected by the classifier, it is excluded in subsequent layers of self-attention. During training, no blocks are thrown away and the classifier is applied to every layer to provide a regularization effect, which leads to small improvements in performance in 5 datasets. During inference, blocks are thrown away at a fixed layer. The reduction in sequence length leads to ~1.5x batch size 1 speed improvements.

-----------------------------------

Strengths of the Paper

1. Lots of experiments, using 5 different datasets and many different pretrained LMs like BERT, RoBERTa and ALBERT. Small but consistent improvements in most settings. A detailed ablation study testing hyperparameters and different components of the model.

3. Cool idea of dropping tokens entirely, this is relatively unexplored in the space of efficient transformer models.

-----------------------------------

Weaknesses of the Paper

1. Training-Inference mismatch: It seems to me the proposed model has a strong mismatch during training and inference. During training, none of the blocks are dropped. During inference, a number of tokens are dropped at a particular layer (which is a inference time hyperparameter). It seems counter-intuitive to me that the subsequent layers of the model are able to successfully process a partial input, especially when this was not done during training. Also, this line is confusing "we forward the skipped blocks directly to the last layer for the QA classifier." Isn't this another training-inference mismatch? During training the QA classifier got the final layer representations for every block, not the intermediate layer representation.

2. Improvements are quite small: in terms of QA performance, the improvement is just 0.3 F1 on an average across datasets. Is this improvement statistically significant? Even in terms of speed, a 1.6x batch size 1 improvement seems relatively modest. Are the speed / accuracy improvements better on tasks requiring long input sequences? Intuitively, the Skim idea seems more well-suited to tasks having inputs much longer than 384 tokens, perhaps you could use some of the benchmarks in the efficient transformer papers?

3. How does this method compare to the numerous efficient transformers that have been proposed (in terms of attention, or adaptive layers)? While I don't expect comparisons against all of them, atleast 1-2 comparisons should be done to ground the observations better. Modifying attention distributions to be sparse over tokens seems like a more general way of getting the same effect as the block skim idea, and this also avoids the training-inference mismatch. 

-----------------------------------

Overall Recommendation

The idea of dropping tokens is interesting, but I'm not very convinced by the results. Performance improvements are modest and I'm really confused about the train-inference mismatch in the proposed architecture. More comparisons with a few alternative efficient transformer papers are needed to make this a stronger submission. Overall I'm leaning reject, but open to increasing my score slightly if I get more clarification regarding weakness #1.

-----------------------------------

Minor: I don't really understand Eq 2. What's the final vector that's used for logistic regression? How exactly is the aggregation done between the question and a block of text?

----------------------------------

**After Author Response**

Thank you for the detailed clarifications. I've raised my score to 5 since I found the random block skipping experiment quite interesting and surprising. However, weakness #2 and #3 are fairly important in my opinion and I don't think the response sufficiently addresses those weaknesses. I encourage the authors to show their method works on longer sequence datasets (where skimming intuitively makes sense) and compare against 1-2 efficient transformers using sparse attention, for better grounding the improvements in existing literature.
review_title: Cool idea but the model is confusing and results are not convincing

Meta-review: The paper presents a model for question answering where blocks of text can be skipped and only relevant blocks are further processed for extracting the answer span.  
 The reviewers mostly praised the general idea.  
 R3 raised concerns on generalizability of the presented approach.  
R4 raised several issues regarding presentation and clarity.  
R2 and R4 have concerns regarding execution and find some of the results unconvincing.  
While I don't necessarily share R2s concern on small improvements (improvements are still statistically significance), and despite the approach being very interesting, there are several issues that reviewers pointed out and wasn't resolved after discussions. 

Decision: Reject