Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose BlockBERT, a model that makes the attention matrix of Transformer models sparse by introducing block structure. This has the dual benefit of reducing memory and reducing training time. The authors show on various question answering tasks that their model is competitive with RoBERTa. 

1. Can the authors add additional details about the training of their model in Section 4.1? It is not clear for example the vocabulary size - is that the RoBERTa vocabulary size of around 50k or the BERT vocabulary size that is smaller? I believe this will affect the memory and training speed. 

2. For the datasets such as TriviaQA and SearchQA, how is RoBERTa finetuned on these tasks? By doing the window approach? 

3. The authors compare to RoBERTa and Sparse BERT as baselines for the section on performance. However, can the authors also include metrics on training time and memory in Table 2 for Sparse BERT as well as other sparse attention transformer architectures proposed (for example the Correia paper or the Sukhbaatar paper)? It is not clear the savings from this architecture compared to sparse Transformers in general. 

4. The analysis in Section 4.3 is quite unclear due to how compressed the description is and how tiny the graphs are. 

5. The authors mention that attention heads can be sparsified due to the memory usage and quadratic time complexity. Other work has also shown that the attention heads are quite redundant and can be pruned away, so attention head dropout is effective. For example https://arxiv.org/abs/1905.09418 

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper introduces a optimisation for BERT models based on using block matrices for the attention layers. This allows to reduce the memory footprint and the processing  time during training while reaching state-of-the-art results on 5 datasets. An interesting study on memory consumption in BERT is conducted. No results are given at test time : is there also a memory and processing time reduction ?

Even if the proposition is interesting, the impact of the paper is limited to the (flourishing) scope optimising Bert models ("Bertology"). The authors do not mention if their code is available.


Table 3 : Humam -> Human	
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper propose to sparsify the attention matrix to decrease memory usage and to speed up training. The authors experiment the model on multiple tasks. The model gains ~20% efficiency with ~20% decrease in memory use while maintaining comparable performance to the state of the art model. To keep the performance comparable, the authors use the same training corpus. The authors also discuss how block size could change the performance of the model. The paper is clear and well organized with good experiment results.
review_title: Official Blind Review #2

Meta-review: This paper proposes blockwise masked attention mechanisms to sparsify Transformer architectures, the main motivation being reducing the memory usage with long sequence inputs. The resulting model is called BlockBERT. The paper falls in a trend of recent papers compressing/sparsifying/distilling Transformer architectures, a very relevant area of research given the daunting resources needed to train these models.

While the proposed contribution is very simple and interesting, it also looks a rather small increment over prior work, namely Sparse Transformer and Adaptive Span Transformer, among others. Experiments are rather limited and the memory/time reduction is not overwhelming (18.7-36.1% less memory, 12.0-25.1% less time), while final accuracy is sometimes sacrificed by a few points. No comparison to other adaptively sparse attention transformer architectures (Correia et al. EMNLP 19 or Sukhbaatar et al. ACL 19) which should as well provide memory reductions due to the sparsity of the gradients, which require less activations to be cached. I suggest addressing this concerns in an eventual resubmission of the paper.
Decision: Reject