Venue: ICLR 2018 conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This high-quality paper tackles the quadratic dependency of memory on sequence length in attention-based models, and presents strong empirical results across multiple evaluation tasks. The approach is basically to apply self-attention at two levels, such that each level only has a small, fixed number of items, thereby limiting the memory requirement while having negligible impact on speed. It captures local information into so-called blocks using self-attention, and then applies a second level of self-attention over the blocks themselves.

The paper is well organized and clearly written, modulo minor language mistakes that should be easy to fix with further proof-reading. The contextualization of the method relative to CNNs/RNNs/Transformers is good, and the beneficial trade-offs between memory, runtime and accuracy are thoroughly investigated, and they're compelling.

I am curious how the story would look if one tried to push beyond two levels...? For example, how effective might a further inter-sentence attention level be for obtaining representations for long documents? 

Minor points:
- Text between Eq 4 & 5: W^{(1)} appears twice; one instance should probably be W^{(2)}.
- Multiple locations, e.g. S4.1: for NLI, the word is *premise*, not *promise*.
- Missing word in first sentence of S4.1: ... reason __ the ...
review_title: Strong support for more efficient attention

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros: 
The paper proposes a “bi-directional block self-attention network (Bi-BloSAN)” for sequence encoding, which inherits the advantages of multi-head (Vaswani et al., 2017) and DiSAN (Shen et al., 2017) network but is claimed to be more memory-efficient. The paper is written clearly and is easy to follow. The source code is released for duplicability. The main originality is using block (or hierarchical) structures; i.e., the proposed models split the an entire sequence into blocks, apply an intra-block SAN to each block for modeling local context, and then apply an inter-block SAN to the output for all blocks to capture long-range dependency. The proposed model was tested on nine benchmarks  and achieve good efficiency-memory trade-off. 

Cons:
- Methodology of the paper is very incremental compared with previous models.  
- Many of the baselines listed in the paper are not competitive; e.g.,  for SNLI, state-of-the-art results are not included in the paper. 
- The paper argues advantages of the proposed models over CNN by assuming the latter only captures local dependency, which, however, is not supported by discussion on or comparison with hierarchical CNN.
- The block splitting (as detailed in appendix) is rather arbitrary in terms of that it potentially divides coherent language segments apart. This is unnatural, e.g., compared  with alternatives such as using linguistic segments as blocks.
- The main originality of paper is the block style. However, the paper doesn’t analyze how and why the block brings improvement. 
-If we remove intra-block self-attention (but only keep token-level self-attention), whether the performance will be significantly worse?

review_title: The methodology of the paper is incremental; the evaluation is comprehensive and in general supports the claims. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces bi-directional block self-attention model (Bi-BioSAN) as a general-purpose encoder for sequence modeling tasks in NLP. The experiments include tasks like natural language inference, reading comprehension (SquAD), semantic relatedness and sentence classifications. The new model shows decent performance when comparing with Bi-LSTM, CNN and other baselines while running at a reasonably fast speed.

The advantage of this model is that we can use little memory (as in RNNs) and enjoy the parallelizable computation as in (SANs), and achieve similar (or better) performance.

While I do appreciate the solid experiment section, I don't think the model itself is sufficient contribution for a publication at ICLR. First, there is not much innovation in the model architecture. The idea of the Bi-BioSAN model simply to split the sentence into blocks and compute self-attention for each of them, and then using the same mechanisms as a pooling operation followed by a fusion level. I think this more counts as careful engineering of the SAN model rather than a main innovation. Second, the model introduces much more parameters. In the experiments, it can easily use 2 times parameters than the commonly used encoders. What if we use the same amount of parameters for Bi-LSTM encoders? Will the gap between the new model and the commonly used ones be smaller?

====

I appreciate the answers the authors added and I change the score to 6.
review_title: solid experiments, but the model is not very exciting

Meta-review: The proposed Bi-BloSAN is a two-levels' block SAN, which has both parallelization efficiency and memory efficiency. The study is thoroughly conducted and well presented.  
Decision: Accept (Poster)