Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed an attention-based document classifier based on BERT-LSTM structure. Experiments on patent and ArXiv datasets show the structure is slightly better than three baselines. This paper also analyzed the effection of gradient update settings in different models. 

This proposed model is lack of novelty, it applied the attention sum of LSTM output on top of BERT in the document classification task. Although this model concatenated more vectors (original LSTM output, argmax, attention weight) into the final prediction vector, the reason of integration those vectors is not explained and no experiment prove the effectiveness of this setting. The experiments are only evaluated on two less studied datasets which makes the comparison with other works less persuasive, more results on general document classification datasets are needed.

Questions: 
1. Why the RNN-BERT and the ATT-BERT only include h2~hm but exclude h1? 
2. As the segment is fixed in each model, why not include all the z1~zm but only include z1?
3. “Recurrent Neural Networks (RNNs) have been used for short text, e.g. sentiment analysis by Socher et al. (2011) “, the cited paper is a recursive neural network. It is not a recurrent neural network.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes to compare different methods to build BERT/GPT  representations of long documents, to bypass the limitation of the input size of these models. One of the proposed method uses attention mechanism to discover the most significant portion of the text which are use to backpropagate the error on the language model. Three combination methods (concatenation, RNN and attention)  are tested on 2 databases plus one modified version of one of the databases to show the impact of the presentation bias in the texts (most important part are at the beginning). 
Results show that the largest improvement is the base BERT model over the previously proposed model : this aspect should be comment : what is the reason of the improvement ? 
Combination of textual part also yields improvement, but to a smaller extend. Hyper-parameter and Training/Testing time are reported, which is useful from a practical point of view if one should decide to implement the proposed method or not, considering the extra computational load and the relatively small improvement. The Shuffling experiment demonstrate an interesting behaviour of the models, that should be confirmed on a real dataset.

 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper presents a methods to combine multiple pre-trained language models using an attention mechanism in order to take the whole document into account. The effectiveness of the methods is evaluated on two long document classification tasks (Arxiv publication and patent classification) with state-of the art results.

Pros:

- the effectiveness of the methods is experimentally demonstrated using two relevant datasets
- the inverted wireless experiment clearly shows the interest of the attention combination strategy 

Cons:

- the methods is very simple (combining multiple segment prediction to perform document classifications) making the contribution of this paper quite weak. In the own terms of the authors, the "main contribution [...] is to investigate the effectiveness of different combination strategies". I am not sure that it is sufficient for the ICLR standards.
- the paper is difficult to read and should be proofread to improve readability

Minor issues:

- Socher et al (2011) uses recursive neural networks (ReNN) and not recurrent neural networks (RNN). RNN are ReNN but restrained to a linear chain structure
- In Introduction: ... in the  domain extremely complex data that is language ... -> I'm not sure the sentence is correct
- In Introduction: the last sentence should be shorten and rephrased.
- many typos 
review_title: Official Blind Review #3

Meta-review: This paper investigates ways of using pretrained transformer models like BERT for classification tasks on documents that are longer than a standard transformer can feasibly encode. 

This seems like a reasonable research goal, and none of the reviewers raised any concerns that seriously questioned the claims of the paper. However, neither of the more confident reviewers were convinced by the experiments in the paper (even after some private discussion) that the methods presented here represent a useful contribution. 

This is not an area that I (the area chair) know well, but it seems as though there aren't any easy fixes to suggest: Additional discussion of the choice of evaluation data (or new data), further ablations, and general refinement of the writing could help.
Decision: Reject