Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a lightweight neural network that achieves state-of-the-art performance on NER. The network allows efficient active incremental training, which significantly reduces the amount of training data needed to match state-of-the-art performance.

The paper is well-written. The ideas are simple, but they seem to work well in the experiments. Interestingly, LSTM decoder seems to have slight advantage over CRF decoder although LSTM does not output the best tag sequence. It'd be good to comment on this.

* After rebuttal
Thank you for your response and revision of the paper. I think the empirical study could be useful to the community.
review_title: the ideas are simple but seems to work well empirically

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper applies active learning to a deep neural model (CNN-CNN-LSTM) for  named-entity recognition, which allows the model to match state-of-the-art performance with about 25% of the full training data.

Strength:
The paper is relatively easy to follow. Experiments show significant reduction of training samples needed.

Weaknesses:
About half of the content is used to explain the CNN-CNN-LSTM architecture, which seems orthogonal to the active learning angle, except for the efficiency gain from replacing the CRF with an LSTM.

The difference in performance among the sampling strategies (as shown in Figure 4) seems very tiny. Therefore, it is difficult to tell what we can really learn from these empirical results.

Other questions and comments:
In Table 4: Why is the performance of LSTM-LSTM-LSTM not reported for OntoNotes 5.0, or was the model simply too inefficient? 

How is the variance of the model performance? At the early stage of active learning, the model uses as few as 1% of the training samples, which might cause large variance in terms of dev/test accuracy. 

The SUBMOD method is not properly explained in Section 4. As one of the active learning techniques being compared in experiments, it might be better to formally describe the approach instead of putting it in the appendix.

review_title: Active learning for BiLSTM-based NER model

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the application of different existing active learning strategies for the deep models for NER.

Pros:
* Active learning may be used for improving the performance of deep models for NER in practice
* All the proposed approaches are sound and the experimental results showed that active learning is beneficial for the deep models for NER

Cons:
* The novelty of this paper is marginal. The proposed approaches turn out to be a combination of existing active learning strategies for selecting data to query with the existing deep model for NER. 
* No conclusion can be drawn by comparing with the 4 different strategies.

======= After rebuttal  ================

Thank you for the clarification and revision on this paper. It looks better now.

I understand that the purpose of this paper is to give actionable insights to the practice of deep learning. However, since AL itself is a meta learning framework and neural net as the base learner has been shown to be effective for AL, the novelty and contribution of a general discussion of applying AL for deep neural nets is marginal.  What I really expected is a tightly-coupled active learning strategy that is specially designed for the particular deep neural network structure used for NER. Apparently, however, none of the strategies used in this work is designed for this purpose (e.g., the query strategy or model update strategy should at least reflex some properties of deep learning or NER). Thus, it is still below my expectation. 

Anyway, since the authors had attempted to improved this paper, and the results may provide some information to practice, I would like to slightly raise my rating to give this attempt a chance.


review_title: This paper studies the application of different existing active learning strategies for the deep models for NER. This paper has several strong and weak points listed in the reviews

Meta-review: The reviewers liked this paper quite a bit. The novelty seems modest and the results are limited to a fairly simple NER task, but there is nothing wrong with the paper, hence recommending acceptance.
Decision: Accept (Poster)