Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a semi-Markov conditional random field model that integrates a filtering step to eliminate irrelevant segments when performing named entity recognition in text. According to the authors this helps reducing the complexity compared to a semi-Markov CRF and dramatically reduces the search space. 

strength_and_weaknesses: Strengths

Attempt to learn segmentation jointly with classification (here named entity recognition or NER) that does not rely on an expansion of the label space (as is commonly done in NER by specializing the labels into begin label, intermediate and outside label).

Weaknesses

-	The model relies on an extra classifier that filters segments. The model is jointly trained by considering the filter classification loss, the segmentation loss and the NER classification loss. It is not clear from the paper how during training the computational complexity is reduced as initially all potential segments need to be considered in the inference step during training. That means that the complexity is equal to the complexity of the semi-Markov model.  The authors acknowledge that “during training, especially in the first stage, the graph size can be large since the filtering is poor.” It is not clear how exactly the filtering is performed during training. 
-	Because the segment filtering is the contribution of the paper, its approach (which should be explained in detail) should be separately evaluated, that is, its behavior and reduction in complexity during training and ablating the assumptions made here. 
-	Results could have been evaluated on many more NER datasets. 
-	The claim that the proposed filtering model drastically reduces the search space compared to a CRF and SemiCRF model is not seen in the model’s throughput numbers, especially not during training. Moreover, the effect of dynamic programming on the model’s throughput during inference on the test data could be investigated. Is dynamic programming also used in the decoding of the CRF and SemiCRF models?

clarity,_quality,_novelty_and_reproducibility: The clarity (and consequently the reproducibility) could be improved (see remark above). The originality is difficult to judge given that the details of the contribution are missing. 
summary_of_the_review: The paper is below the threshold for acceptance at ICLR because of:
- Important details and evaluations are missing.

Several of my questions were answered during the rebuttal, for which I thank the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on reducing the inference complexity of semi-Markov CRF and thus achieving better performance on NER by introducing a filtering step before forward algorithm and Viterbi decoding. More specifically, segments that are predicted to be null and whose label does not achieve the highest predicted score according to an additional local classifier will be filtered. Experiments are conducted on the task of NER and evaluated on three datasets: CONLL-2003, OntoNotes 5.0 and Arabic ACE.
strength_and_weaknesses: Strength:
1. Experiment results on performance and throughput verify the claim that Filtered Semi-Markov CRF are better than CRF and Semi-Markov CRF on NER tasks.

Weakness:
1. Some parts of the paper is a little hard to follow, especially Section 3. For example, what is the model architecture of local classifier compared to the global ones?  Do they share parameters? Besides, without reading the rest part in Section 3, it is hard to recognize that Eq. (6) chooses segments whose labels are not predicted to null but also achieves the highest scores. There also exist some typos. For example, in the 4th row from the bottom of Page 3, $j_k<i_{k^*}$, which should be $i_{k^*} < j_k$.

2. Lack of detailed analysis about why filtering leads to better performance. FSemiCRF does Viterbi decoding on a subset of segments compared to the original SemiCRF, whose optimum should be worse than that of SemiCRF in theory. I'm curious whether the introducing of local filtering models and corresponding segment filtering in training or segment filtering during inference is the primary cause of performance improvement. an ablation study and analysis will be helpful. For example, how FSemiCRF performs if it is trained as described in Section 3 but still follows original SemiCRF (i.e., run Viterbi decoding on all the segments) during inference?

3. Model throughput needs further explanation. During inference, both SemiCRF and FSemiCRF have two steps: (1) to calculate segment scores for all segments; (2) Viterbi decoding, in which SemiCRF does Viterbi decoding directly while FSemiCRF runs segment filtering and does Viterbi decoding on the rest segments. It will be better to show which step is the bottleneck (e.g., to show empirical wall-clock time of these two steps) and how FSemiCRF improves SemiCRF in the second step (e.g., to show wall-clock time of segment filtering and Viterbi decoding on the rest segments).

4. What about training complexity (e.g., wall-clock time) of FSemiCRF compared to SemiCRF.
clarity,_quality,_novelty_and_reproducibility: The idea of accelerating the inference of SemiCRF is novel and the proposed filtering method is well-motivated. However, current analysis and experiment results do not support the claim well and more detailed analysis is needed. 
summary_of_the_review: In general, this paper focuses on an important problem w.r.t. SemiCRF which prevents its widespread application as CRF and proposes an applicable method for resolving it. However, this method lacks thorough analysis (especially experiments), which weakens the claims made in the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: A method is proposed for filtering spans to be labeled in a semi-Markov CRF, which empirically reduces training/inference complexity and increases evaluation metrics relative to basic CRF or basic semi-CRF.
strength_and_weaknesses: The paper is clearly written. The method is, to my knowledge, original.

It does not compare to state-of-the-art methods, which generally use something like transformer decoders with greedy decoding. (https://paperswithcode.com/sota/named-entity-recognition-ner-on-conll-2003, https://paperswithcode.com/sota/named-entity-recognition-ner-on-ontonotes-v5, https://paperswithcode.com/sota/named-entity-recognition-on-ace-2005) The results are a bit worse than SOTA.

The motivation is to address the shortcoming of semiCRF that it considers all spans. It turns out that the proposed method is not only faster but gets better evaluation scores than pure semi-CRF. (The authors might want to speculate on why that is-- it isn't clear to me a priori that the proposed method should be more accurate.) However the proposed method has some shortcomings: it introduces a hyperparameter beta, it has the possibility of discarding good spans in the filtering phase, and the worst-case complexity is higher. For many tasks it is possible to give a reasonable upper bound on the span length that needs to be considered, so complexity of semiCRF training/inference is reduced to linear from quadratic in sentence length. What about a baseline that uses a maximum segment length and allows null labels only for spans of length 1? I think that should address the weaknesses of semiCRF in a cleaner way and I would not be surprised to see it working quite well in terms of evaluation.
clarity,_quality,_novelty_and_reproducibility: Clarity: mostly high
Quality: a bit weak
Novelty: probably new
Reproducibility: good
summary_of_the_review: The proposed method does well in ablation experiments (reducing to CRF, or expanding to full semi-CRF) but some obvious baselines were neglected (greedy decoding over Transformer, bounding maximum span of semiCRF).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new method, Filtered Semi-Markov CRF for NER tasks, which addresses the complexity and performance issues of Semi-CRF. Although the authors have addressed some of the reviewers' comments and improved the paper, the paper still suffers from a lack of clarity regarding the details of the method. The method can only operate on flat NER data and there are many transformer-based NER methods that can perform well on non-flat data. Furthermore, there are no comparisons with the SOTA.
justification_for_why_not_higher_score: Although the authors have addressed some of the reviewers' comments and improved the paper, the paper still suffers from a lack of clarity regarding the details of the method. The method can only operate on flat NER data, and many transformer-based NER methods can perform well on non-flat data. Furthermore, there are no comparisons with the SOTA.
Decision: Reject