Training sequence labeling models using prior knowledge
Authors: Dani El-Ayyass
Venue: ICLR 2022 Submitted (2022)

Abstract: Sequence labeling task (part-of-speech tagging, named entity recognition) is one of the most common in NLP. At different times, the following architectures were used to solve it: CRF, BiLSTM, BERT (in chronological order). The combined model BiLSTM / BERT + CRF, where the last one is the topmost layer, however, performs better than just BiLSTM / BERT.

It is common when there is a small amount of labeled data available for the task. Hence it is difficult to train a model with good generalizing capability, so one has to resort to semi-supervised learning approaches. One of them is called pseudo-labeling, the gist of what is increasing the training samples with unlabeled data, but it cannot be used alongside with the CRF layer, as this layer simulates the probability distribution of the entire sequence, not of individual tokens.

In this paper, we propose an alternative to the CRF layer — the Prior Knowledge Layer (PKL), that allows one to obtain probability distributions of each token and also takes into account prior knowledge concerned the structure of label sequences.

Under review as a conference paper at ICLR 2022
TRAINING SEQUENCE LABELING MODELS PRIOR KNOWLEDGE
USING
Anonymous authors Paper under double-blind review
ABSTRACT
Sequence labeling task (part-of-specch tagging, named entity recognition) is one of the most common in NLP. At different times, the following architectures were used to solve it: CRF, BiLSTM, BERT (in chronological order). The combined model BiLSTM BERT + CRE, where the last one is the topmost layer, however, performs better than just BiLSTM/ BERT. It is common when there is a small amount of labeled data available for the task. Hence it is difficult to train a model with good generalizing capability, so one has resort approaches. One of them is called pseudo-labeling, the gist of what is increasing the training samples with unlabeled data, but cannot be used alongside with the CRF layer, as this layer simulates the probability distribution of the entire sequence, not of individual tokens. this paper, we propose an alternative to the CRF layer the Prior Knowledge Layer, that allows one to obtain probability distributions of each token and also takes into account prior knowledge concerned the structure of label sequences.
supervssde learning
INTRODUCTION
Sequence labeling, along with the text classification, one of the most common in natural language processing (NLP). In general, the task is to match each token word or sub-word in sentence with a corresponding label. Particular cases of this problem are part-of-speech tagging (POS tagging) and named entity recognition (NER).
Before the rise in popularity of deep learning methods, the sequence labeling problem was solved using graph probabilistic models, among which the Conditional Random Fields (CRF) (Lafferty et al. 2001) architecture showed the best quality. Now, to tackle this problem, both Recurrent Neural Networks (RNN) (Rumelhart al. 1986) (Karpathy) 2015), for example Bidirectional Long Short-Term Memory (BiLSTM) (Hochreiter & Schmidhuber 1997) (Schuster & Paliwal 1997). and neural networks based on the Transformer (Vaswani al. 22017 77 architecture, such Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. 2018) were used.
The combined model BiLSTM/ / BERT CRF (Huang al. 2015) (Lample et al. 2016), where the last one is the topmost layer, performs better than just BiL.TM// BERT, demonstrating quality close state-of-the-art This fact could be explained that CRF layer models a joint probability distribution over the entire sequence, which allows one to take into account the structure of the label sequences, for example, the presence of forbidden subsequences of labels.
It is common when there is a small amount of labeled data, and therefore it becomes challenging to train a neural network with good generalizing capability and one has to resort to semi-supervised learning approaches. One of them is called pseudo-labeling (Lee) 2013). the gist of what is increas- ing the training samples with unlabeled data. To implement this approach, model returns probability distribution cannot be done, since this layer models the probability distribution over the entire sequence.
required that the each token in the sequence, but when using CRF, this
To achieve the desired result, BiLSTM/ BERT model without the CRF layer could be used, but the structure of the label sequences will not be taken into account, which leads to worse quality of the model.
In this paper, we propose an alternative to the CRF layer - the Prior Knowledge Layer (PKL), that allows one obtain probability distributions of each token through baseline model such as

---

Under review as a conference paper at ICLR 2022
BiLSTM/ / BERT, and also takes into account prior knowledge of the structure of the label sequences, which makes it possible to use this architecture together with pseudo-labeling approach.
2
BACKGROUND
2.1 SEQUENCE LABELING MODELS
The problem of sequence labeling is widely known and there are many ways to approach it. One of the first models used to solve this is the Hidden Markov Model (HMM), which has its origins in Leonard Baum's 's 1966 work (Baum & Petrie 1966) (Rabiner & Juang 1986). This model has found its application not only in sequence labeling, but also speech recognition and analysis of biological sequences, in particular DNA. This model was replaced with the Maximum Entuupy Markov Model (MEMM, 2000) (McCallum et al. 2000), which combines the features of the HMM and Maximum Entropy models. A breakthrough was marked with the appearance of the Conditional Random Fields (CRF, 2001) model, which at that time demonstrated state-of-the-ar quality. There also modifications and generalizations of CRF, such as CRF with partial training (Mann & McCallum 2007). Hidden-state CRF (HCRF) (Quattoni al.] 2007), Dynamic CRF (DCRF) (Sutton et al. and Continuous CRF (CCRF) (Qin et al. 2008).
2007
The disadvantage of the aforementioned model is the need for manual feature generation and domain knowledge to train a model with good generalizing capability. This problem was solved by deep neural networks, which do not require manual generation of features, and also show better results, provided that there is a sufficient amount of labeled data.
Based on the sequence labeling problem structure. which implies a sequence at the input and a sequence of the same length at the output, the architecture of the Recurrent Neural Network is well-suited for its solution. RNNs are known to have a vanishing gradient problem, so the other architectures such as Long Short-TTerm Memory (LSTM, 1997) (Hochreiter & Schmidhuber 1997) and Gated Recurrent Unit (GRU, 2004) (Cho et al., 2014) (Chung et al. 2014) have been invented to solve this problem.
Sequence labeling task allows reading the entire input sequence to make a decision, that makes possible to use bidirectional architectures, particularly Bidirectional RNN (BiRNN) (Schuster & Paliwal 1997). to process the input sequence two directions from left to right and from right to left for better performance. But there is important point, that such architectures not "true" bidirectional, but rather two unidirectional ones.
Thus, we come to one of the most commonly used models for solving the problem of sequence labeling, namely Bidirectional Long Short-Term Memory (BiLSTM) (Hochreiter & Schmidhuber, 1997 (Schuster& & Paliwal 1997).
In 2017, as the article "Attention Is All You Need" (Vaswani et al. 2017) was published, in which the authors presented completely new architecture called Transformer. This architecture surpassed all previous saate-of-thh-att models in the task of machine translation. The authors of the article continued the ideas of the seq2seq (Cho al. 2014) (Sutskever et al. 2014) models and the attention mechanism (Bahdanau et al. 2014) introducing two new operations: Scaled Dot-Product Attention and Multi-Head Attention. These operations allow to consider information about all tokens in the sentence when processing specific token, which makes them "truly" bidirectional.
Transformer is an incoder-decoded model, and therefore cannot be used for sequence labeling. But it was just matter of time before the bidirectional attention mechanism and Transformer architec- ture was adapted other NL.P tasks, and 2018 the Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. 2011 88) model appeared. Architecturally, BERT is a Transformer encoder, and this model can be used to solve a wide range of NLP tasks, including sequence labeling.
The next architectural improvement was to combine the baseline model like BiLSTM/ BERT with CRF (Huang et al. 2015) (Lample et al., 2016), where the last acts as the topmost layer and models the joint probability distribution over the entire sequence. In this approach, the need for manual generation of features becomes irrelevant, since the logits of the baseline model, which are used as features, enter the CRE BERT CRF model shows quality close to state-of-the-art.
2

---

Under review as a conference paper at ICLR 2022
2.2 PSEUDO-LABELING
Pseudo-labeling is a simple and effective semi-supervised method, that was originally invented for deep neural networks. The of it that the model is jointly trained using both labeled and unlabeled data. First, model trained on labeled data, and then used for marking unlabeled data taking label with the maximum predicted probability given by the model for each data sample. Then these labels used as true labels for fitting the model on all data (Lec 2013).
2.3 NAMED ENTITY RECOGNITION TAGGING
Named Entity Recognition task consists in determining and extracting subsequences of words, called spans, related to one named entity. In order to classify entire spans as named entities, special markups (Carpenter 2009) are used. The most popular markups are BIO (Ramshaw Mareus 1995) and BILUO (BIOES, BMEWO) (Borthwick) 1999).
Name BIO stands for Beginning- Inside- -utiidee Within this markup, there are tags such named entity names and O (Outside), followed by one of the two prefixes B- (Beginning), I- (Inside). The B- prefix corresponds to the beginning of the named entity, the I- prefix corresponds to the continuation of the named entity, in case if the named entity consists of more than one token.
BILUO markup extends BIO with two additional prefixes L- (L.ast), U- (Unit), which denote the end of named entity, in case the named entity consists of more than one token, and single named entity, consisting of one token, respectively.
Note that after applying one of these markups, the number of labels becomes greater than the number of named entities. Let the number of named entities be N, then:
the number of labels the BIO markup is 2N +
the number of labels in the BILUO markup is 4N +
Despite the increase in the number of labels, not every label can follow arbitrary other label. Let's denote the named entity by X. BIO markup:
the label I-X can only follow the label B-X
BILUO markup:
the labels I-X and L-X can only follow the label B-X
label O can only follow label O, L-X or U-X
In other words, there are pairs of consecutive labels that are not correct and contradict the rules described above.
is important to note that the structure of label sequences can be in any sequence labeling task, but this structure is most indicative in NER, namely BIO / BILUO markups.
3 METHOD
3.1 PROBLEM STATEMENT
Let [N] be the set of labeled sequences (z, y), where:
(I1, -.., mi) sequence of tokens from X
y (y,,..., y) a sequence of labels from
In the sequence (x. v) E D. the token corresponds to the label y:-
Consider an arbitrary probabilistic model with parameters 9: (y((x)
The Maximum Likelihood principle for solving the sequence labeling problem looks as follows:
Σ
(n(o))) - max
EDD

---

Under review as a conference paper at ICLR 2022
Optimal sequence of labels with known parameters A:
arg max (0()22)
3.2 PRIOR KNOWLEDGE MATRIX
For convenient representation of correct incorrect pairs of consecutive labels, it is convenient to use matrix, which we call the Prior Knowledge Matrix (PKM). The rows of the matrix will correspond to the labels at the position, and the columns will correspond the labels at the position. Let be incorrect pair of labels, and O a correct one.
This approach generalized to correct incorrect sequences of labels of length more than two by transition from matrices to multidimensional arrays (tensors) of dimension, corresponding to the length of the largest forbidden sequence. The number of incorrect label sequences increases exponentially with the length of the label sequence, while the number of correct sequences increases much more slowly. This leads to the fact that the resulting multidimensional array is very sparse.
3.3 PRIOR KNOWLEDGE MATRIX FOR NER
Consider the NER task with two named entities:
person (PER)
location (LOC)
Let's build a PKM for both BIO and BILUO markups.
PKM row/column names for:
BIO: O, B-PER, B-LOC, I-PER, I-LOC
BILUO: O, B-PER, B-LOC, I-PER, I-LOC, L-PER, L-LOC, U-PER, U-LOC
0
1
1
1
0
1
0
1
1 0
Figure 1: Prior Knowledge Matrix for BIO tagging
0
0
0
1
0
0
1
0
0
0
1
1
1
0
0
1
0
1
1
1
1
1
1
0
O
0
0
1
0
0
0
0
1
0
0
0
1
1
0
Figure 2: Prior Knowledge Matrix for BILUO tagging
3.4 PRIOR KNOWLEDGE LAYER
Let's 's take a look at how the Prior Knowledge Layer (PKL) works. Since this layer takes into account knowledge about incorrect label subsequences, this layer initializes with Prior Knowledge Matrix.

---

Under review as a conference paper at ICLR 2022
PKL has trainable parameters. It receives imput the logits of each token obtained by the baseline BiLSTM/ BERT model and returns a scalar loss a - PKL loss that acts as an weighted additive term in the minimized loss function. Let's convert the Maximum Likelihood principle to minimization optimization problem multiplying by 11 and add scalar additive term:
Σ
po(lar) + λoe min
(E.))ED
acts a regularization parameter and could be tuned hyperparameter.
PKL loss is equal to the sum of all scalar penalties, calculated on all consecutive pairs (in the general case, sequences of arbitrary length) tokens, taking into account an information of incorrect labels subsequences Prior Knowledge Matrix. To calculate one scalar penalties, we need to define an operation on two probability distributions (over two consecutive tokens) and one matrix (PKM), which will penalize model for high probability values corresponding to incorrect pairs of labels.
Let's denote the probability distribution matrices, corresponding each token in consecutive pair as P1 and P2, correspondingly and Prior Knowledge Matrix as K.
One way to specify such an operation is the average over diagonal elements of the product of the following three matrices P1, P2 and K:
mean(diog( P P2K)2))
Since the loss function additive term, provided by Prior Knowledge Layer, takes into account infor- mation about incorrect label subsequences, this leads to:
better generalization capability, especially with small amount of training data
faster convergence, comparing with baseline model
Prior Knowledge Layer uses only during training phase. During inference the model behaves the same as the baseline BiLSTM/ BERT model without CRF layer.
NER Labels
NER Labels
CRF Layer
Prior Prowr-Knowledge Layer
BERT CRF
BILSTM/ / BERT
BiLSTM BERT
1
1
1
Word Fmbeddings
Word Fmbeddingg
Hello
Hello
name
(a) CRF Layer
(b) PKI. Layer
Figure 3: Baseline models with different topmost layers
REFERENCES
Dzmitry Bahdanau et al. Neural machine translation by jointly learning to align and translate. 2014.
L.E. Baum and T. Petrie. Statistical inference for probabilistic functions of finite state markov chains. 1966.
Andrew Borthwick. A maximum entropy approach named entity recognition. 1999.
Bob Carpenter. Coding chunkers as taggers: Io, bio, bmewo, and bmewo+. 2009.

---

Under review as a conference paper at ICLR 2022
Kyunghyun Cho al. Learning phrase representations using encoder-decoder for statistical machine translation. 2014.
Junyoung Chung et al. Empirical evaluation of gated recurrent neural networks on sequence model- ing. 2014.
Jacob Devlin al. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018.
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
Zhiheng Huang al. Bidirectional Istm-erf models for sequence tagging. 2015.
Andrej Karpathy. The unreasonable effectiveness of recurrent neural networks. 2015.
John Lafferty, Andrew McCallum, and Fernando C.N. Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. 2001.
Guillaume Lample et al. Neural architectures for named entity recognition. 2016.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. 2013.
Gideon Mann and Andrew McCallum. Efficient computation of entropy gradient for semi- supervised conditional random fields. 2007.
Andrew McCallum, Dayne Freitag, and Fernando Pereira. Maximum entropy markov models for information extraction and segmentation. 2000.
Tao Qin et al. Global ranking using continuous conditional random fields. 2008
Ariadna Quattoni et al. Hidden conditional random fields. 2007.
L.R. Rabiner and B. H. Juang. introduction to hidden markov models. 1986.
Lance A. Ramshaw and Mitchell P. Marcus. Text chunking using transformation-base learning. 1995.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propag errors. Nature, 1986.
M. Schuster and K.K. Paliwal. Bidirectional recurrent neural networks. 1997.
Ilya Sutskever et al. Sequence to sequence learning with neural networks. 2014.
Charles Sutton, Andrew McCallum, and Khashayar Rohanimanesh. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. 2007.
Ashish Vaswani et al. Attention is all you need. 2017.


Reviews:
[{"review_id": "gJZG2O1eACp", "reviewer": "Reviewer_n5Vw", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- this paper attempts to address an important problem in sequence labelling of data sparsity, as well as a weakness of CRF classifiers.\n\nWeaknesses: \n- The technical and experimental rigour of this paper is critically lacking. \n- The paper has no experimental section - no benchmarks vs current state of the art, no ablations for the PKM layer, and no validation of any of the claims made about the effectiveness of the PKM layer.\n- For example, these claims are left completely unvalidated: \n-- The number of incorrect label sequences increases exponentially with the length of the label sequence, while the number of correct sequences increases much more slowly. This leads to the fact that the resulting multidimensional array is very sparse.\n-- better generalization capability, especially with a small amount of training data \n-- faster convergence, comparing with baseline model\n- the presentation and communication is also unclear in various areas, including when describing the current literature (e.g. unclear what tasks are being considered for sequence labelling)", "summary_of_the_paper": "The paper proposes a new type of classification layer, called the prior knowledge layer (PKL) that estimates the probability distribution of tokens and can incorporate prior knowledge about the prior knowledge of the structure of label sequences. The authors do not validate any claims made in the paper with experimental results, benchmarks, or datasets.", "summary_of_the_review": "Overall, the paper critically lacks any experimental rigour so it is impossible to evaluate any of the technical claims made.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Tpp9MBlId-n", "reviewer": "Reviewer_E7aG", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is not a paper, instead technically more like a toy report. The architecture of the entire manuscript is way incomplete, without enough technique detail, not to mention the lost experiment part. Also, the idea of prior knowledge integration is naive.", "summary_of_the_paper": "The authors try to propose a `Prior Knowledge Layer' as an alternative to the CRF layer for the standard sequence labeling framework in NLP.", "summary_of_the_review": "Introduction: without a motivation; Related work: not informative; Method: no detail & incomplete; Experiment: missing. Reference: bad formation. So I would recommend a rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GkbFXZ74_9c", "reviewer": "Reviewer_7Kg2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm afraid that the idea of this paper is too simple. In fact, utilizing prior knowledge of BIO tagging scheme for safe decoding is very prevalent in practice. For example, I-ORG must follow B-ORG rather than O. Besides, no experiments are found in the paper. The whole paper just consists of 6 pages. It seems that the authors have submitted an incomplete version.", "summary_of_the_paper": "This paper proposes a prior knowledge matrix to displace CRF in sequence labeling. The author also claims that this approach is useful for low-resource settings, especially considering using pseudo-labeling.", "summary_of_the_review": "1, too simple idea, even not a novel one\n2, no experiments", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3mUNvbjRU_3", "reviewer": "Reviewer_UZd9", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The authors addressed state transition problems in sequence labeling problems, and proposed an approach that seems sound. This could be useful in semi-supervised learning scenarios when sequences are sparsely labeled at the word level.\n\nWeaknesses:\n- There is no experimental results that validates the proposed approach. The authors should conduct experiments to show that their approach outperforms CRF in the semi-supervised scenarios they proposed.\n- The authors should compare their approach with CRF with partially labeled sequences, e.g., [1].\n\n[1] Tsuboi, Yuta, et al. \"Training conditional random fields using incomplete annotations.\" Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008). 2008.", "summary_of_the_paper": "The paper formulated an approach to impose valid state transitions from t to t+1 for sequence labeling problems. The proposed approach replaces the usual conditional random fields with what is named as a prior knowledge layer (PKL), which simply multiplies a state transition matrix which zeros invalid transition probabilities. \n", "summary_of_the_review": "I would recommend to reject this paper as the theoretical novelty is insufficient, and there are no empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for using prior knowledge to constrain transitions for consecutive time steps and aims to replace conditional random fields for sequence tagging tasks in sequence labeling. However, the paper seems incomplete with no experimental results and analysis to validate the proposed ideas.
Decision: Reject