SlowGuess/NAACL_2021_OCR huggingface dataset:
Yangming Li<sup>1</sup>, Lemao Liu<sup>1</sup>, Kaisheng Yao<sup>2</sup>
$^{2}$ Ant Group
{newmanli,redmondliu}@tencent.com
kaisheng.yao@antgroup.com
Prior methods to text segmentation are mostly at token level. Despite the adequacy, this nature limits their full potential to capture the long-term dependencies among segments. In this work, we propose a novel framework that incrementally segments natural language sentences at segment level. For every step in segmentation, it recognizes the leftmost segment of the remaining sequence. Implementations involve LSTM-minus technique to construct the phrase representations and recurrent neural networks (RNN) to model the iterations of determining the leftmost segments. We have conducted extensive experiments on syntactic chunking and Chinese part-of-speech (POS) tagging across 3 datasets, demonstrating that our methods have significantly outperformed previous all baselines and achieved new state-of-the-art results. Moreover, qualitative analysis and the study on segmenting long-length sentences verify its effectiveness in modeling long-term dependencies.
Sequence segmentation, as an important task in natural language understanding (NLU), partitions a sentence into multiple segments. The first two rows of Table 1 show a case from a syntactic chunking dataset. The input sentence is a sequence of tokens and the output segments are multiple labeled phrases. These segments are nonoverlapping and fully cover the input sentence.
In previous works, there are two dominant approaches to sequence segmentation. The most common is to regard it as a sequence labeling problem with resorting to IOB tagging scheme (Huang et al., 2015; Akbik et al., 2018; Liu et al., 2019). This method is simple yet very effective, providing tons of state-of-the-art performances. For example, Huang et al. (2015) present Bidirectional LSTM-CRF for named entity recognition (NER) and POS tagging, which adopts BiLSTM (Hochre
Sentence	Tangible capital will be about $ 115 million.
Segments	(Tangible capital, NP), (will be, VP), (about $ 115 million, NP), (.O)
IOB Tags	B-NP I-NP B-VP I-VP
B-NP I-NP I-NP I-NP O
Transition Actions	SHIFT SHIFT REDUCE-NP
SHIFT SHIFT REDUCE-VP
SHIFT SHIFT SHIFT SHIFT
REDUCE-NP OUT
Table 1: The first two rows show an example extracted from CoNLL-2000 dataset (Sang and Buchholz, 2000). The last two rows are two types of token-level labels commonly used to represent the segments.
iter and Schmidhuber, 1997) to read the input sentence and CRF (Lafferty et al., 2001) to decode the label sequence. An alternative method employs a transition-based system to incrementally segment and label an input token sequence (Zhang et al., 2016, 2018). For instance, Zhang et al. (2016) present a transition-based model for Chinese word segmentation that exploits not only character embedding but also token embedding. This type of method enjoys a number of attractive properties, including theoretically lower time complexity and capturing non-local features.
The above two approaches are essentially at token level, where a single segment is represented by multiple token-level labels (e.g., transition actions). In spite of the adequacy, the labels used to model the relation among output segments are far more than the segments themselves. As demonstrated in Figure 1, modeling the transition between the two segments, "will be" and "about $ 115 million", consumes 6 IOB tags or 8 transition actions. This ill-posed design certainly limits the full potential of segmentation models to capture the long-term dependencies among segments.
Previously, Kong et al. (2015) attempted to de
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1476-1486 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: This case illustrates how our method segments a natural language sentence.
velop segment-level models, which define a joint probability distribution over the partition of an input sentence and the labeling of the segments. Such an approach circumvents using token-level labels. However, we find that, in experiments, it underperforms current token-level models. Moreover, its use of dynamic programming (DP) incurs quadratic running time, which is too slow for both training and inference (see Section 3.7).
In this paper, we introduce a novel framework that incrementally segments a sentence at segment level. The segmentation process is iterative and incremental. At each iteration, the proposed framework determines the leftmost segment of the remaining unprocessed sequence. Under this scheme, we don't resort to token-level labels and enjoy linear time complexity. The implementation contains two stages. Firstly, we utilize LSTM-minus (Wang and Chang, 2016; Cross and Huang, 2016) technique to construct the representations for all the phrases. Secondly, we adopt LSTM to model the iterative segmentation process, which captures the strong correlation among segments. At every step, the input consists of the previous segment and the remaining unprocessed sequence, and the output is the leftmost segment.
Figure 1 depicts how our framework segments the sentence in Table 1. The output segments are obtained in an iterative and incremental manner. At each iteration, the leftmost segment of the remaining sequence is extracted and labeled. Compared with token-level models, we take much fewer steps to complete the segmentation process.
Extensive experiments have been conducted on syntactic chunking and Chinese part-of-speech (POS) tagging across 3 datasets. The proposed framework has obtained new state-of-the-art performances on all of them. Besides, qualitative study and the results on segmenting long-length sentences confirm its effectiveness in capturing
long-term dependencies.
- we present a novel framework that incrementally segments a natural language sentence at segment level. In comparison, previous approaches are mostly at token-level;
- we have notably outperformed previous baselines and established new state-of-the-art results on the 3 datasets of syntactic chunking and Chinese POS tagging. Experiments also show that our model is competitive with strong NER baselines;
- compared with prior methods, our model well captures the long-term dependencies among segments. This is strongly verified by qualitative study and the experiment on segmenting long-length sentences.
The source code of this work is available at https://github.com/LeePleased/LeftmostSeg.
We denote a  $n$ -length input sentence as  $\mathbf{x} = [x_1, x_2, \dots, x_n]$ , where  $x_i$  is a token (such as a word or a character). The output segments are represented as  $\mathbf{y} = [y_1, y_2, \dots, y_m]$ , where  $m$  is the amount of segments. Every segment  $y_k$  is denoted as a triple  $(i_k, j_k, l_k)$ .  $(i_k, j_k)$  is the span of the segment, corresponding to the phrase  $\mathbf{x}_{i_k,j_k} = [\mathbf{x}_{i_k}, \mathbf{x}_{i_k + 1}, \dots, \mathbf{x}_{j_k}]$ .  $l_k$  is from a predefined label space  $\mathcal{L}$  and specifies the label of the segment. These segments are non-overlapping and fully cover the input sentence.
The example in Table 1 is represented as
$$
\begin{array}{c} \mathbf {x} = [ \text {T a n g i b l e , c a p i t a l , w i l l , b e , a b o u t , \S}, 1 1 5, \\ \text {m i l l i o n , \cdot} ] \end{array} .
$$
$$
\mathbf {y} = [ (1, 2, \mathrm {N P}), (3, 4, \mathrm {V P}), (5, 8, \mathrm {N P}), (9, 9, \mathrm {O}) ]
$$
Figure 2: An example to show the incremental process to recognize leftmost segments.
2.1 Phrase Representation Construction
Our goal here is to construct the representation for every phrase  $\mathbf{x}_{i,j}$ . Later, we will use the phrase representation to embed the segment  $y_{k-1}$  and the unprocessed sequence  $\mathbf{x}_{i_k,n}$ .
Firstly, each token  $x_{k}$  is represented as
$$
\mathbf {e} _ {k} = \mathbf {E} ^ {\mathrm {t}} \left(x _ {k}\right) \oplus \operatorname {C h a r C N N} \left(x _ {k}\right), \tag {1}
$$
where  $\mathbf{E}^{\mathrm{t}}$  is a token embedding matrix and  $\oplus$  is the column-wise vector concatenation. Following previous works (Ma and Hovy, 2016; Liu et al., 2019), we use CharCNN to extract the character-level representations.
Secondly, we utilize bidirectional LSTMs  $\overrightarrow{f}^{\mathrm{e}}$  and  $\overleftarrow{f}^{\mathrm{e}}$  to compute the context-sensitive representation for each token  $x_{k}$ :
$$
\left\{ \begin{array}{l} \overrightarrow {\mathbf {h}} _ {k} ^ {\mathrm {c}} = \overrightarrow {f} ^ {\mathrm {e}} \left(\overrightarrow {\mathbf {h}} _ {k - 1} ^ {\mathrm {c}}, \mathbf {e} _ {k}\right) \\ \overleftarrow {\mathbf {h}} _ {k} ^ {\mathrm {c}} = \overleftarrow {f} ^ {\mathrm {e}} \left(\overleftarrow {\mathbf {h}} _ {k + 1} ^ {\mathrm {c}}, \mathbf {e} _ {k}\right). \\ \mathbf {h} _ {k} ^ {\mathrm {c}} = \overrightarrow {\mathbf {h}} _ {k} ^ {\mathrm {c}} \oplus \overleftarrow {\mathbf {h}} _ {k} ^ {\mathrm {c}} \end{array} \right. \tag {2}
$$
Inspired by previous works (Wang and Chang, 2016; Gaddy et al., 2018) in syntactic analysis, we integrate LSTM-minus features into phrase representations. Specifically, the representation  $\mathbf{h}_{i,j}^{\mathrm{p}}$  for a phrase  $\mathbf{x}_{i,j}$  is computed as the concatenation of the difference of LSTM hidden states:
$$
\mathbf {h} _ {i, j} ^ {\mathrm {p}} = \mathbf {h} _ {j} ^ {\mathrm {c}} \oplus \left(\mathbf {h} _ {j} ^ {\mathrm {c}} - \mathbf {h} _ {i} ^ {\mathrm {c}}\right) \oplus \mathbf {h} _ {i} ^ {\mathrm {c}}. \tag {3}
$$
Inside algorithm (Lari and Young, 1990) is another method to extracting phrase representation. Its advantage is to incorporate the potential hierarchical structure of natural language without using treebank annotation. For example, Drozdov et al. (2019) utilize inside algorithm to recursively compute the content representations. Despite the attractive property, its time complexity  $\mathcal{O}(n^3)$  is too inefficient to practice use.
2.2 Leftmost Segment Determination
Figure 2 demonstrates how our method iteratively and incrementally segments the sentence in Table 1. LSTM is used as the backbone to model the iterative process. At every step, the input consists of the prior segment and the unprocessed sequence, and the output is the predicted segment.
Firstly, we embed the previous segment  $y_{k - 1}$  and the unprocessed sequence  $\mathbf{x}_{i_k,n}$ . The previous segment is represented as
$$
\mathbf {h} _ {k - 1} ^ {\mathrm {s}} = \left\{ \begin{array}{c c} \mathbf {h} _ {i _ {k - 1}, j _ {k - 1}} ^ {\mathrm {p}} \oplus \mathbf {E} ^ {\mathrm {l}} \left(l _ {k - 1}\right) & k > 1 \\ \mathbf {v} & k = 1 \end{array} , \right. \tag {4}
$$
where  $\mathbf{E}^{\mathrm{l}}$  is a label embedding matrix and  $\mathbf{v}$  is a trainable vector. The unprocessed sequence is embedded as  $\mathbf{h}_{i_k,n}^{\mathrm{p}}$ .
At each iteration  $k$ , we use another LSTM  $f^{\mathrm{d}}$  to model the dependency among segments:
$$
\mathbf {h} _ {k} ^ {\mathrm {d}} = f ^ {\mathrm {d}} \left(\mathbf {h} _ {k - 1} ^ {\mathrm {d}}, \mathbf {h} _ {k - 1} ^ {\mathrm {s}} \oplus \mathbf {h} _ {i _ {k}, n} ^ {\mathrm {p}}\right). \tag {5}
$$
During training,  $i_k$  is known since the ground truth segments  $\mathbf{y}$  is reachable. At evaluation time, we set  $i_k = j_{k-1} + 1$ .
Then, we separately predict the span and the label of a segment. We define a set  $S_{k}$  containing all valid span candidates for prediction:
$$
\mathcal {S} _ {k} = \left\{\left(i _ {k}, i _ {k}\right), \left(i _ {k}, i _ {k} + 1\right), \dots , \left(i _ {k}, n\right) \right\}. \tag {6}
$$
The probability of a span  $(i,j)\in S_k$  is
$$
Q _ {k, i, j} ^ {\mathrm {s}} \propto \exp \left(\left(\mathbf {h} _ {k} ^ {\mathrm {d}}\right) ^ {\top} \mathbf {W} ^ {\mathrm {s}} \mathbf {h} _ {i, j} ^ {\mathrm {p}}\right). \tag {7}
$$
The probability of a label  $l \in \mathcal{L}$  for a span  $(i,j)$  is
$$
Q _ {k, i, j, l} ^ {\mathrm {l}} \propto \exp \left(\mathbf {E} ^ {\mathrm {l}} (l) ^ {\top} \mathbf {W} ^ {\mathrm {l}} \left(\mathbf {h} _ {i, j} ^ {\mathrm {p}} \oplus \mathbf {h} _ {k} ^ {\mathrm {d}}\right)\right). \tag {8}
$$
The matrices  $\mathbf{W}^{\mathrm{s}}$  and  $\mathbf{W}^{1}$  are learnable.
Finally, the leftmost segment is obtained as
$$
\left\{ \begin{array}{c} \left(\hat {i} _ {k}, \hat {j} _ {k}\right) = \underset {(i, j) \in \mathcal {S} _ {k}} {\arg \max } Q _ {k, i, j} ^ {\mathrm {s}} \\ \hat {l} _ {k} = \underset {l \in \mathcal {L}} {\arg \max } Q _ {k, i, j, l} ^ {\mathrm {l}}. \\ \hat {y} _ {k} = \left(\hat {i} _ {k}, \hat {j} _ {k}, \hat {l} _ {k}\right) \end{array} \right. \tag {9}
$$
The iterative process ends when the remaining sequence  $\mathbf{x}_{j_k + 1,n}$  is empty (i.e.,  $j_{k} = n$ ).
2.3 Training and Inference
During training, we use teacher forcing where every segment  $y_{k}$  is predicted using its previous ground-truth segments  $[y_1, y_2, \dots, y_{k-1}]$ . A hybrid loss is induced as
$$
\mathcal {J} = - \sum_ {y _ {k} \in \mathbf {y}} \left(\log Q _ {k, i _ {k}, j _ {k}} ^ {\mathrm {s}} + \log Q _ {k, i _ {k}, j _ {k}, l _ {k}} ^ {\mathrm {l}}\right). \tag {10}
$$
At test time, every segment  $\hat{y}_k$  is inferred in terms of the previous predicted segments  $[\hat{y}_1,\hat{y}_2,\dots ,\hat{y}_{k - 1}]$ . Algorithm 1 demonstrates how our proposed framework makes inference. Note that Algorithm 1 uses greedy search to get  $\hat{y}$  because it is both fast in speed and effective in accuracy in our experiments, although beam search may be better in accuracy.
Extensive experiments have been conducted on syntactic chunking and Chinese POS tagging across 3 datasets. Firstly, our models have obtained new
state-of-the-art performances on all the datasets. Then, we have investigated ablation studies to understand the importance of each component. Lastly, case study and the results on segmenting long-length sentences confirm the effectiveness of the proposed framework in capturing the long-term dependencies among segments.
Syntactic chunking segments a word sequence into multiple labeled groups of words. We use CoNLL-2000 dataset (Sang and Buchholz, 2000), which defines 11 syntactic chunk types (NP, VP, PP, etc.). Standard data includes a training set and a test set. Following Xin et al. (2018), we randomly sample 1000 sentences from the training set as the development set. Chinese POS tagging converts a Chinese character sequence into a token sequence and associates every word with a POS tag. We use Penn Chinese Treebank 9.0 (CTB9) (Xue et al., 2005) and Universal Dependencies 1.4 (UD1) (Nivre et al., 2016). CTB9 contains the source text in various genres, covering its previous versions (e.g., CTB6). We use the Chinese section of UD1. We follow the same format and partition of the two datasets as Shao et al. (2017).
We use the same neural network configuration for all 3 datasets. The dimensions of token embedding and label embedding are respectively set as 300 and 50. The hidden unit sizes for the encoder and decoder are 256 and 512, respectively. The layers of two LSTMs are both 2. L2 regulariza
Approach	CoNLL-2000
Segmental RNN (Kong et al., 2015)	95.08
Bi-LSTM + CRF (Huang et al., 2015)	94.46
Char-IntNet-5 (Xin et al., 2018)	95.29
GCDT (Liu et al., 2019)	95.17
Flair Embedding (Akbik et al., 2018)	96.72
Cross-view Training (Clark et al., 2018)	97.00
GCDT w/ BERT (Liu et al., 2019)	96.81
This Work	Our Model	96.13
Our Model w/ BERT	97.05
Table 2: The performances of the baselines and our models on CoNLL-2000 dataset.
Approach	PTB9	UD1
Segmental RNN (Kong et al., 2015)	92.16	90.01
Bi-RNN + CRF (single) (Shao et al., 2017)	91.89	89.41
Bi-RNN + CRF (ensemble) (Shao et al., 2017)	92.34	89.75
Lattice LSTM (Zhang and Yang, 2018)	92.13	90.09
Glyce + Lattice LSTM (Meng et al., 2019)	92.38	90.87
BERT (Devlin et al., 2019)	92.29	94.79
Glyce + BERT (Meng et al., 2019)	93.15	96.14
This Work	Our Model	92.56	91.65
Our Model w/ BERT	93.38	96.43
Table 3: The results on the two datasets of Chinese POS tagging.
tion is set as  $1 \times 10^{-6}$  and dropout ratio is set as 0.4 for reducing overfit. The above setting is obtained by grid search. We adopt Adam (Kingma and Ba, 2014) as the optimization algorithm and adopt the suggested hyper-parameters. For CoNLL2000 dataset, the cased, 300d Glove (Pennington et al., 2014) is used to initialize token embedding. CharCNN is not used in Chinese tasks. The batch size is set as 16. All our models in experiments are running on NVIDIA Tesla P100.
At test time, following previous literature, we convert the prediction of our model into IOB format and use the standard conllical script to get the F1 score. We select the model that works the best on development set, and then evaluate it on test set. In all the experiments, the improvements of our models over the baselines are statistically significant with  $p < 0.05$  under t-test.
3.2 Results on Syntactic Chunking
Our models are compared with two groups of baselines. One of them is trained without any external resources besides the training data:
Segmental RNN It's a segment-level model that defines a joint probability distribution over the partition of an input sequence and the labeling of the segments;
Bi-LSTM + CRF It utilizes bidirectional LSTM to read the input sentence and CRF to decode the label sequence;
Char-IntNet-5 It is a funnel-shaped CNN model with no down-sampling which learns a better internal structure for tokens;
GCDT It deepens the state transition path at each position in a sentence and assigns each token with a global representation learned from the entire sentence.
The other uses extra unlabeled corpora or finetunes on a pre-trained language model:
Flair Embedding It firstly pre-trains a character-level language model on a large corpus, and then uses a sequence labeling model (e.g., BiLSTM + CRF) to fine-tune on it;
Cross-view Training It designs a LSTM based sentence encoder to facilitate semi-supervised
<sup>1</sup>https://www.clips.uantwerpen.be/conll2000/chunking/conlleval.txt.
Approach	CoNLL-2000	PTB9	UD1
Our Model	96.13	92.56	91.65
w/o CharCNN	95.81	-	-
w/o LSTM-minus, w/ Inside Algorithm	96.35	92.71	91.87
w/o LSTM Decoder \( f^d \), w/ MLP	94.91	91.28	90.05
w/o Phrase Representation \( h_{l_{k-1},j_{k-1}}^p \) in Equation 4	95.78	92.09	91.27
w/o Label Representation \( E^l(l_{k-1}) \) in Equation 4	95.82	91.92	91.04
w/o Greedy Search, w/ Beam Search	96.22	92.77	91.72
Table 4: The results of ablation experiments on all three datasets.
learning. The model can benefit from massive unlabeled corpora;
GCDT w/ BERT It adopts BERT as additional token embeddings to improve GCDT.
We adopt most of the results of baselines as reported in Huang et al. (2015); Akbik et al. (2018); Xin et al. (2018). Since the evaluation method of GCDT is not standard (see the experiment setup in Luo et al. (2020)), we correct its source code<sup>2</sup> to retest the performance. The result for Segmental RNN is from our re-implementation.
Table 2 demonstrates that we have notably outperformed previous methods and achieved new state-of-the-art results on CoNLL-2000 dataset. When not using external resource, we obtain the F1 score of 96.13, which outperforms Segmental RNN by  $1.10\%$ , Bi-LSTM + CRF by  $1.77\%$ , Char-IntNet-5 by  $0.88\%$ , and GCDT by  $1.01\%$ . Note that Segmental RNN, a segment-level model, underperforms Char-IntNet-5, a token-level model, by  $0.22\%$ . To make a fair comparison with the baselines using additional unlabeled corpora, we also use BERT (Devlin et al., 2019), a powerful pretrained language model, to replace our token embedding. In this way, we achieve the F1 score of 97.05, which outnumber Flair Embedding by  $0.34\%$ , Cross-view Training by  $0.05\%$ , and GCDT w/ BERT by  $0.25\%$ . All these results verify the effectiveness of our framework.
3.3 Results on Chinese POS Tagging
We categorize the baselines into two types. The methods without using external resources:
Segmental RNN Also described in Section 3.2;
Bi-RNN + CRF It utilizes bidirectional RNN and CRF to model joint word segmentation and
POS tagging. Ensemble learning is also used to improve the results;
Lattice LSTM It's a lattice-structured LSTM that encodes Chinese characters as well as all potential tokens that match a lexicon;
Glyce + Lattice LSTM It incorporates Chinese glyph information into Lattice LSTM.
Others using a pre-trained language model:
BERT It is a language model pre-training on a large corpus. It uses the representations from the last layer to predict IOB tags;
Glyce + BERT It integrates Chinese glyph information into BERT Tagging model.
We take most of the performances of baselines from Meng et al. (2019). The results for Segmental RNN are from our re-implementation.
Table 3 shows that our models have achieved state-of-the-art results on the two datasets, PTB9 and UD1. When BERT is not used, we have obtained the F1 scores of 92.56 and 91.65, which outperform Glyce + Lattice LSTM by  $0.19\%$  and  $0.86\%$  and Bi-RNN + CRF (ensemble) by  $0.24\%$  and  $2.12\%$ . Note that Segmental RNN, a segment-level model, underperforms Lattice LSTM, a token-level model, by  $0.09\%$  on UD1. When using BERT, even without incorporating Chinese glyph information, we still obtain the F1 scores of 93.38 and 96.43, which outperform Glyce + BERT by  $0.25\%$  and  $0.30\%$ . These results further confirm the effectiveness of our proposed framework.
As shown in Table 4, we conduct ablation studies to explore the impact of every component.
$^{2}$ https://github.com/Adaxry/GCDT.
Approach	1-22 (711)	23-44 (1030)	45-66 (248)	67-88 (23)	Overall
Bi-LSTM + CRF	94.01	94.76	92.98	87.09	94.23
GCDT	94.95	95.52	93.77	87.14	95.17
Our Model	96.01	96.55	94.82	91.07	96.13
Table 5: The F1 scores for the sentences of different length ranges.
Input Sentence	Other antibodies sparked by the preparation are of a sort rarely present in large quantities in infected or ill individuals
Output Segments	GCDT	(Other antibodies, NP) (sparked, VP) (by, PP) (the preparation, NP) (are, VP) (of, PP) (a sort, NP) (rarely present, VP) (in, PP) (large quantities, NP) (in, PP) (infected or ill individuals, NP)
Our Model	(Other antibodies, NP) (sparked, VP) (by, PP) (the preparation, NP) (are, VP) (of, PP) (a sort, NP) (rarely present, ADJP) (in, PP) (large quantities, NP) (in, PP) (infected or ill individuals, NP)
Table 6: The case is from CoNLL-2000 dataset. The predicted segments of our model is correct.
Effect of Representation Learning. Following prior works (Ma and Hovy, 2016; Liu et al., 2019), we employ CharCNN to incorporate character information into word representations. By removing it, the F1 score on CoNLL-2000 decreases by  $0.33\%$ . Inside algorithm is another technique to construct phrase representations. After using it to replace LSTM-minus, the results on the three datasets are slightly improved by  $0.23\%$ ,  $0.16\%$ , and  $0.24\%$ . Our implementation of inside algorithm is the same as described in Drozdov et al. (2019). Despite the slight improvements, its time complexity  $\mathcal{O}(n^3)$  is too slow for both training and inference. Empirically, we find that the running time of inside algorithm is about 7 times slower than that of LSTM-minus.
Effect of Modeling the Dependencies Among Segments. LSTM decoder  $f^d$  models the long-term dependencies among segments. The prediction of every segment  $y_k$  is conditional on prior segments  $y_{k'}, 1 \leq k' < k$ . By replacing it with multilayer perceptron (MLP), the performances fall by  $1.29\%$ ,  $1.40\%$ , and  $1.78\%$  on the three datasets. We use both phrase representation  $\mathbf{h}_{l_{k-1}, j_{k-1}}^p$  and label representation  $\mathbf{E}^l(l_{k-1})$  to embed the previous segment  $y_{k-1}$ . After removing the phrase representations, the F1 scores decrease by  $0.37\%$ ,  $0.51\%$ , and  $0.42\%$  on the three datasets. By removing the label representations, the results also drop by  $0.32\%$ ,  $0.70\%$ , and  $0.67\%$ .
Effect of Inference Algorithm Beam search is a widely used technique in language generation