SlowGuess/ACL_2020_OCR huggingface dataset:
Yuanhe Tian $^{\text{♥} *}$ , Yan Song $^{\text{♣} \dagger}$ , Xiang Ao $^{\text{♣}}$ , Fei Xia $^{\text{♥}}$ , Xiaojun Quan $^{\text{△}}$ , Tong Zhang $^{\text{◇}}$ , Yonggang Wang $^{\text{♣}}$
$\text{心}$  University of Washington, \*Sinovation Ventures
\*Chinese Academy of Sciences, Sun Yat-sen University
$\text{心}\{\mathrm{yhtian,}$  fxia  $\} @uw.edu$  \*clksong@gmail.com
\*aoxiang@ict.ac.cn  $\triangle$ quanxj3@mail.sysu.edu.cn
$\diamond$  tongzhang@ust.hk  $\spadesuit$  wangyonggang@chuangxin.com
Chinese word segmentation (CWS) and part-of-speech (POS) tagging are important fundamental tasks for Chinese language processing, where joint learning of them is an effective one-step solution for both tasks. Previous studies for joint CWS and POS tagging mainly follow the character-based tagging paradigm with introducing contextual information such as n-gram features or sentential representations from recurrent neural models. However, for many cases, the joint tagging needs not only modeling from context features but also knowledge attached to them (e.g., syntactic relations among words); limited efforts have been made by existing research to meet such needs. In this paper, we propose a neural model named TwASP for joint CWS and POS tagging following the character-based sequence labeling paradigm, where a two-way attention mechanism is used to incorporate both context feature and their corresponding syntactic knowledge for each input character. Particularly, we use existing language processing toolkits to obtain the auto-analyzed syntactic knowledge for the context, and the proposed attention module can learn and benefit from them although their quality may not be perfect. Our experiments illustrate the effectiveness of the two-way attentions for joint CWS and POS tagging, where state-of-the-art performance is achieved on five benchmark datasets. $^{1}$
Chinese word segmentation (CWS) and part-of-speech (POS) tagging are two fundamental and crucial tasks in natural language processing (NLP) for Chinese. The former one aims to find word
He will report the content of the book to the whole class.
Figure 1: An example sentence with CWS and POS tagging results, where the ambiguous part (in green color) has dependencies from distant words (in yellow color).
boundaries in a sentence and the latter, on the top of segmentation results, assigns a POS tag to each word to indicate its syntactical property in the sentence. To effectively perform CWS and POS tagging, combining them into a joint task is proved to have better performance than separately conducting the two tasks in a sequence (Ng and Low, 2004). Therefore, many studies were proposed in the past decade for joint CWS and POS tagging (Jiang et al., 2008, 2009; Sun, 2011; Zeng et al., 2013; Zheng et al., 2013; Kurita et al., 2017; Shao et al., 2017; Zhang et al., 2018). These studies, regardless of whether they used conventional approaches (Jiang et al., 2008, 2009; Sun, 2011; Zeng et al., 2013) or deep learning based approaches (Zheng et al., 2013; Kurita et al., 2017; Shao et al., 2017; Zhang et al., 2018), focused on incorporating contextual information into their joint tagger.
In addition, it is well known that syntactic structure is also able to capture and provide the information of long-distance dependencies among words. For example, Figure 1 shows an example of local ambiguity, where the green highlighted part has two possible interpretations – “报告_VV/书_NN” (report a book) and “报告书_NN” (the report). The ambiguity can be resolved with syntactic analysis; for instance, the dependency structure, if available, would prefer the first interpretation. While the subject and the object of the sentence (highlighted in yellow) are far away from the ambiguous part in
$^{1}$ TwASP (code and the best performing models) is released at https://github.com/SVAIGBA/TwASP.
8286
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8286-8296 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: The architecture of TwASP for the joint CWS and POS tagging with the two-way attention mechanism, which is presented with example context features and their dependency knowledge (highlighted in yellow) from auto-analyzed results for a character (i.e., "分" (split) highlighted in green) in the given sentence.
the surface word order, they are much closer in the dependency structure (the subject depends on "报告_VV" and "书_NN" depends on the object). This example shows that syntactic structure provides useful cues for CWS and POS tagging.
Syntactic knowledge can be obtained from manually constructed resources such as treebanks and grammars, but such resources require considerate efforts to create and might not be available for a particular language or a particular domain. A more practical alternative is to use syntactic structures automatically generated by off-the-shelf toolkits. Some previous studies (Huang et al., 2007; Jiang et al., 2009; Wang et al., 2011; Zhang et al., 2018) verified the idea for this task by learning from autoprocessed corpora. However, their studies treat auto-processed corpora as gold reference and thus are unable to distinguishly use it according to its quality (the resulted knowledge is not accurate in most cases). Therefore, the way to effectively leverage such auto-generated knowledge for the joint CWS and POS tagging task is not fully explored.
In this paper, we propose a neural model named TwASP with a two-way attention mechanism to improve joint CWS and POS tagging by learning from auto-analyzed syntactic knowledge, which are generated by existing NLP toolkits and provide necessary (although not perfect) information for the task. In detail, for each input character, the proposed attention module extracts the context features associated with the character and their corresponding knowledge instances according to the
auto-analyzed results, then computes the attentions separately for features and knowledge in each attention way, and finally concatenates the attentions from two ways to guide the tagging process. In doing so, our model can distinguish the important auto-analyzed knowledge based on their contributions to the task and thus avoid being influenced by some inferior knowledge instances. Compared to another prevailing model, i.e., key-value memory networks (Miller et al., 2016), which can learn from pair-wisely organized information, the two-way attentions not only are able to do so, but also fully leverage features and their knowledge rather than using one to weight the other. We experiment with three types of knowledge, namely, POS labels, syntactic constituents, and dependency relations, in our experiments. The experimental results on five benchmark datasets illustrate the effectiveness of our model, where state-of-the-art performance for the joint task is achieved on all datasets. We also perform several analyses, which confirm the validity of using two-way attentions and demonstrate that our model can be further improved by synchronously using multiple types of knowledge.
2 The Model
The architecture of TwASP is illustrated in Figure 2. The left part shows the backbone of the model for the joint CWS and POS tagging following
2We explain it in later part of the paper that, the output of key-value memory networks mainly rely on the value embeddings, where keys are used to weight such embeddings.
8287
(c)
Figure 3: Examples of context features and their corresponding knowledge from (a) POS labels, (b) syntactic constituents and (c) dependency relations. Features and knowledge for the character "分" are highlighted in yellow.
the character-based sequence labeling paradigm, where the input is a character sequence  $\mathcal{X} = x_{1}x_{2}\dots x_{i}\dots x_{l}$  and the output is a sequence of joint labels  $\mathcal{Y} = y_{1}y_{2}\dots y_{i}\dots y_{l}$ . To enhance the backbone paradigm, the proposed two-way attention module (as shown in the right part of Figure 2) takes the syntactic knowledge produced from the input sentence, analyzes it and then feeds it to the tagging process. In this section, we firstly introduce the auto-analyzed knowledge, then explain how the two-way attentions consume such knowledge, and finally describe how the joint CWS and POS tagging works with the resulted attentions.
2.1 Auto-analyzed Knowledge
Auto-analyzed knowledge is demonstrated to be an effective type of resources to help NLP systems understand the texts (Song et al., 2017; Seyler et al., 2018; Huang and Carley, 2019). One challenge for leveraging external knowledge for the joint task is that gold-standard annotations are extremely rare for text in most domains, especially the syntactic annotations. An alternative solution is to use off-the-shelf NLP systems to produce such knowledge, which is proved to be useful in previous studies (Huang et al., 2007; Jiang et al., 2009; Wang et al., 2011; Zhang et al., 2018). Rather than processing an entire corpus and then extracting features or training embeddings from the resulted corpus as in previous studies, our model does not treat knowledge as gold references: it generates auto-analyzed knowledge for each sentence and learns the weights of the corresponding features. Formally, for a character sequence  $\mathcal{X}$ , let
$\mathcal{S}$  and  $\kappa$  denote the lists of context features and knowledge for  $\mathcal{X}$ , respectively. For each character  $x_{i}$  in  $\mathcal{X}$ , let  $S_{i} = [s_{i,1}, s_{i,2}, \dots, s_{i,j}, \dots, s_{i,m_{i}}]$  and  $\mathcal{K}_{i} = [k_{i,1}, k_{i,2}, \dots, k_{i,j}, \dots, k_{i,m_{i}}]$  be the sublists of  $\mathcal{S}$  and  $\kappa$  for  $x_{i}$ . Here,  $s_{i,j}$  and  $k_{i,j}$  denote a context feature and a knowledge instance, respectively.
In this paper, we use three types of syntactic knowledge for the joint task, namely POS labels, syntactic constituents, and dependency relations, where POS labels indicate the syntactic information of individual words, syntactic constituents provide the structural grouping information for a text span, and dependencies offer dependency relations between words. Figure 3 shows an example sentence and the corresponding  $S$  and  $\mathcal{K}$ . For character "分" (highlighted in green), its  $S_{i}$  and  $\mathcal{K}_{i}$  are highlighted in yellow. In order to distinguish same knowledge appearing with different context features, we use a feature-knowledge combination tag to represent each knowledge instance (e.g., "分子  $NN$ ", "分子  $NP$ ", and "分子  $dobj$ " in Figure 3). We explain each type of knowledge below.
POS Labels Figure 3 (a) shows that, for each  $x_{i}$  (e.g.,  $x_{6} =$  "分"), we use a 2-word window for both sides to extract context features from  $S$  to form  $S_{i}$  (i.e.,  $S_{6} =$  ["分子", "结合", "成", "时"]), and then get their corresponding knowledge instances of POS labels from  $\mathcal{K}$  to form  $\mathcal{K}_{i}$  (i.e.,  $\mathcal{K}_{6} =$  ["分子  $NN$ ", "结合  $_{V}V$ ", "成  $_{V}V$ ", "时  $LC$ ”]).
Syntactic Constituents As shown in Figure 3 (b), the rule for extracting syntactic constituency knowledge is as follows. We start with the word containing the given character  $x_{i}$ , go up the con
8288
stituency tree to the first ancestor whose label is in a pre-defined syntactic label list,3 then use all the words under this node to select context features from  $S$ , and finally combine the words with the syntactic label of the node to select knowledge instances from  $\kappa$ . For example, for  $x_{6} =$  "分", the lowest syntactic node governing "分子" is NP (highlighted in yellow); thus  $S_{6} =$  ["分子"] and  $\kappa_{6} =$  ["分子  $NP$  ]. Another example is  $x_{5} =$  "成", the lowest acceptable node on its syntactic path is VP; therefore,  $S_{5} =$  ["结合", "成", "分子"] and  $\kappa_{5} =$  ["结合  $VP$  ", "成  $VP$  ", "分子  $VP$  "].
Dependency Relations Given a character  $x_{i}$ , let  $w_{i}$  be the word that contains  $x_{i}$ . The context features  $S_{i}$  include  $w_{i}$ ,  $w_{i}$ 's governor, and  $w_{i}$ 's dependents in the dependency structure; those words combined with their inbound dependency relation labels form  $\mathcal{K}_{i}$ . For example, for  $x_{6} =$  "分",  $w_{6} =$  "分子", which depends on "结合" with a dependency label  $obj$ . Therefore,  $S_{6} =$  ["分子","结合"], and  $\mathcal{K}_{6} =$  ["分子_object","结合_root"].
2.2 Two-Way Attentions
Attention has been shown to be an effective method for incorporating knowledge into NLP systems (Kumar et al., 2018; Margatina et al., 2019) but it cannot be used directly for feature and knowledge in pair-wise forms. Previous studies on the joint task normally directly concatenate the embeddings from context features and knowledge instances into the embeddings of characters (Zhang et al., 2018), which could be problematic for incorporating auto-analyzed, error-prone syntactic knowledge obtained from off-the-shelf toolkits.
For both features and their knowledge instances for  $\mathcal{X}$ , we use a two-way attention design to have separate attention for  $S$  and  $\mathcal{K}$ . Particularly, the two ways, namely, the feature way and the knowledge way, are identical in architecture, where each way has a feed-forward attention module (Raffel and Ellis, 2015). For each  $x_{i}$ , its  $S_{i}$  and  $\mathcal{K}_{i}$  are firstly fed into the feature attention way and the knowledge attention way, respectively, then computed within each way, and their final attention vectors are combined to feedback to the backbone model.
Take the feature way as an example, the attention
weight for each context feature  $s_{i,j}$  is computed by
$$
a _ {i, j} ^ {s} = \frac {\exp \left(\mathbf {h} _ {i} ^ {\top} \cdot \mathbf {e} _ {i , j} ^ {s}\right)}{\sum_ {j = 1} ^ {m _ {i}} \exp \left(\mathbf {h} _ {i} ^ {\top} \cdot \mathbf {e} _ {i , j} ^ {s}\right)} \tag {1}
$$
where  $\mathbf{h}_i$  is the vector from a text encoder for  $x_{i}$  and  $\mathbf{e}_{i,j}^{s}$  the embedding of  $s_{i,j}$ . Then we have the weighted embedding  $\mathbf{a}_i^s$  for all  $s_{i,j}$  in  $\mathcal{S}_i$  via
$$
\mathbf {a} _ {i} ^ {s} = \sum_ {j = 1} ^ {m _ {i}} a _ {i, j} ^ {s} \mathbf {e} _ {i, j} ^ {s} \tag {2}
$$
where  $\sum$  denotes a element-wise sum operation.
For the knowledge way, the same process is applied to get  $\mathbf{a}_i^k$  by distinguishing and weighting each knowledge instance  $k_{i,j}$ . Finally, the output of the two attention ways are obtained through an concatenation of the two vectors:  $\mathbf{a}_i = \mathbf{a}_i^s\oplus \mathbf{a}_i^k$
2.3 Joint Tagging with Two-way Attentions
To functionalize the joint tagging, the two-way attentions interact with the backbone model through the encoded vector  $\mathbf{h}_i$  and its output  $\mathbf{a}_i$  for each  $x_i$ .
For  $\mathbf{h}_i$ , one can apply many prevailing encoders, e.g., Bi-LSTM or BERT (Devlin et al., 2019), to get the vector list  $[\mathbf{h}_1\mathbf{h}_2\dots \mathbf{h}_i\dots \mathbf{h}_l]$  for  $\mathcal{X}$ .
Once  $\mathbf{a}_i$  is obtained, we concatenate it with  $\mathbf{h}_i$  and send it through a fully connected layer to align the dimension of the output for final prediction:
$$
\mathbf {o} _ {i} = \mathbf {W} \cdot \left(\mathbf {h} _ {i} \oplus \mathbf {a} _ {i}\right) + \mathbf {b} \tag {3}
$$
where  $\mathbf{W}$  and  $\mathbf{b}$  are trainable parameters. Afterwards, conditional random fields (CRF) is used to estimate the probability for  $y_{i}$  over all possible joint CWS and POS tags under  $x_{i}$  and  $y_{i - 1}$  by
$$
p (y _ {i} | x _ {i}) = \frac {\exp (\mathbf {W} _ {c} \cdot \mathbf {o} _ {i} + \mathbf {b} _ {c})}{\sum_ {y _ {i - 1} y _ {i}} \exp (\mathbf {W} _ {c} \cdot \mathbf {o} _ {i} + \mathbf {b} _ {c})} \tag {4}
$$
Here,  $\mathbf{W}_c$  and  $\mathbf{b}_c$  are the weight matrix and the bias vector, respectively, and they are estimated using the  $(y_{i-1}, y_i)$  tag pairs in the gold standard.
We employ five benchmark datasets in our experiments, where four of them, namely, CTB5, CTB6, CTB7, and CTB9, are from the Penn Chinese TreeBank $^{4}$  (Xue et al., 2005) and the fifth one is
$^{3}$ Following Chen et al. (2006), the list has 12 syntactic labels, namely, ADJP, ADVP, CLP, DNP, DP, DVP, LCP, LST, NP, PP, QP, and VP.
4We obtain the Penn Chinese TreeBank data from the official release of Linguistic Data Consortium. The catalog numbers for CTB5, CTB6, CTB7, and CTB9 are LDC2005T01, LDC2007T36, LDC2010T07, and LDC2016T13, respectively.
8289
Datasets	Char	Word	Sent	OOV %
CTB5	Train	805K	494K	18K	-
Dev	12K	7K	350	8.1
Test	14K	8K	348	3.5
CTB6	Train	1,056K	641K	23K	-
Dev	100K	60K	2K	5.4
Test	134K	82K	3K	5.6
CTB7	Train	1,160K	718K	31K	-
Dev	387K	237K	10K	5.5
Test	399K	245K	10K	5.2
CTB9 (general)	Train	2,643K	1,696K	106K	-
Dev	210K	136K	10K	2.9
Test	379K	242K	16K	3.1
UD	Train	156K	99K	4K	-
Dev	20K	13K	500	12.1
Test	19K	12K	500	12.4
CTB9 (genres)	BC	275K	184K	12K	2.8
BN	483K	287K	10K	5.1
CS	228K	160K	17K	5.5
DF	644K	421K	20K	3.7
MZ	403K	258K	8K	7.5
NW	427K	251K	10K	5.1
SC	430K	304K	44K	4.0
WB	342K	210K	10K	5.3
Table 1: The statistics of all experimental datasets in terms of character, word and sentence numbers. For normal splits, OOV % is computed according to the training set; for each genre in CTB9, OOV % is computed with respect to the union of other seven genres.
the Chinese part of Universal Dependencies (UD) $^{5}$  (Nivre et al., 2016). The CTB datasets are in simplified Chinese characters while the UD dataset is in traditional Chinese. Following Shao et al. (2017), we convert the UD dataset into simplified Chinese $^{6}$  before conducting experiments on it.
CTB uses 33 POS tags, and we split CTB5-CTB9 following previous studies (Wang et al., 2011; Jiang et al., 2008; Shao et al., 2017). In addition, because the data in CTB9 come from eight genres – broadcast conversation (BC), broadcast news (BN), conversational speech(CS), discussion forums (DF), magazine articles (MZ), newswire (NW), SMS/chat messages (SC), and weblog (WB) – we also use CTB9 in a cross-domain study (see Section 3.4). UD uses two POS tagsets, namely the universal tagset (15 tags) and language-specific tagset (42 tags for Chinese). We refer to the corpus with the two tagsets as UD1 and UD2, respectively, and use the official splits of train/dev/test in our experiments. The statistics for the aforementioned datasets are in Table 1.
CTB5	CTB6	CTB7	CTB9	UD
S	20K	23K	24K	41K	7K
K	SCT	POS	22K	25K	27K	46K	7K
Syn.	70K	82K	87K	141K	31K
Dep.	32K	39K	42K	77K	8K
BNP	POS	22K	26K	28K	48K	8K
Syn.	69K	81K	85K	136K	29K
Table 2: Numbers of context features  $(S)$  and their corresponding knowledge instances  $(\mathcal{K})$  for five benchmark datasets, based on the output of SCT and BNP. Note that the  $\mathcal{K}$  for the UD dataset follows the CTB criteria, because SCT and BNP were trained on CTB.
To obtain the aforementioned three types of knowledge, we use two off-the-shelf toolkits, Stanford CoreNLP Toolkit (SCT)<sup>7</sup> (Manning et al., 2014) and Berkeley Neural Parser (BNP)<sup>8</sup> (Kitaev and Klein, 2018): the former tokenizes and parses a Chinese sentence, producing POS tags, phrase structure and dependency structure of the sentence; the latter does POS tagging and syntactic parsing on a pre-tokenized sentence. Both toolkits were trained on CTB data and thus produced CTB POS tags. To extract knowledge, we firstly use SCT to automatically segment sentences and then run both SCT and BNP for POS tagging and parsing. Table 2 shows the size of  $S$  and  $\mathcal{K}$  for all the datasets.
We test the model with three encoders: two of them, namely Bi-LSTM and BERT $^{9}$  (Devlin et al., 2019), are widely used; the third encoder is ZEN $^{10}$  (Diao et al., 2019), which is a recently released Chinese encoder pre-trained with n-gram information and outperforms BERT in many downstream tasks. For the Bi-LSTM encoder, we set its hidden state size to 200 and use the character embeddings released by Shao et al. (2017) to initialize its input representations. For BERT and ZEN, we follow their default settings, e.g., 12 layers of self-attention with the dimension of 768.
For the two-way attention module, we randomly initialize the embeddings for all context features and their corresponding knowledge instances, where one can also use pre-trained embeddings (Song et al., 2018; Grave et al., 2018; Zhang et al., 2019; Yamada et al., 2020) for them. For all the
<sup>5</sup>We use its version 2.4 downloaded from https://universaldependencies.org/.
The conversation scripts are from https://github.com/skydark/nstools/tree/master/zhtools
<sup>7</sup>We use its version 3.9.2 downloaded from https://stanfordnlp.github.io/CoreNLP/.
<sup>8</sup>We download the model from https://github.com/nikitakit/self-attentive-parser.
9We use the Chinese base model from https://s3.amazon.com/models.huggingface.co/.10https://github.com/sinovation/ZEN
8290
CTB5	CTB6	CTB7	CTB9	UD1	UD2
Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint
SCT	98.02	95.49	96.62	90.85	96.53	92.73	93.63	88.23	80.50*	0.00*	80.50*	36.11*
BNP	-	95.50	-	94.43	-	92.95	-	88.09	-	0.00*	-	37.16*
Bi-LSTM	97.69	93.73	95.46	90.63	95.46	89.98	96.45	91.80	94.96	88.72	95.01	88.75
+ POS (SCT)	98.07	94.68	96.23	91.04	96.32	91.60	96.75	92.36	94.86	88.90	95.08	88.99
+ Syn. (SCT)	98.03	95.66	96.06	90.97	95.90	91.90	96.57	92.40	94.88	88.87	94.71	88.90
+ Dep. (SCT)	97.84	94.25	95.85	90.70	95.87	91.08	96.63	92.26	94.88	88.93	94.91	89.05
+ POS (BNP)	98.06	95.34	96.46	93.31	96.58	92.87	96.73	93.38	95.02	89.27	94.89	89.17
+ Syn. (BNP)	98.01	94.82	96.08	92.33	96.06	91.04	96.65	92.97	94.48	88.84	94.86	89.20
BERT	98.28	96.03	97.36	94.65	96.78	93.38	97.33	94.40	97.74	94.82	97.70	94.76
+ POS (SCT)	98.77	96.77	97.43	94.82	97.31	94.12	97.75	94.87	98.32	95.60	98.33	95.46
+ Syn. (SCT)	98.75	96.66	97.37	94.73	97.07	93.84	97.67	94.83	98.11	95.43	98.10	95.42
+ Dep. (SCT)	98.65	96.69	97.35	94.87	97.10	93.89	97.67	94.82	98.10	95.41	98.11	95.36
+ POS (BNP)	98.63	96.60	97.34	94.95	97.25	94.21	97.65	94.82	98.16	95.51	98.22	95.23
+ Syn. (BNP)	98.75	96.72	97.39	94.99	97.32	94.28	97.69	94.85	98.25	95.42	98.17	95.18
ZEN	98.61	96.60	97.35	94.70	97.09	93.80	97.64	94.64	98.14	95.15	98.02	95.05
+ POS (SCT)	98.81	96.92	97.45	94.87	97.27	94.20	97.77	94.88	98.33	95.69	98.18	95.49
+ Syn. (SCT)	98.85	96.86	97.42	94.72	97.31	94.32	97.73	94.85	98.17	95.48	98.35	95.50
+ Dep. (SCT)	98.82	96.85	97.38	94.75	97.25	94.22	97.70	94.85	98.27	95.68	98.28	95.32
+ POS (BNP)	98.72	96.83	97.47	95.02	97.24	94.18	97.69	94.82	98.26	95.52	98.22	95.28
+ Syn. (BNP)	98.83	96.83	97.44	94.95	97.25	94.18	97.67	94.86	98.22	95.49	98.20	95.45
Table 3: Experimental results (the F-scores for segmentation and joint tagging) of TwASP using different encoders with and without auto-analyzed knowledge on the five benchmark datasets. "Syn." and "Dep." refer to syntactic constituents and dependency relations, respectively. The results of SCT and BNP are also reported as references, where * marks that the segmentation and POS tagging criteria from the toolkits and the UD dataset are different.
models, we set the maximum character length of the input sequence to 300 and use negative log-likelihood loss function. Other hyper-parameters of the models are tuned on the dev set and the tuned models are evaluated on the test set for each dataset (each genre for CTB9). F-scores for word segmentation and the joint CWS-POS tags are used as main evaluation metrics $^{11}$  in all experiments.
In our main experiment, we run our TwASP on the five benchmark datasets using the three encoders, i.e., Bi-LSTM, BERT, and ZEN. The results on the F-scores of word segmentation and joint CWS and POS tagging are in Table 3, which also includes the performance of the baselines without attention and the two toolkits (i.e., SCT and BNP). The results of SCT and BNP on the UD dataset are bad because they were trained on CTB, which used different segmentation and POS tagging criteria.
There are several observations. First, for all encoders, the two-way attentions provide consistent enhancement to the baselines with different types of knowledge. Particularly, although the baseline model is well-performed when BERT (or ZEN) serves as the encoder, the attention mod