SlowGuess/NAACL_2021_OCR huggingface dataset:
Lu Xu $^{1,2}$ , Zhanming Jie $^{1,3}$ , Wei Lu $^{1}$ , Lidong Bing $^{2}$
$^{1}$  StatNLP Research Group, Singapore University of Technology and Design
2 DAMO Academy, Alibaba Group 3 ByteDance
xu LU@mymail.sutd.edu.sg, allan@bytedance.com
luwei@sutd.edu.sg, l.bing@alibaba-inc.com
It has been shown that named entity recognition (NER) could benefit from incorporating the long-distance structured information captured by dependency trees. We believe this is because both types of features – the contextual information captured by the linear sequences and the structured information captured by the dependency trees may complement each other. However, existing approaches largely focused on stacking the LSTM and graph neural networks such as graph convolutional networks (GCNs) for building improved NER models, where the exact interaction mechanism between the two different types of features is not very clear, and the performance gain does not appear to be significant. In this work, we propose a simple and robust solution to incorporate both types of features with our Synergized-LSTM (Syn-LSTM), which clearly captures how the two types of features interact. We conduct extensive experiments on several standard datasets across four languages. The results demonstrate that the proposed model achieves better performance than previous approaches while requiring fewer parameters. Our further analysis demonstrates that our model can capture longer dependencies compared with strong baselines.<sup>1</sup>
Named entity recognition (NER) is one of the most fundamental and important tasks in natural language processing (NLP). While the literature (Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2019) largely focuses on training deep language models to improve the contextualized word representations, previous studies show that
Figure 1: A sentence annotated with dependency trees and named entities. The paths to connect two entities are shown below the sentence.
the structured information such as interactions between non-adjacent words can also be important for NER (Finkel et al., 2005; Jie et al., 2017; Aguilar and Solorio, 2019).
However, sequence models such as bidirectional LSTM (Hochreiter and Schmidhuber, 1997) are not able to fully capture the long-range dependencies (Bengio, 2009). For instance, Figure 1 (top) shows one type of structured information in NER. The words "Precision Castparts Corp." can be easily inferred as ORGANIZATION by its context (i.e., Corp.). However, the second entity "PCP" could be misclassified as a PRODUCT entity if a model relies more on the context "begin trading with" but ignores the hidden information that "PCP" is the symbol of "Precision Castparts Corp."
Previous research works (Li et al., 2017; Jie and Lu, 2019; Wang et al., 2019) have been using the parse trees (Chomsky, 1956, 1969; Sandra and Taft, 2014) to incorporate such structured information. Figure 1 (Dependency Path) shows that the first entity can be connected to the second entity following the dependency tree with 5 hops. Incorporating the dependency information can be done with graph neural networks (GNNs) such as graph convolutional networks (GCNs) (Kipf and Welling, 2017). However, simply stacking the LSTM and GCN architectures for NER can only provide us with modest improvements; sometimes, it decreases performance (Jie and Lu, 2019). Based on the depen
* Lu Xu is under the Joint PhD Program between Alibaba and Singapore University of Technology and Design. The work was done when Zhanming Jie was a PhD student in Singapore University of Technology and Design.
<sup>1</sup>We make our code publicly available at https://github.com/xuuluuuu/SynLSTM-for-NER.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3457-3469 June 6-11, 2021. ©2021 Association for Computational Linguistics
dency path in Figure 1, it requires a 5-layer GCN to capture the connections between these two entities. However, deep GCN architectures often face training difficulties, which cause a performance drop (Hamilton et al., 2017b; Kipf and Welling, 2017). Directly stacking GCN and LSTM has difficulties in modeling the interaction between dependency trees and contextual information.
To address the above limitations, we propose the Synergized-LSTM (Syn-LSTM), a new recurrent neural network architecture that considers an additional graph-encoded representation to update the memory and hidden states, as shown in Figure 2. More specifically, the graph-encoded representation for each word can be obtained with GCNs. Our proposed Syn-LSTM allows the cell to receive the structured information from the graph-encoded representation. With the newly designed gating mechanism, our model is able to make independent assessments on the amounts of information to be retrieved from the word representation and the graph-encoded representation respectively. Such a mechanism allows for better integration of both contextual and structured information.
- We propose a simple and robust Syn-LSTM model to better incorporate the structured information conveyed by dependency trees. The output of the Syn-LSTM cell is jointly determined by both contextual and structured information. We adopt the classic conditional random fields (CRF) (Lafferty et al., 2001) on top of the Syn-LSTM for NER.
- We conduct extensive experiments on several standard datasets across four languages. The proposed model significantly outperforms previous approaches on these datasets.
- We show that the proposed model can capture long-distance interactions between entities. Our further analysis statistically demonstrates the proposed gating mechanism is able to aggregate the structured information selectively.
2 Synergized-LSTM
2.1 Incorporating Structured Information
To incorporate the long-range dependencies, we consider an additional graph-encoded representation  $\mathbf{g}_t$  (Figure 2) as the model input to integrate
Figure 2: Syn-LSTM cell.  $t$  is the current time step.
contextual and structured information. The graph-encoded representation  $\mathbf{g}_t$  can be derived from Graph Neural Networks (GNNs) such as GCN (Kipf and Welling, 2017), which are capable of bringing in structured information through graph structure (Hamilton et al., 2017a).
However, structured information sometimes is hard to encode, as we can see from the example in Figure 1. One naive approach is to use a deep GNN to capture such information along multiple dependency arcs between two words, which could mess up information and lead to training difficulties. A straightforward solution is to integrate both structured and contextual information via LSTM. As shown in Figure 1 (Hybrid Paths), the structured information can be passed to neighbors or context, which allows a model to use less number of GNN layers and alleviate such issues for long-range dependencies. The input to the LSTM can simply be the concatenation of word representation  $\mathbf{x}_t$  and  $\mathbf{g}_t$  at each position (Jie and Lu, 2019) $^2$ . However, because such an approach requires both  $\mathbf{x}_t$  and  $\mathbf{g}_t$  to decide the value of the input gate jointly, it could be a potential victim of two sources of uncertainties: 1) the uncertainty of the quality of graph-encoded representation  $\mathbf{g}_t$ , and 2) the uncertainty of the exact interaction mechanism between the two types of features. These may lead to sub-optimal performance, especially if the graph-encoded representation  $\mathbf{g}_t$  is unsatisfactory. Thus, we need to design a new approach to incorporate both types of information from  $\mathbf{x}_t$  and  $\mathbf{g}_t$  with a more explicit interaction mechanism, with which we hope to alleviate the above issues.
2They concatenate the current word and head word representations.
2.2 Syn-LSTM Cell
We propose the Synergized-LSTM (Syn-LSTM) to better integrate the contextual and structured information to address the above limitations. The inputs of the Syn-LSTM cell include previous cell state  $\mathbf{c}_{t-1}$ , previous hidden state  $\mathbf{h}_{t-1}$ , current cell input  $\mathbf{x}_t$ , and an additional graph-encoded representation  $\mathbf{g}_t$ . The outputs of the Syn-LSTM cell include current cell state  $\mathbf{c}_t$  and current hidden state  $\mathbf{h}_t$ . Within the cell, there are four gates: input gate  $\mathbf{i}_t$ , forget gate  $\mathbf{f}_t$ , output gate  $\mathbf{o}_t$ , and an additional new gate  $\mathbf{m}_t$  to control the flow of information. Note that the forget gate  $\mathbf{f}_t$  and output gate  $\mathbf{o}_t$  are not just looking at  $\mathbf{h}_{t-1}$  and  $\mathbf{x}_t$ ; they are also affected by the graph-encoded representation  $\mathbf{g}_t$ . The cell state  $\mathbf{c}_t$  and hidden state  $\mathbf{h}_t$  are computed as follows:
$$
\mathbf {f} _ {t} = \sigma \left(W ^ {(f)} \mathbf {x} _ {t} + U ^ {(f)} \mathbf {h} _ {t - 1} + Q ^ {(f)} \mathbf {g} _ {t} + \mathbf {b} ^ {(f)}\right) \tag {1}
$$
$$
\mathbf {o} _ {t} = \sigma \left(W ^ {(o)} \mathbf {x} _ {t} + U ^ {(o)} \mathbf {h} _ {t - 1} + Q ^ {(o)} \mathbf {g} _ {t} + \mathbf {b} ^ {(o)}\right) \tag {2}
$$
$$
\mathbf {i} _ {t} = \sigma \left(W ^ {(i)} \mathbf {x} _ {t} + U ^ {(i)} \mathbf {h} _ {t - 1} + \mathbf {b} ^ {(i)}\right) \tag {3}
$$
$$
\mathbf {m} _ {t} = \sigma \left(W ^ {(m)} \mathbf {g} _ {t} + U ^ {(m)} \mathbf {h} _ {t - 1} + \mathbf {b} ^ {(m)}\right) \tag {4}
$$
$$
\tilde {\mathbf {c}} _ {t} = \tanh  \left(W ^ {(u)} \mathbf {x} _ {t} + U ^ {(u)} \mathbf {h} _ {t - 1} + \mathbf {b} ^ {(u)}\right) \tag {5}
$$
$$
\tilde {\mathbf {s}} _ {t} = \tanh  \left(W ^ {(n)} \mathbf {g} _ {t} + U ^ {(n)} \mathbf {h} _ {t - 1} + \mathbf {b} ^ {(n)}\right) \tag {6}
$$
$$
\mathbf {c} _ {t} = \mathbf {f} _ {t} \odot \mathbf {c} _ {t - 1} + \mathbf {i} _ {t} \odot \tilde {\mathbf {c}} _ {t} + \mathbf {m} _ {t} \odot \tilde {\mathbf {s}} _ {t} \tag {7}
$$
$$
\mathbf {h} _ {t} = \mathbf {o} _ {t} \odot \tanh  \left(\mathbf {c} _ {t}\right) \tag {8}
$$
where  $\sigma$  is the sigmoid function,  $W^{(\cdot)}, U^{(\cdot)}, Q^{(\cdot)}$  and  $\mathbf{b}^{(\cdot)}$  are learnable parameters.
The additional new gate  $\mathbf{m}_t$  is used to control the information from the graph-encoded representation directly. Such a design allows the original input gates  $\mathbf{i}_t$  and our new gate  $\mathbf{m}_t$  to make independent assessments on the amounts of information to be retrieved from the word representation  $\mathbf{x}_t$  and the graph-encoded representation  $\mathbf{g}_t$  respectively. On the other hand, we also have a different candidate state  $\tilde{\mathbf{s}}_t$  to represent the cell state that corresponds to the graph-encoded representation separately.
With the proposed Syn-LSTM, the structured information captured by the dependency trees can be passed to each cell, and the additional gate  $\mathbf{m}_t$  is able to control how much structured information can be incorporated. The additional gate enables the model to feed the contextual and structured information into the LSTM cell separately. Such a mechanism allows our model to aggregate the information from linear sequence and dependency trees selectively.
Figure 3: Syn-LSTM-CRF architecture.
Similar to the previous work (Levy et al., 2018), it is also possible to show that the cell state  $\mathbf{c}_t$  implicitly computes the element-wise weighted sum of the previous states by expanding Equation 7:
$$
\begin{array}{l} \mathbf {c} _ {t} = \mathbf {f} _ {t} \odot \mathbf {c} _ {t - 1} + \mathbf {i} _ {t} \odot \tilde {\mathbf {c}} _ {t} + \mathbf {m} _ {t} \odot \tilde {\mathbf {s}} _ {t} (9) \\ = \sum_ {j = 0} ^ {t} \left(\mathbf {i} _ {j} \odot \prod_ {k = j + 1} ^ {t} \mathbf {f} _ {k}\right) \odot \tilde {\mathbf {c}} _ {j} \\ + \sum_ {j = 0} ^ {t} \left(\mathbf {m} _ {j} \odot \prod_ {k = j + 1} ^ {t} \mathbf {f} _ {k}\right) \odot \tilde {\mathbf {s}} _ {j} (10) \\ = \sum_ {j = 0} ^ {t} \mathbf {a} _ {j} ^ {t} \odot \tilde {\mathbf {c}} _ {j} + \sum_ {j = 0} ^ {t} \mathbf {q} _ {j} ^ {t} \odot \tilde {\mathbf {s}} _ {j} (11) \\ \end{array}
$$
Note that the two terms,  $\mathbf{a}_j^t$  and  $\mathbf{q}_j^t$ , are the product of gates. The value of the two terms are in the range from 0 to 1. Since the  $\tilde{\mathbf{c}}_t$  and  $\tilde{\mathbf{s}}_t$  represent contextual and structured features, the corresponding weights control the flow of information.
3 Syn-LSTM-CRF
The goal of named entity recognition is to predict the label sequence  $\mathbf{y} = \{y_1, y_2, \dots, y_n\}$  given the input sequence  $\mathbf{w} = \{w_1, w_2, \dots, w_n\}$ , where  $w_t$  represents the  $t$ -th word and  $n$  is the number of words. Our model is mainly constructed with three layers: input representation layer, bi-directional Syn-LSTM layer, and CRF layer. The architecture of our Syn-LSTM-CRF is shown in Figure 3.
Input Representation Layer Similar to the work by Lample et al. (2016), our input representation also includes the character embeddings, which are the hidden states of character-based BiLSTM. Jie and Lu (2019) highlight that the dependency relation helps to enhance the input representation. Furthermore, previous methods (Wang et al., 2018;
Wang and Lu, 2018) use embeddings of part-of-speech (POS) tags as additional input representation. The input representation  $\mathbf{x}_t$  of our model is the concatenation of the word embedding  $\mathbf{v}_t$ , the character representation  $\mathbf{e}_t$ , the dependency relation embedding  $\mathbf{r}_t$ , and the POS embedding  $\mathbf{p}_t$ :
$$
\mathbf {x} _ {t} = \left[ \mathbf {v} _ {t}; \mathbf {e} _ {t}; \mathbf {r} _ {t}; \mathbf {p} _ {t} \right] \tag {12}
$$
where both  $\mathbf{r}_t$  and  $\mathbf{p}_t$  embeddings are randomly initialized and are fine-tuned during training. For experiments with the contextualized representations (e.g., BERT (Devlin et al., 2019)), we further concatenate the contextual word representation to  $\mathbf{x}_t$ .
For our task, we employ the graph convolutional network (Kipf and Welling, 2017; Zhang et al., 2018b) to get the graph-encoded representation  $\mathbf{g}_t$ . Given a graph, an adjacency matrix  $A$  of size  $n\times n$  is able to represent the graph structure, where  $n$  is the number of nodes;  $A_{i,j} = 1$  indicates that node  $i$  and node  $j$  are connected. We transform dependency tree into its corresponding adjacency matrix  $^3 A$ , and  $A_{i,j} = 1$  denotes that node  $i$  and node  $j$  have dependency relation. Note that the purpose of graph-encoded representation  $\mathbf{g}_t$  is to incorporate the dependency information from neighbor nodes. The input and output representations of the  $l$ -th layer GCN at  $t$ -th position are denoted as  $\mathbf{g}_t^{l - 1}$  and  $\mathbf{g}_t^l$  respectively. Similar to the work by Zhang et al. (2018b), we use  $d_t = \sum_{j = 1}^n A_{t,j}$ , which is the total number of neighbors of node  $t$ , to normalize the representation before going through the nonlinear function. The GCN operation is defined as:
$$
\mathbf {g} _ {t} ^ {l} = \operatorname {R e L U} \left(\sum_ {j = 1} ^ {n} A _ {t, j} W ^ {l} \mathbf {g} _ {t} ^ {l - 1} / d _ {t} + \mathbf {b} ^ {l}\right) \tag {13}
$$
where  $W^{l}$  is a linear transformation and  $\mathbf{b}^l$  is a bias. The initial  $\mathbf{g}_t^0$  is the concatenation of word embedding  $\mathbf{v}_t$ , character embedding  $\mathbf{e}_t$ , and dependency relation embedding  $\mathbf{r}_t$ :  $\mathbf{g}_t^0 = [\mathbf{v}_t; \mathbf{e}_t; \mathbf{r}_t]$ .
Bi-directional Syn-LSTM Layer With the word representation  $\mathbf{x}_t$  and the graph-encoded representation  $\mathbf{g}_t$ , a bi-directional Syn-LSTM is applied to generate contextual representation. The forward and backward Syn-LSTM enable the model to integrate the contextual and structured information from both directions. We concatenate the hidden state  $\vec{\mathbf{h}}_t$  from forward Syn-LSTM and hidden state
Dataset	# Sent.	# Entity in Sentence Length
≤ 14	15 - 29	30 - 44	45 - 59	≥ 60
Catalan	Train	8,709	944	4,821	5,309	2,815	1,389
Dev	1,445	135	836	815	477	168
Test	1,698	243	919	946	518	284
Spanish	Train	9,022	855	4,031	6,656	4,279	1,446
Dev	1,419	125	612	911	707	260
Test	1,705	175	703	1,143	783	242
English	Train	59,924	13,309	33,853	22728	8,099	3,839
Dev	8,528	1,778	4,830	2,882	1,051	525
Test	8,262	1,785	4,673	3,171	1,082	546
Chinese	Train	36,487	8,424	21,033	17,260	8,392	7,434
Dev	6,083	1,493	3,250	2,284	1,099	978
Test	4,472	968	2,517	2,149	1,024	836
$\overleftarrow{\mathbf{h}_t}$  from backward Syn-LSTM to form the contextual representation of  $t$ -th token:  $\mathbf{h}_t = [\overrightarrow{\mathbf{h}}_t; \overleftarrow{\mathbf{h}}_t]$ .
CRF Layer The CRF (Lafferty et al., 2001) is widely used in NER tasks as it is capable of capturing the structured correlations between adjacent output labels. Given the sentence  $\mathbf{w}$  and dependency tree  $\tau$ , the probability of the label sequence  $\mathbf{y}$  is defined as:
$$
P (\mathbf {y} | \mathbf {w}, \tau) = \frac {\exp (s c o r e (\mathbf {w} , \tau , \mathbf {y}))}{\sum_ {\mathbf {y} ^ {\prime}} \exp (s c o r e (\mathbf {w} , \tau , \mathbf {y} ^ {\prime}))} \tag {14}
$$
The score function is defined as:
$$
\operatorname {s c o r e} (\mathbf {w}, \tau , \mathbf {y}) = \sum_ {t = 0} ^ {n} T _ {y t, y t + 1} + \sum_ {t = 1} ^ {n} E _ {y t} \tag {15}
$$
where  $T_{yt,y_{t + 1}}$  denotes the transition score from label  $y_{t}$  to  $y_{t + 1}$ ,  $E_{y_t}$  denotes the score of label  $y_{t}$  at the  $t$ -th position and the scores are computed using the hidden state  $\mathbf{h}_t$ . We learn the model parameters by minimizing the negative log-likelihood and employ the Viterbi algorithm to obtain the best label sequence during evaluation.
Datasets The proposed model is evaluated on four benchmark datasets: SemEval 2010 Task 1 (Recasens et al., 2010) Catalan and Spanish datasets, and OntoNotes 5.0 (Weischedel et al., 2013) English and Chinese datasets. We choose these four datasets as they have explicit dependency annotations which allow us to evaluate the effectiveness of our approach when dependency trees of different qualities are used. For SemEval 2010 Task 1 datasets, there are 4 entity types: PER, LOC and ORG and MISC. For OntoNotes 5.0 datasets, there are 18 entity types in total. Following the work
3We treat the dependency edge as undirected and add a self-loop for each node:  $A_{i,j} = A_{j,i}$  and  $A_{i,i} = 1$
by Jie and Lu (2019), we transform the parse trees into the Stanford dependency trees (De Marneffe and Manning, 2008) by using Stanford CoreNLP (Manning et al., 2014). Detailed statistics of each dataset can be found in Table 1. Intuitively, longer sentences would require the model to capture more long-distance interactions in the sentences. We present the number of entities in terms of different sentence lengths to show that these datasets have a modest amount of entities in long sentences.
Experimental Setup For Catalan, Spanish, and Chinese, we use the FastText (Grave et al., 2018) 300 dimensional embeddings to initialize the word embeddings. For OntoNotes 5.0 English, we adopt the publicly available GloVE (Pennington et al., 2014) 100 dimensional embeddings to initialize the word embeddings. For experiments with the contextualized representation, we adopt the pre-trained language model BERT (Devlin et al., 2019) for the four datasets. Specifically, we use bert-as-service (Xiao, 2018) to generate the contextualized word representation without fine-tuning. Following Luo et al. (2020), we use the cased version of BERT large model for the experiments on the OntoNotes 5.0 English data. We use the cased version of BERT base model for the experiments on the other three datasets. For the character embedding, we randomly initialize the character embeddings and set the dimension as 30, and set the hidden size of character-level BiLSTM as 50. The hidden size of GCN and Syn-LSTM is set as 200, the number of GCN layer is 2. We adopt stochastic gradient descent (SGD) to optimize our model with batch size 100, L2 regularization  $10^{-8}$ , initial learning rate  $lr$  0.2 and the learning rate is decayed with respect to the number of epoch. We select the best model based on the performance on the dev set and apply it to the test set. We use the bootstrapping t-test to compare the results.
Baselines We compare our model with several baselines with or without dependency tree information. The first one is BERT-CRF, where we apply a CRF layer on top of BERT (Devlin et al., 2019). Secondly, we compare with the BERT implementation by HuggingFace (Wolf et al., 2019). For models with dependency trees, we take the models BiLSTM-GCN-CRF and dependency
Models	Catalan	Spanish
P.	R.	F1	P.	R.	F1
BiLSTM-CRF†	76.83	63.47	69.51	78.33	69.89	73.87
BiLSTM-GCN-CRF†	81.25	75.22	78.12	84.10	79.88	81.93
GCN-BiLSTM-CRF*	80.95	74.19	77.43	84.36	79.48	81.85
DGLSTM-CRF (2019)	83.35	80.00	81.64	84.05	82.90	83.47
Syn-LSTM-CRF (Ours)	83.90	81.65	82.76	86.22	84.24	85.09
+ Contextualized Word Representation
BERT-CRF*	76.34	76.05	76.19	79.30	77.22	78.24
Wolf et al. (2019)*	82.82	85.7	84.23	81.36	85.58	83.42
BiLSTM-CRF + ELMO†	77.85	76.22	77.03	81.72	79.09	80.38
BiLSTM-CRF + BERT*	81.21	79.90	80.55	83.28	80.11	81.66
BiLSTM-GCN-CRF + ELMO†	83.68	83.16	83.42	85.31	85.19	85.25
GCN-BiLSTM-CRF + BERT*	87.60	86.39	86.99	88.07	87.46	87.76
DGLSTM-CRF (2019) + ELMO	84.71	83.75	84.22	87.79	87.33	87.56
DGLSTM-CRF + BERT*	85.92	84.50	85.20	85.67	85.00	85.33
Syn-LSTM-CRF + BERT (Ours)	89.07	89.04	89.05	89.66	90.54	90.10
Table 2: Experimental results [%] on SemEval 2010 Task 1 Catalan and Spanish test set. The models with * symbol are our implementations. The models with † symbol are retrieved from Jie and Lu (2019).
guided LSTM-CRF (DGLSTM-CRF) proposed by Jie and Lu (2019), and our implemented GCN-BiLSTM-CRF. The BiLSTM-GCN-CRF model simply stacks the GCN on top of the BiLSTM to incorporate the dependency trees. The GCN-BiLSTM-CRF model takes the concatenation of the graph-encoded representation from GCN and word embedding as input into BiLSTM. The DGLSTM-CRF takes the concatenation of the head word representation and word embedding as input into BiLSTM. Note that the original implementation of DGLSTM-CRF uses ELMo (Peters et al., 2018), but we also implement it with BERT. Besides, we compare our model with previous works that have results on these datasets.
SemEval 2010 Task 1 Table 2 shows comparisons of our model with baseline models on the SemEval 2010 Task 1 Catalan and Spanish datasets. Our Syn-LSTM-CRF model outperforms all existing models with  $F_{1}$  82.76 and 85.09 ( $p < 10^{-5}$ ) compared to DGLSTM-CRF on Catalan and Spanish datasets when FastText word embeddings are used. Our model outperforms the BiLSTM-CRF model by 13.25 and 11.22  $F_{1}$  points, and outperforms BiLSTM-GCN-CRF (Jie and Lu, 2019) model by 4.64 and 3.16 on Catalan and Spanish. The large performance gap between BiLSTM-GCN-CRF and our model indicates that SynLSTM-CRF shows better compatibility with GCN, and this confirms that simply stacking GCN on top of the BiLSTM does not perform well. Our method outperforms GCN-BiLSTM-CRF model
4We set the decay as 0.1 and the learning rate for each epoch equals to  $lr / (1 + decay*(epoch - 1))$  The experimental results on the dev set and other experimental details can be found in the Appendix.
by 5.33 and  $3.24F_{1}$  points on Catalan and Spanish. This shows that our proposed model demonstrates a better integration of contextual information and structured information. Furthermore, our proposed method brings 1.12 and  $1.62F_{1}$  points improvement on Catalan and Spanish datasets compare to the DGLSTM-CRF (Jie and Lu, 2019). The DGLSTM-CRF employs 2-layer dependency guided BiLSTM to capture grandchild dependencies, which leads to longer training time and more model parameters. However, our Syn-LSTM-CRF is able to get better performance with fewer model parameters and shorter training time because of the fewer LSTM layers. Such results demonstrate that our proposed Syn-LSTM-CRF manages to capture structured information effectively.