SlowGuess/ACL_2020_OCR huggingface dataset:
Chen Jia†‡ and Yue Zhang‡§
†Fudan University, China
$^{\S}$ Institute of Advanced Technology, Westlake Institute for Advanced Study, China
{jiachen,zhangyue}@westlake.edu.cn
Cross-domain NER is a challenging yet practical problem. Entity mentions can be highly different across domains. However, the correlations between entity types can be relatively more stable across domains. We investigate a multi-cell compositional LSTM structure for multi-task learning, modeling each entity type using a separate cell state. With the help of entity typed units, cross-domain knowledge transfer can be made in an entity type level. Theoretically, the resulting distinct feature distributions for each entity type make it more powerful for cross-domain transfer. Empirically, experiments on four few-shot and zero-shot datasets show our method significantly outperforms a series of multi-task learning methods and achieves the best results.
Named entity recognition (NER) is a fundamental task in information extraction, providing necessary information for relation classification (Mooney and Bunescu, 2006), event detection (Popescu et al., 2011), sentiment classification (Mitchell et al., 2013), etc. NER is challenging because entity mentions are an open set and can be ambiguous in the context of a sentence. Due to relatively high cost in manual labeling, cross-domain NER has received increasing research attention. Recently, multi-task learning methods (Yang et al., 2017; Wang et al., 2018, 2019; Zhou et al., 2019; Jia et al., 2019) have achieved great success for cross-domain NER. Other methods such as fine-tuning (Rodriguez et al., 2018), share-private (Cao et al., 2018; Lin and Lu, 2018) and knowledge distill (Yang et al., 2019) also show effectivenesses for cross-domain NER.
There are three main sources of challenges in cross-domain NER. First, instances of the same type entities can be different across domains. For example, typical person names can include "Trump"
and "Clinton" in the political news domain, but "James" and "Trout" in the sports domain. Second, different types of entities can exhibit different degrees of dissimilarities across domains. For example, a large number of location names are shared in the political news domain and the sports domain, such as "Barcelona" and "Los Angeles", but the case is very different for organization names across these domains. Third, even types of entities can be different across domains. For example, while disease names are a type of entities in the medical domain, it is not so in the biochemistry domain.
We investigate a multi-cell compositional LSTM structure to deal with the above challenges by separately and simultaneously considering the possibilities of all entity types for each word when processing a sentence. As shown in Figure 1, the main idea is to extend a standard LSTM structure by using a separate LSTM cell to model the state for each entity type in a recurrent step. Intuitively, the model differs from the baseline LSTM by simultaneously considering all possible entity types. A compositional cell (C cell) combines the entity typed cells (ET cells) for the next recurrent state transition by calculating a weighted sum of each ET cell, where the weight of each ET cell corresponds to the probability of its corresponding entity type. Different from naive parameter sharing on LSTM (Yang et al., 2017), source domain and target domain in our multi-task learning framework share only the ET cells corresponding to the same entity types and the same C cell, but not for the domain-specific ET cells. In this way, our model learns domain-invariant in the entity level.
Intuitively, our model addresses the above challenges by modeling entity type sequences more explicitly, which are relatively more robust across domains compared with entity instances. For example, the pattern "PER O PER O LOC" can exist in both the political and sports domains, despite
5906
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5906-5917 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(a) Baseline LSTM unit.
(b) Multi-cell compositional LSTM unit.
Figure 1: Overall structures. The red, blue and purple in (c) represent target, source and shared parts, respectively.
(c) Multi-task learning framework.
that the specific PER instances can be different. In addition, thanks to the merging operation at each step, our method effectively encodes multiple entity type sequences in linear time by having a sausage shaped multi-cell LSTM. Thus it allows us to learn distributional differences between entity type chains across domains. This effectively reduces the confusions of different entities when source domain and target domain have different entity types in few-shot transfer, where the target domain has a few training data. In zero-shot transfer where the target domain has no training data, a target-domain LM transfers source-domain knowledge. This knowledge transfer is also in the entity level thanks to the compositional weights which are supervised by gold-standard entity type knowledge in source-domain training.
Theoretically, our method creates distinct feature distributions for each entity type across domains, which can give better transfer learning power compared to representation networks that do not explicitly differentiate entity types (§3.4). Empirically, experiments on four few-shot and zero-shot datasets show that our method gives significantly better results compared to standard BiLSTM baselines with the same numbers of parameters. In addition, we obtain the best results on four cross-domain NER datasets. The code is released at https://github.com/jiachenwestlake/Multi-Cell_LSTM.
Given a sentence  $\mathbf{x} = [x_{1},\ldots ,x_{m}]$ , the vector representation  $\mathbf{w}_t$  for each word  $x_{t}$  is the concatenation of its word embedding and the output of a character level CNN, following Yang et al. (2018). A bi-directional LSTM encoder is used to obtain sequence level features  $\mathbf{h} = [\mathbf{h}_1,\dots ,\mathbf{h}_m]$ . We use the forward LSTM component to explain the de
tails in the following subsections. Finally, a CRF layer outputs the label sequence  $\mathbf{y} = l_{1},\dots ,l_{m}$
2.1 Baseline LSTM
We adopt the standard LSTM (Graves and Schmidhuber, 2005) for the baseline. At each time step  $t$ $(t\in [1,\dots,m])$ , the baseline calculates a current hidden vector  $\mathbf{h}^{(t)}$  based on a memory cell  $\mathbf{c}^{(t)}$ . In particular, a set of input gate  $\mathbf{i}^{(t)}$ , output gate  $\mathbf{o}^{(t)}$  and forget gate  $\mathbf{f}^{(t)}$  are calculated as follows:
$$
\left[ \begin{array}{c} \mathbf {i} ^ {(t)} \\ \mathbf {o} ^ {(t)} \\ \mathbf {f} ^ {(t)} \\ \widetilde {\mathbf {c}} ^ {(t)} \end{array} \right] = \left[ \begin{array}{c} \sigma \\ \sigma \\ \sigma \\ \tanh  \end{array} \right] \left(\mathbf {W} [ \mathbf {h} ^ {(t - 1)}; \mathbf {w} ^ {(t)} ] + \mathbf {b}\right) \tag {1}
$$
$$
\mathbf {c} ^ {(t)} = \mathbf {i} ^ {(t)} \odot \widetilde {\mathbf {c}} ^ {(t)} + \mathbf {f} ^ {(t)} \odot \mathbf {c} ^ {(t - 1)}
$$
$$
\mathbf {h} ^ {(t)} = \mathbf {o} ^ {(t)} \odot \tanh  (\mathbf {c} ^ {(t)}),
$$
where  $[\mathbf{W};\mathbf{b}]$  are trainable parameters.  $\sigma$  represents the sigmoid activation function.
2.2 Multi-Cell Compositional LSTM
As shown in Figure 1 (b), we split cell computation in the baseline LSTM unit into  $l$  copies, each corresponding to one entity type. These cells are shown in black. A compositional cell (shown in red) is used to merge the entity typed LSTM cells into one cell state for calculating the final hidden vector. In this process, a weight is assigned to each entity type according to the local context.
Entity typed LSTM cells (ET cells). Given  $\mathbf{w}^{(t)}$  and  $\hat{\mathbf{h}}^{(t - 1)}$ , the input gate  $\mathbf{i}_k^{(t)}$  and the temporary memory cell state  $\widetilde{\mathbf{c}}_k^{(t)}$  of the  $k$ -th ( $k \in [1, \dots, l]$ ) entity typed cells (ET cells) are computed as:
$$
\left[ \begin{array}{l} \mathbf {i} _ {k} ^ {(t)} \\ \widetilde {\mathbf {c}} _ {k} ^ {(t)} \end{array} \right] = \left[ \begin{array}{l} \sigma \\ \tanh  \end{array} \right] \left(\mathbf {W} _ {k} [ \hat {\mathbf {h}} ^ {(t - 1)}; \mathbf {w} ^ {(t)} ] + \mathbf {b} _ {k}\right), \tag {2}
$$
where the  $[\mathbf{W}_k;\mathbf{b}_k]$  represent the trainable parameters specific to the  $k$ -th ET cell.
Then a copy of the compositional memory cell state  $\hat{\mathbf{c}}^{(t - 1)}$  of the previous time step  $(t - 1)$  is used to update the temporary memory cell state.
$$
\mathbf {c} _ {k} ^ {(t)} = \mathbf {i} _ {k} ^ {(t)} \odot \widetilde {\mathbf {c}} _ {k} ^ {(t)} + \left(\mathbf {1} - \mathbf {i} _ {k} ^ {(t)}\right) \odot \hat {\mathbf {c}} ^ {(t - 1)} \tag {3}
$$
The above operations are repeated for  $l$  ET cells with the same  $\hat{\mathbf{c}}^{(t - 1)}$ . We finally acquire a list of ET cell states  $[\mathbf{c}_1^{(t)},\dots ,\mathbf{c}_l^{(t)}]$
Compositional LSTM cell (C cell). For facilitating integration of ET cells, a input gate  $\hat{\mathbf{i}}^{(t)}$  and a temporary cell state  $\widetilde{\hat{\mathbf{c}}}^{(t)}$  of the compositional cell (C cell) are computed similarly to those of the ET cells, but another output gate  $\hat{\mathbf{o}}^{(t)}$  is added, which are computed as follows:
$$
\left[ \begin{array}{c} \hat {\mathbf {i}} ^ {(t)} \\ \hat {\mathbf {o}} ^ {(t)} \\ \widetilde {\mathbf {c}} ^ {(t)} \end{array} \right] = \left[ \begin{array}{c} \sigma \\ \sigma \\ \tanh \end{array} \right] \left(\hat {\mathbf {W}} [ \hat {\mathbf {h}} ^ {(t - 1)}; \mathbf {w} ^ {(t)} ] + \hat {\mathbf {b}}\right), \quad (4)
$$
where  $[\hat{\mathbf{W}};\hat{\mathbf{b}}]$  are trainable parameters of the C cell.
Merging. We use the temporary cell state of the C cell  $\widetilde{\hat{\mathbf{c}}}^{(t)}$  to weigh the internal representations of ET cells  $[\mathbf{c}_1^{(t)},\dots,\mathbf{c}_l^{(t)}]$  for obtaining a compositional representation. To this end, additive attention (Dzmitry et al., 2015) is used, which achieves better results in our development compared with other attention mechanism (Vaswani et al., 2017). The temporary memory cell state of the C cell  $\hat{\mathbf{c}}_{\alpha}^{(t)}$  is a weighted sum of  $[\mathbf{c}_1^{(t)},\dots,\mathbf{c}_l^{(t)}]$ :
$$
\hat {\mathbf {c}} _ {\alpha} ^ {(t)} = \sum_ {k = 1} ^ {l} \alpha_ {k} ^ {(t)} \mathbf {c} _ {k} ^ {(t)} \quad s. t. \sum_ {k = 1} ^ {l} \alpha_ {k} ^ {(t)} = 1 \tag {5}
$$
The weight  $\alpha_{k}^{(t)}$  reflects the similarity between  $\widetilde{\hat{\mathbf{c}}}^{(t)}$  and the  $k$ -th ET cell state  $\mathbf{c}_k^{(t)}$ .  $\alpha_{k}^{(t)}$  is computed as:
$$
I _ {k} ^ {(t)} = \mathbf {v} ^ {\top} \tanh  (\mathbf {P} \tilde {\hat {\mathbf {c}}} ^ {(t)} + \mathbf {Q} \mathbf {c} _ {k} ^ {(t)})
$$
$$
\alpha_ {k} ^ {(t)} = \frac {\exp \left(I _ {k} ^ {(t)}\right)}{\sum_ {j = 1} ^ {l} \exp \left(I _ {j} ^ {(t)}\right)}, \tag {6}
$$
where  $[\mathbf{P};\mathbf{Q};\mathbf{v}]$  are trainable parameters. The memory cell state of the C cell is updated as:
$$
\hat {\mathbf {c}} ^ {(t)} = \hat {\mathbf {i}} ^ {(t)} \odot \hat {\mathbf {c}} _ {\alpha} ^ {(t)} + (\mathbf {1} - \hat {\mathbf {i}} ^ {(t)}) \odot \hat {\mathbf {c}} ^ {(t - 1)} \tag {7}
$$
Finally, we obtain the hidden state  $\hat{\mathbf{h}}^{(t)}$ :
$$
\hat {\mathbf {h}} ^ {(t)} = \hat {\mathbf {o}} ^ {(t)} \odot \tanh  (\hat {\mathbf {c}} ^ {(t)}) \tag {8}
$$
2.3 Training Tasks
Below we discuss the two auxiliary tasks before introducing the main NER task. The auxiliary tasks are designed in addition to the main NER task in order to better extract entity type knowledge from a set of labeled training data for training ET cells and C cell. Formally, denote a training set as  $\mathcal{D}_{ent} = \{(x^n,e^n)\}_{n = 1}^N$ , where each training instance consists of word sequence  $x = [x_{1},\ldots ,x_{m}]$  and its corresponding entity types  $e = [e_1,\dots ,e_m]$ . Here each entity type  $e_t$  is a label such as [PER, O, Loc,...] without segmentation tags (e.g., B/I/E).
Entity type prediction. Given the ET cell states of  $x_{t}$ :  $\mathbf{c}^{(t)} = [\overleftarrow{\mathbf{c}}_1^{(t)} \oplus \overleftarrow{\mathbf{c}}_1^{(t)}, \dots, \overrightarrow{\mathbf{c}}_l^{(t)} \oplus \overleftarrow{\mathbf{c}}_l^{(t)}]$ , we define the aligned entity distribution for  $x_{t}$ :
$$
p \left(e _ {k} \mid x _ {t}\right) = \frac {\exp \left\{\mathbf {w} _ {k} ^ {\top} \mathbf {c} _ {k} ^ {(t)} + b _ {k} \right\}}{\sum_ {j = 1} ^ {l} \exp \left\{\mathbf {w} _ {j} ^ {\top} \mathbf {c} _ {j} ^ {(t)} + b _ {j} \right\}}, \tag {9}
$$
Where  $[\mathbf{w}_k;b_k]$  are parameters specific to the  $k$  -th entity type  $e_{k}$  . The negative log-likelihood loss is used for training on  $\mathcal{D}_{ent}$  ..
$$
\mathcal {L} _ {e n t} = - \frac {1}{| \mathcal {D} _ {e n t} |} \sum_ {n = 1} ^ {N} \sum_ {t = 1} ^ {m} \log \left(p \left(e _ {t} ^ {n} \mid \boldsymbol {x} _ {t} ^ {n}\right)\right) \tag {10}
$$
Attention scoring. Similar to the entity type prediction task, given the attention scores between the temporary C cell and ET cells in Equation 6:  $\mathbf{I}^{(t)} = [(\vec{T}_1^{(t)} + \vec{T}_1^{(t)}) / 2,\dots ,(\vec{T}_l^{(t)} + \vec{T}_l^{(t)}) / 2]$ , we convert the attention scores to entity aligned distributions for  $x_{t}$  using softmax:
$$
p \left(e _ {k} \mid x _ {t}\right) = \frac {\exp \left(\mathbf {I} _ {k} ^ {(t)}\right)}{\sum_ {j = 1} ^ {l} \exp \left(\mathbf {I} _ {j} ^ {(t)}\right)} \tag {11}
$$
Similar to the loss of entity type prediction:
$$
\mathcal {L} _ {\text {a t t e n}} = - \frac {1}{| \mathcal {D} _ {\text {e n t}} |} \sum_ {n = 1} ^ {N} \sum_ {t = 1} ^ {m} \log \left(p \left(e _ {t} ^ {n} \mid \boldsymbol {x} _ {t} ^ {n}\right)\right) \tag {12}
$$
While entity type prediction brings supervised information to guide the ET cells, attention scoring introduces supervision to guide the C cell.
NER. This is the main task across domains. Standard CRFs (Ma and Hovy, 2016) are used. Given  $\mathbf{h} = [\vec{\mathbf{h}}_1\oplus \vec{\mathbf{h}}_1,\dots ,\vec{\mathbf{h}}_m\oplus \vec{\mathbf{h}}_m]$ , the output probability  $p(\boldsymbol {y}|\boldsymbol {x})$  over labels  $\boldsymbol {y} = l_{1},\ldots ,l_{m}$  is:
$$
p (\boldsymbol {y} \mid \boldsymbol {x}) = \frac {\exp \left\{\sum_ {t} \left(\mathbf {w} _ {\mathrm {C R F}} ^ {l _ {t}} \cdot \mathbf {h} _ {t} + b _ {\mathrm {C R F}} ^ {\left(l _ {t - 1} , l _ {t}\right)}\right) \right\}}{\sum_ {\boldsymbol {y} ^ {\prime}} \exp \left\{\sum_ {t} \left(\mathbf {w} _ {\mathrm {C R F}} ^ {l _ {t} ^ {\prime}} \cdot \mathbf {h} _ {t} + b _ {\mathrm {C R F}} ^ {\left(l _ {t - 1} ^ {\prime} , l _ {t} ^ {\prime}\right)}\right) \right\}}, \tag {13}
$$
where  $\pmb{y}^{\prime}$  represents an arbitrary labal sequence, and  $\mathbf{w}_{\mathrm{CRF}}^{l_t}$  is a model parameter specific to  $l_{t}$ , and  $b_{\mathrm{CRF}}^{(l_{t - 1},l_t)}$  is a bias specific to  $l_{t - 1}$  and  $l_{t}$ .
5908
Input: Source-domain NER dataset  $S_{ner}$ , target-domain NER dataset  $\mathcal{T}_{ner}$  or raw data  $\mathcal{T}_{lm}$  and entity dictionary  $D_e$   
Output: Target-domain model
A sentence-level negative log-likelihood loss is used for training on  $\mathcal{D}_{ner} = \{(x^n,y^n)\}_{n = 1}^N$ :
$$
\mathcal {L} _ {n e r} = - \frac {1}{| \mathcal {D} _ {n e r} |} \sum_ {n = 1} ^ {N} \log \left(p \left(\boldsymbol {y} ^ {n} \mid \boldsymbol {x} ^ {n}\right)\right) \tag {14}
$$
3 Transfer Learning
The multi-cell LSTM structure above is domain agnostic, and can therefore be used for in-domain NER too. However, the main goal of the model is to transfer entity sequence knowledge across domains, and therefore the ET cells and C cell play more significant roles in the transfer learning setting. Below we introduce the specific roles each cell is assigned in cross-domain settings.
3.1 Multi-Task Structure
Following the common cross-domain setting, we use source-domain NER dataset  $S_{ner}$  and the target-domain NER dataset  $\mathcal{T}_{ner}$  or raw data  $\mathcal{T}_{lm}$ . The entity type sets of source and target domains are represented as  $\mathcal{E}_d$ , where  $d \in \{S, T\}$ , respectively.
As shown in Figure 1 (c), our multi-task learning structure follows Yang et al. (2017), which consists of shared embedding layer and shared BiLSTM layer, as well as domain-specific CRF layers. Our method replaces LSTM with multi-cell LSTM, following we introduce the multi-task parameter sharing mechanism in multi-cell LSTM.
ET cells. All ET cells  $\{C_k\}_{k\in \mathcal{E}_S\cup \mathcal{E}_T}$  in multicell LSTM are a composition of entity-specific cells from both source and target domains. For each domain  $d\in \{S,T\}$ , the actually used ET cells are the domain-specific subset  $\{C_k\}_{k\in \mathcal{E}_d}$ , aiming to conserve domain-specific features.
C cell. In order to make the source and target domains share the same feature space in a word level, we use a shared C cell  $\hat{C}$  across domains.
To better leverage target-domain knowledge without target-domain NER labeled data, we conduct the auxiliary dictionary matching and language modeling tasks on target-domain raw data  $\mathcal{T}_{lm} = \{(x^n)\}_{n=1}^N$ .
Auxiliary tasks. To better extract entity knowledge from raw data, we use a pre-collected named entity dictionary  $\mathrm{D}_e$  by Peng et al. (2019) to label  $\mathcal{T}_{lm}$  and obtain a set of entity words  $\mathcal{D}_{ent}^{+}$ , which are used to train entity prediction task and attention scoring task jointly.
Language modeling. Following Jia et al. (2019), we use sampling softmax to compute forward LM probability  $p^{f}(\boldsymbol{x}_{t}|\boldsymbol{x}_{<t})$  and backward LM probability  $p^{b}(\boldsymbol{x}_{t}|\boldsymbol{x}_{>t})$ , respectively:
$$
p ^ {f} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {<   t}\right) = \frac {1}{Z} \exp \left(\mathbf {w} _ {x _ {t}} ^ {\top} \vec {\mathbf {h}} _ {t - 1} + b _ {x _ {t}}\right) \tag {15}
$$
$$
p ^ {b} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {> t}) = \frac {1}{Z} \exp \left(\mathbf {w} _ {x _ {t}} ^ {\top} \overleftarrow {\mathbf {h}} _ {t + 1} + b _ {x _ {t}}\right),
$$
where  $\mathbf{w}_x$  and  $b_{x}$  are the target word vector and bias, respectively.  $Z$  is the normalization item computed by the target word and negative samples.
The LM loss function on  $\mathcal{T}_{lm}$  is:
$$
\begin{array}{l} \mathcal {L} _ {l m} ^ {T} = - \frac {1}{2 | \mathcal {T} _ {l m} |} \sum_ {n, t = 1} ^ {N, m} \left\{\log \left(p ^ {f} \left(\boldsymbol {x} _ {t} ^ {n} \mid \boldsymbol {x} _ {<   t} ^ {n}\right)\right) \right. \tag {16} \\ \left. + \log \left(p ^ {b} \left(\boldsymbol {x} _ {t} ^ {n} \mid \boldsymbol {x} _ {> t} ^ {n}\right)\right) \right\} \\ \end{array}
$$
Algorithm 1 is the transfer learning algorithm under both supervised and unsupervised domain adaptation settings. Both source- and target-domain training instances undertake auxiliary tasks and obtain the loss  $\mathcal{L}_a$ , which is a combination of  $\mathcal{L}_{ent}$  and  $\mathcal{L}_{atten}$  weighted by  $\lambda_{ent}$  and  $\lambda_{atten}$ , respectively (line 6).
Supervised domain adaptation. The auxiliary tasks as well as source- and target-domain NER tasks (line 8, 11) form the final training objective:
$$
\mathcal {L} _ {\mathrm {S D A}} = \sum_ {d \in \{S, T \}} \left\{\lambda_ {d} \mathcal {L} _ {n e r} ^ {d} + \mathcal {L} _ {a} ^ {d} \right\} + \frac {\lambda}{2} \| \Theta \| ^ {2}, \tag {17}
$$
5909
where  $\lambda_{d}$  ( $d\in \{S,T\}$ ) are the domain weights for NER tasks.  $\lambda$  is the  $L_{2}$  regularization parameters and  $\Theta$  represents the parameters set.
Unsupervised domain adaptation. The training objective for UDA is similar to that of SDA, except for using target-domain LM task (line 13) instead of target-domain NER task:
$$
\mathcal {L} _ {\mathrm {U D A}} = \mathcal {L} _ {n e r} ^ {S} + \mathcal {L} _ {l m} ^ {T} + \mathcal {L} _ {a} ^ {S} + \mathcal {L} _ {a} ^ {T} + \frac {\lambda}{2} \| \Theta \| ^ {2} \tag {18}
$$
3.4 Theoretical Discussion
Below we show theoretically that our method in §2.2 is stronger than the baseline method in §2.1 for domain adaptation. Following Ben-David et al. (2010), a domain is defined as a pair of input distribution  $\mathcal{D}$  on  $\mathcal{X}$  and a labeling function  $y\colon \mathcal{X} \to \mathcal{Y}$  where  $\mathcal{Y}$  is a  $(l - 1)$ -simplex<sup>1</sup>. According to this definition,  $\langle \mathcal{D}_S, y_S \rangle$  and  $\langle \mathcal{D}_T, y_T \rangle$  represent source and target domains, respectively. A hypothesis is a function  $h\colon \mathcal{X} \to \{1, \dots, l\}$ , which can be a classification model.
Target-domain error is defined as the probability  $h_T$  disagrees with  $y_T$ ,  $\epsilon(h_T) = \epsilon(h_T, y_T) = \mathrm{E}_{x \sim \mathcal{D}_T}[|y_T - h_T(x)|]$ . The training target for  $h$  is to minimize a convex weighted combination of source and target errors,  $\epsilon_{\alpha}(h) = \alpha \epsilon_T(h) + (1 - \alpha) \epsilon_S(h)$ , where  $\alpha \in [0, 1)$  is the domain weight, when  $\alpha = 0$ , it is the setting of UDA.
Theorem 1 Let  $h$  be a hypothesis in class  $\mathcal{H}$ , then:
$$
\epsilon_ {T} (h) \leq \epsilon_ {\alpha} (h) + (1 - \alpha) \left(\frac {1}{2} d _ {\mathcal {H} \Delta \mathcal {H}} (\mathcal {D} _ {S}, \mathcal {D} _ {T}) + \lambda\right),
$$
$$
\begin{array}{l} d_{\mathcal{H}\Delta \mathcal{H}}\left(\mathcal{D}_{S},\mathcal{D}_{T}\right) = 2\sup_{h^{\prime},h^{\prime \prime}\in \mathcal{H}}\bigg|\operatorname{Pr}_{x\sim \mathcal{D}_{S}}\left[h^{\prime}(x)\neq h^{\prime \prime}(x)\right] \\ \left. - \operatorname * {P r} _ {x \sim \mathcal {D} _ {T}} \left[ h ^ {\prime} (x) \neq h ^ {\prime \prime} (x) \right] \right| \\ \end{array}
$$
Here  $\lambda$  is a constant that values the shared error of the ideal joint hypothesis. In  $d_{\mathcal{H}\Delta \mathcal{H}}(\mathcal{D}_S,\mathcal{D}_T)$  sup denotes the supremum of the right term for  $\forall h^{\prime},h^{\prime \prime}\in \mathcal{H}$ .  $\operatorname *{Pr}_{x\sim \mathcal{D}_S}[h'(x)\neq h''(x)]$  denotes the probability according to the distribution  $\mathcal{D}_S$  that  $h^\prime$  disagrees with  $h^{\prime \prime}$  and  $\operatorname *{Pr}_{x\sim \mathcal{D}_T}[h'(x)\neq h''(x)]$  is similar. Intuitively, the theorem states the upper bound of  $\epsilon_T(h)$  based on  $\epsilon_{\alpha}(h)$  and the distance between  $\mathcal{D}_S$  and  $\mathcal{D}_T$  in the  $\mathcal{H}\Delta \mathcal{H}$  space, which is measured as the discrepancy between the two classifiers  $h^\prime$  and  $h^{\prime \prime}$ .
The original theorem, however, concerns only one model  $h$  for transfer learning. In our supervised settings, in contrast, their CRF layers are specific to the source and target domains, respectively. Below we use  $h^*$  to denote our overall model with shared multi-cell LSTM model and domain-specific CRF layers. Further, we use  $h_1$  to denote the target domain subsystem that consists of the shared multicell LSTM model and the target-specific CRF layer, and  $h_2$  to denote its source counterpart. Theorem 1 can be extended to our settings as follows:
Lemma 1 If  $\epsilon_{\alpha}(h^{*}) = \alpha \epsilon_{T}(h_{1}) + (1 - \alpha)\epsilon_{S}(h_{2})$ , then:
$$
\epsilon_ {T} \left(h _ {1}\right) \leq 2 \epsilon_ {\alpha} \left(h ^ {*}\right) + (1 - \alpha) \left(\frac {3}{2} d _ {\mathcal {H} \Delta \mathcal {H}} \left(\mathcal {D} _ {S}, \mathcal {D} _ {T}\right) + \lambda^ {*}\right)
$$
Proof. The proof is mainly based on triangle inequalities, see Appendix A for details.  $\square$