SlowGuess/ACL_2020_OCR huggingface dataset:
Yutai Hou $^{1}$ , Wanxiang Che $^{1*}$ , Yongkui Lai $^{1}$ , Zhihan Zhou $^{3}$ , Yijia Liu $^{2}$ , Han Liu $^{3}$ , Ting Liu $^{1}$
<sup>1</sup>Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology
$^{2}$ Alibaba Group  $^{3}$ Department of Computer Science, Northwestern University {ythou, car, yklai, tliu}@ir.hit.edu.cn, oneplus.lau@gmail.com zhihanzhou2020@u.northwestern.edu, hanliu@northwestern.edu
In this paper, we explore the slot tagging with only a few labeled support sentences (a.k.a. few-shot). Few-shot slot tagging faces a unique challenge compared to the other few-shot classification problems as it calls for modeling the dependencies between labels. But it is hard to apply previously learned label dependencies to an unseen domain, due to the discrepancy of label sets. To tackle this, we introduce a collapsed dependency transfer mechanism into the conditional random field (CRF) to transfer abstract label dependency patterns as transition scores. In the few-shot setting, the emission score of CRF can be calculated as a word's similarity to the representation of each label. To calculate such similarity, we propose a Label-enhanced Task-Adaptive Projection Network (L-TapNet) based on the state-of-the-art few-shot classification model - TapNet, by leveraging label name semantics in representing labels. Experimental results show that our model significantly outperforms the strongest few-shot learning baseline by 14.64 F1 scores in the one-shot setting.
Slot tagging (Tur and De Mori, 2011), a key module in the task-oriented dialogue system (Young et al., 2013), is usually formulated as a sequence labeling problem (Sarikaya et al., 2016). Slot tagging faces the rapid changing of domains, and the labeled data is usually scarce for new domains with only a few samples. Few-shot learning technique (Miller et al., 2000; Fei-Fei et al., 2006; Lake et al., 2015; Vinyals et al., 2016) is appealing in this scenario since it learns the model that borrows the prior experience from old domains and adapts to new domains quickly with only very few examples (usually one or two examples for each class).
Figure 1: Our few-shot CRF framework for slot tagging.
Previous few-shot learning studies mainly focused on classification problems, which have been widely explored with similarity-based methods (Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Yan et al., 2018; Yu et al., 2018). The basic idea of these methods is classifying an (query) item in a new domain according to its similarity with the representation of each class. The similarity function is usually learned in prior rich-resource domains and per class representation is obtained from few labeled samples (support set). It is straightforward to decompose the few-shot sequence labeling into a series of independent few-shot classifications and apply the similarity-based methods. However, sequence labeling benefits from taking the dependencies between labels into account (Huang et al., 2015; Ma and Hovy, 2016). To consider both the item similarity and label dependency, we propose to leverage the conditional random fields (Lafferty et al., 2001, CRFs) in few-shot sequence labeling (see Figure 1). In this paper, we translate the emission score of CRF into the output of the similarity-based method and calculate the transition score with a specially designed transfer mechanism.
The few-shot scenario poses unique challenges in learning the emission and transition scores of CRF. It is infeasible to learn the transition on the
$^{1}$ Code is available at: https://github.com/
AtmaHou/FewShotTagging
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1381-1393
few labeled data, and prior label dependency in source domain cannot be directly transferred due to discrepancy in label set. To tackle the label discrepancy problem, we introduce the collapsed dependency transfer mechanism. It transfers label dependency information from source domains to target domains by abstracting domain-specific labels into abstract domain-independent labels and modeling the label dependencies between these abstract labels.
It is also challenging to compute the emission scores (word-label similarity in our case). Popular few-shot models, such as Prototypical Network (Snell et al., 2017), average the embeddings of each label's support examples as label representations, which often distribute closely in the embedding space and thus cause misclassification. To remedy this, Yoon et al. (2019) propose TapNet that learns to project embedding to a space where words of different labels are well-separated. We introduce this idea to slot tagging and further propose to improve label representation by leveraging the semantics of label names. We argue that label names are often semantically related to slot words and can help word-label similarity modeling. For example in Figure 1, word rain and label name weather are highly related. To use label name semantic and achieve good-separating in label representation, we propose Label-enhanced TapNet (L-TapNet) that constructs an embedding projection space using label name semantics, where label representations are well-separated and aligned with embeddings of both label name and slot words. Then we calculate similarities in the projected embedding space. Also, we introduce a pair-wise embedding mechanism to representation words with domain-specific context.
One-shot and five-shot experiments on slot tagging and named entity recognition show that our model achieves significant improvement over the strong few-shot learning baselines. Ablation tests demonstrate improvements coming from both L-TapNet and collapsed dependency transfer. Further analysis for label dependencies shows it captures non-trivial information and outperforms transition based on rules.
Our contributions are summarized as follows: (1) We propose a few-shot CRF framework for slot tagging that computes emission score as word-label similarity and estimate transition score by transferring previously learned label dependencies. (2) We introduce the collapsed dependency transfer
mechanism to transfer label dependencies across domains with different label sets. (3) We propose the L-TapNet to leverage semantics of label names to enhance label representations, which help to model the word-label similarity.
We define sentence  $\pmb{x} = (x_{1}, x_{2}, \ldots, x_{n})$  as a sequence of words and define label sequence of the sentence as  $\pmb{y} = (y_{1}, y_{2}, \ldots, y_{n})$ . A domain  $\mathcal{D} = \left\{ (\pmb{x}^{(i)}, \pmb{y}^{(i)}) \right\}_{i=1}^{N_{D}}$  is a set of  $(\pmb{x}, \pmb{y})$  pairs. For each domain, there is a corresponding domain-specific label set  $\mathcal{L}_{\mathcal{D}} = \{\ell_{i}\}_{i=1}^{N}$ . To simplify the description, we assume that the number of labels  $N$  is same for all domains.
As shown in Figure 2, few-shot models are usually first trained on a set of source domains  $\{\mathcal{D}_1,\mathcal{D}_2,\ldots \}$ , then directly work on another set of unseen target domains  $\{\mathcal{D}_1',\mathcal{D}_2',\ldots \}$  without fine-tuning. A target domain  $\mathcal{D}_j^\prime$  only contains few labeled samples, which is called support set  $S = \left\{(\pmb {x}^{(i)},\pmb {y}^{(i)})\right\}_{i = 1}^{N_S}$ .  $S$  usually includes  $k$  examples (K-shot) for each of  $N$  labels (N-way).
The K-shot sequence labeling task is defined as follows: given a K-shot support set  $S$  and an input query sequence  $\pmb{x} = (x_{1}, x_{2}, \dots, x_{n})$ , find  $\pmb{x}$ 's best label sequence  $\pmb{y}^{*}$ :
$$
\boldsymbol {y} ^ {*} = \left(y _ {1}, y _ {2}, \dots , y _ {n}\right) = \arg \max  _ {\boldsymbol {y}} p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {S}).
$$
In this section, we first show the overview of the proposed CRF framework (§3.1). Then we discuss how to compute label transition score with collapsed dependency transfer (§3.2) and compute emission score with L-TapNet (§3.3).
Conditional Random Field (CRF) considers both the transition score and the emission score to find the global optimal label sequence for each input. Following the same idea, we build our few-shot slot tagging framework with two components: Transition Scorer and Emission Scorer.
We apply the linear-CRF to the few-shot setting by modeling the label probability of label  $\pmb{y}$  given query sentence  $\pmb{x}$  and a K-shot support set  $S$ :
$$
p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {S}) = \frac {1}{Z} \exp (\mathrm {T R A N S} (\boldsymbol {y}) + \lambda \cdot \mathrm {E M I T} (\boldsymbol {y}, \boldsymbol {x}, \mathcal {S})),
$$
where  $Z = \sum_{\pmb{y}^{\prime}\in \pmb{Y}}\exp (\mathrm{TRANS}(\pmb{y}^{\prime}) + \lambda \cdot \mathrm{EMIT}(\pmb{y}^{\prime},\pmb {x},\pmb {S}))$
$\mathrm{TRANS}(\pmb {y}) = \sum_{i = 1}^{n}f_{T}(y_{i - 1},y_{i})$  is the Transition
Figure 2: Overviews of training and testing. This figure illustrates the procedure of training the model on a set of source domains, and testing it on an unseen domain with only a support set.
Figure 3: An example of collapsed label dependency transfer. We learn a collapsed label transition  $\tilde{T}$  and obtain specific label transition  $T$  by filling each position of it with value from  $\tilde{T}$  in the same color.
Scorer output and  $\mathrm{EMIT}(\pmb {y},\pmb {x},\pmb {S}) = \sum_{i = 0}^{n}f_{E}(y_{i},\pmb {x},\pmb {S})$  is the Emission Scorer output.  $\lambda$  is a scaling parameter which balances weights of the two scores.
We take  $L_{\mathrm{CRF}} = -\log (p(\boldsymbol {y}\mid \boldsymbol {x},\mathcal{S}))$  as loss function and minimize it on data from source domains. After the model is trained, we employ Viterbi algorithm (Forney, 1973) to find the best label sequence for each input.
3.2 Transition Scorer
The transition scorer component captures the dependencies between labels. We model the label dependency as the transition probability between two labels:
$$
f _ {T} (y _ {i - 1}, y _ {i}) = p (y _ {i} \mid y _ {i - 1}).
$$
Conventionally, such probabilities are learned from training data and stored in a transition matrix  $T^{N \times N}$ , where  $N$  is the number of labels. For example,  $T_{\mathrm{B - loc,B - team}}$  corresponds to  $p(\mathrm{B - loc} \mid \mathrm{B - team})$ . But in the few-shot setting, a model faces different label sets in the source domains (train) and the target domains (test). This mismatch on labels blocks the trained transition scorer directly working on a target domain.
Collapsed Dependency Transfer Mechanism We overcome the above issue by directly model
ing the transition probabilities between abstract labels. Intuitively, we collapse specific labels into three abstract labels:  $O$ ,  $B$  and  $I$ . To distinguish whether two labels are under the same or different semantics, we model transition from  $B$  and  $I$  to the same  $B$  ( $sB$ ), a different  $B$  ( $dB$ ), the same  $I$  ( $sI$ ) and a different  $I$  ( $dI$ ). We record such abstract label transition with a Table  $\tilde{T}^{3\times 5}$  (see Figure 3). For example,  $\tilde{T}_{B,sB} = p(B - \ell_m \mid B - \ell_m)$  is the transition probability of two same  $B$  labels. And  $\tilde{T}_{B,dI} = p(I - \ell_n \mid B - \ell_m)$  is the transition probability from a  $B$  label to an  $I$  label with different types, where  $\ell_m \neq \ell_n$ .  $\tilde{T}_{O,sB}$  and  $\tilde{T}_{O,sI}$  respectively stand for the probability of transition from  $O$  to any  $B$  or  $I$  label.
To calculate the label transition probability for a new domain, we construct the transition matrix  $\pmb{T}$  by filling it with values in  $\tilde{\pmb{T}}$ . Figure 3 shows the filling process, where positions in the same color are filled by the same values. For example, we fill  $T_{\mathrm{B - loc,B - team}}$  with value in  $\tilde{T}_{B,dB}$ .
3.3 Emission Scorer
As shown in Figure 4, the emission scorer independently assigns each word an emission score with regard to each label:
$$
f _ {E} \left(y _ {i}, \boldsymbol {x}, \mathcal {S}\right) = p \left(y _ {i} \mid \boldsymbol {x}, \mathcal {S}\right).
$$
In few-shot setting, a word's emission score is calculated according to its similarity to representations of each label. To compute such emission, we propose the L-TapNet by improving TapNet (Yoon et al., 2019) with label semantics and prototypes.
3.3.1 Task-Adaptive Projection Network
TapNet is the state-of-the-art few-shot image classification model. Previous few-shot models, such as Prototypical Network, average the embeddings of each label's support example as label representations and directly compute word-label similarity in word embedding space. Different from them,
2Here, we ignore Start and End labels for simplicity. In practice, Start and End are included as two additional abstract labels.
TapNet calculates word-label similarity in a projected embedding space, where the words of different labels are well-separated. That allows TapNet to reduce misclassification. To achieve this, TapNet leverages a set of per-label reference vectors  $\Phi = [\phi_1;\dots ;\phi_N]$  as label representations. and construct a projection space based on these references. Then, a word  $x$  's emission score for label  $\ell_{j}$  is calculated as its similarity to reference  $\phi_j$  ..
$$
f _ {E} (y _ {j}, \boldsymbol {x}, \mathcal {S}) = \operatorname {S o f t m a x} \left\{\operatorname {S I M} (\mathbf {M} (E (x)), \mathbf {M} (\phi_ {j}) \right\},
$$
where  $\mathbf{M}$  is a projecting function,  $E$  is an embedder and SIM is a similarity function. TapNet shares the references  $\Phi$  across different domains and constructs  $\mathbf{M}$  for each specific domain by randomly associating the references to the specific labels.
Task-Adaptive Projection Space Construction Here, we present a brief introduction for the construction of projection space. Let  $\mathbf{c}_j$  be the average of the embedded features for words with label  $\ell_j$  in support set  $S$ . Given the  $\Phi = [\phi_1; \dots; \phi_N]$  and support set  $S$ , TapNet constructs the projector  $\mathbf{M}$  such that (1) each  $\mathbf{c}_j$  and corresponding reference vector  $\phi_j$  align closely when projected by  $\mathbf{M}$ . (2) words of different labels are well-separated when projected by  $\mathbf{M}$ .
To achieve these, TapNet first computes the alignment bias between  $\mathbf{c}_j$  and  $\phi_j$  in original embedding space, then it finds a projection  $\mathbf{M}$  that eliminates this alignment bias and effectively separates different labels at the same time. Specifically, TapNet takes the matrix solution of a linear error nulling process as the embedding projector  $\mathbf{M}$ . For the detail process, refer to the original paper.
3.3.2 Label-enhanced TapNet
As mentioned in the introduction, we argue that label names often semantically relate to slot words and can help word-label similarity modeling. To enhance TapNet with such information, we use label semantics in both label representation and construction of projection space.
Projection Space with Label Semantics Let prototype  $\mathbf{c}_j$  be the average of embeddings of words with label  $\ell_j$  in support set. And  $\mathbf{s}_j$  is semantic representation of label  $\ell_j$  and Section 3.3.3 will introduce how to obtain it in detail. Intuitively, slot values  $(\mathbf{c}_j)$  and corresponding label name  $(s_j)$  often have related semantics and they should be close
Figure 4: Emission Scorer with L-TapNet. It first constructs a projection space  $M$  by linear error nulling for given domain, and then predicts a word's emission score with its distance to label representation  $\Omega$  in the projection space.
in embedding space. So, we find a projector  $\mathbf{M}$  that aligns  $\mathbf{c}_j$  to both  $\phi_j$  and  $\mathbf{s}_j$ . The difference with TapNet is that it only aligns  $\mathbf{c}_j$  to references  $\phi_j$  but we also require alignments with label representation. The label-enhanced reference is calculated as:
$$
\boldsymbol {\psi} _ {j} = (1 - \alpha) \cdot \boldsymbol {\phi} _ {j} + \alpha \mathbf {s} _ {j},
$$
where  $\alpha$  is a balance factor. Label semantics  $\mathbf{s}_j$  makes  $\mathbf{M}$  specific for each domain. And reference  $\phi_j$  provides cross domain generalization.
Then we construct an  $\mathbf{M}$  by linear error nulling of alignment error between label enhanced reference  $\psi_{j}$  and  $\mathbf{c}_j$  following the same steps of TapNet.
Emission Score with Label Semantic For emission score calculation, compared to TapNet that only uses domain-agnostic reference  $\phi$  as label representation, we also consider the label semantics and use the label-enhanced reference  $\psi_{j}$  in label representation.
Besides, we further incorporate the idea of Prototypical Network and represent a label using a prototype reference  $\mathbf{c}_j$  as  $\Omega_j = (1 - \beta) \cdot \mathbf{c}_j + \beta \psi_j$ . Finally, the emission score of  $x$  is calculated as its similarity to label representation  $\Omega$ :
$$
f _ {E} \left(y _ {j}, \boldsymbol {x}, \mathcal {S}\right) = \operatorname {S o f t m a x} \left\{\operatorname {S I M} \left(\mathbf {M} (E (x)), \mathbf {M} \left(\boldsymbol {\Omega} _ {j}\right) \right\} \right.,
$$
where SIM is the dot product similarity function and  $E$  is a word embedding function which will be introduced in the next section.
3.3.3 Embeddings for Word and Label Name
For the word embedding function  $E$ , we proposed a pair-wise embedding mechanism. As shown in Figure 5, a word tends to mean differently when concatenated to a different context. To tackle the representation challenges for similarity computation, we consider the special query-support setting in few-shot learning and embed query and
Figure 5: An example of pair-wise embedding. When embedding query and support sentences separately (left), it is hard to tag blackbird according to its similarity to labels. But if we embed query by pairing it with different support sentences (right), the domain specific context provide blackbird certain meanings close to pet and song respectively.
Domain	1-shot	5-shot
Ave. |S|	Samples	Ave. |S|	Samples
We	6.15	2,000	28.91	1,000
Mu	7.66	2,000	34.43	1,000
Pl	2.96	2,000	13.84	1,000
Bo	4.34	2,000	19.83	1,000
Se	4.29	2,000	19.27	1,000
Re	9.41	2,000	41.58	1,000
Cr	1.30	2,000	5.28	1,000
Table 1: Overview of few-shot slot tagging data. Here, "Ave.  $|S|$ " corresponds to the average support set size of each domain. And "Sample" stands for the number of few-shot samples we build from each domain.
support words pair-wisely. Such pair-wise embedding can make use of domain-related context in support sentences and provide domain adaptive embeddings for the query words. This will further help to model the query words' similarity to domain-specific labels. To achieve this, we represent each word with self-attention over both query and support words. We first copy query sentence  $\pmb{x}$  for  $N_{S} = |\mathcal{S}|$  times, and pair them with all support sentences. Then the  $N_{S}$  pairs are passed to a BERT (Devlin et al., 2019) to get  $N_{S}$  embeddings for each query word. We represent each word as the average of  $N_{S}$  embeddings. Now, representations of query words are conditioned on domain-specific context. We use BERT as it can naturally capture the relation between sentence pairs.
To get label representation  $\mathbf{s}$ , we first concatenate abstract label name (e.g., begin and inner) and label name (e.g., weather). Then, we insert a [CLS] token at the first position, and input them into a BERT. Finally, the representation of [CLS] is used as the label semantic embedding.
We evaluate the proposed method on slot tagging and test its generalization ability on a similar
sequence labeling task: name entity recognition (NER). Due to space limitation, we only present the detailed results for 1-shot/5-shot slot tagging, which transfers the learned knowledge from source domains (training) to an unseen target domain (testing) containing only a 1-shot/5-shot support set. The results of NER are consistent and we present them in the supplementary Appendix B.
Dataset For slot tagging, we exploit the snips dataset (Coucke et al., 2018), because it contains 7 domains with different label sets and is easy to simulate the few-shot situation. The domains are Weather (We), Music (Mu), PlayList (Pl), Book (Bo), Search Screen (Se), Restaurant (Re) and Creative Work (Cr). Information about original datasets is shown in Appendix A.
To simulate the few-shot situation, we construct the few-shot datasets from original datasets, where each sample is the combination of a query data  $(\pmb{x}^q,\pmb{y}^q)$  and corresponding K-shot support set  $S$ . Table 1 shows the overview of the experiment data.
Few-shot Data Construction Different from the simple classification of single words, slot tagging is a structural prediction problem over the entire sentence. So we construct support sets with sentences rather than single words under each tag.
As a result, the normal  $N$ -way  $K$ -shot few-shot definition is inapplicable for few-shot slot tagging. We cannot guarantee that each label appears  $K$  times while sampling the support sentences, because different slot labels randomly co-occur in one sentence. For example in Figure 1, in the 1-shot support set, label [B-weather] occurs twice to ensure all labels appear at least once. So we approximately construct K-shot support set  $\mathcal{S}$  following two criteria: (1) All labels within the domain appear at least  $K$  times in  $\mathcal{S}$ . (2) At least one label will appear less than  $K$  times in  $\mathcal{S}$  if any  $(x,y)$  pair is removed from it. Algorithm 1 shows the detail process.
Here, we take the 1-shot slot tagging as an example to illustrate the data construction procedure. For each domain, we sample 100 different 1-shot support sets. Then, for each support set, we sample 20 unincluded utterances as queries (query set). Each support-query-set pair forms one few-shot episode.
<sup>3</sup>Due to the removing step, Algorithm 1 has a preference for sentences with more slots. So in practice, we randomly skip removing by the chance of  $20\%$ .
Algorithm 1: Minimum-including
Input: # of shot  $K$ , domain  $\mathcal{D}$ , label set  $\mathcal{L}_{\mathcal{D}}$
1: Initialize support set  $S = \{\}$ ,  $\mathrm{Count}_{\ell_j} = 0$  ( $\forall \ell_j \in \mathcal{L}_{\mathcal{D}}$ )
2: for  $\ell$  in  $\mathcal{L}_{\mathcal{D}}$  do
3: for each  $(\pmb{x}^{(i)},\pmb{y}^{(i)})$  in  $S$  do
4: Return  $\mathcal{S}$
Eventually, we get 100 episodes and  $100 \times 20$  samples (1 query utterance with a support set) for each domain.
Evaluation To test the robustness of our framework, we cross-validate the models on different domains. Each time, we pick one target domain for testing, one domain for development, and use the rest domains as source domains for training. So for slot tagging, all models are trained on 10,000 samples, and validated as well as tested on 2,000 samples respectively.
When testing model on a target domain, we evaluate F1 scores within each few-shot episode. Then we average 100 F1 scores from all 100 episodes as the final result to counter the randomness from support-sets. All models are evaluated on same support-query-set pairs for fairness.
To control the nondeterministic of neural network training (Reimers and Gurevych, 2017), we report the average score of 10 random seeds.
Hyperparameters We use the uncased BERT-Base (Devlin et al., 2019) to calculate contextual embeddings for all models. We use ADAM (Kingma and Ba, 2015) to train the models with batch size 4 and a learning rate of 1e-5. For the CRF framework, we learn the scaling parameter  $\lambda$  during training, which is important to get stable results. For L-TapNet, we set  $\alpha$  as 0.5 and  $\beta$  as 0.7. We fine-tune BERT with Gradual Unfreezing trick (Howard and Ruder, 2018). For both proposed and baseline models, we take early
stop in training and fine-tuning when there is no loss decay withing a fixed number of steps.