SlowGuess/NAACL_2022_OCR huggingface dataset:
S. K. Hong and Tae Young Jang
Samsung SDS, ML Research Center
{s.k.hong, tae10.jang}@samsung.com
In recent years, NLP has advanced greatly along with the proliferation of pre-trained language models. The pre-trained language models are also properly adapted to downstream tasks when there is sufficient labeled data. However, in real-world applications, we often encounter the deficiency of labeled data. When only given a few instances for a new task, extracting task-aware features from a pre-trained language model regardless of the adaptation is a promising alternative. In the study, we propose a novel embedding transfer method, called LEA, for leveraging pre-trained language models with even only few-shot instances. LEA derives meta-level attention aspects using our new meta-learning framework. We evaluate our method on five text classification benchmark datasets. The results show that the novel method robustly provides the competitive performance compared to recent few-shot learning methods.
A deficiency of supervised data is often experienced in real-world NLP applications. Few-shot learning aims to yield an AI-driven NLP model capable of recognizing unseen tasks using a few labeled data. Meanwhile, fine-tuning pre-trained models (PTMs) (Howard and Ruder, 2018; Devlin et al., 2019; Lan et al., 2019; Liu et al., 2019) has been the most successful approach in recent years of NLP. Unfortunately, it is still challenging to utilize PTMs (Lee et al., 2019) in few-shot learning.
To address this subtle problem (Sun et al., 2019), we propose a meta-knowledge driven self-attentive embedding transfer method, called LEA (LEarning-to-Arrend), based on a novel meta-learning framework, through which meta-level attention aspects are derived by encoding how to attend for given tasks. LEA is an efficient and practical method that facilitates the utilization of large-sized PTMs in few-shot learning.
There are the two common transfer learning paradigms in NLP: feature-based transfer (Cer et al., 2018) and fine-tuning (Houlsby et al., 2019). Our approach belongs to the feature-based transfer.
LEA includes two key ideas: (1) construction of a meta-level attention aspects dictionary and (2) inference of the task-specific attention aspects upon the arrival of a new task. The former is a process by which useful meta-level attention aspects across tasks are derived based on a particular PTM via our meta-learning framework. The latter refers to as a task-adaption process, where a subset of task-specific attention aspects is inferred by determining the top-  $k$  most relevant attention aspects from the meta-level attention aspects dictionary. While LEA can be applied to a wide variety of downstream tasks, we demonstrate LEA on few-shot text classification problems in the paper.
Few-shot text classification: In (Geng et al., 2019), INDUCTION is proposed to build class-wise embedding to represent each class using a particular dynamic routing algorithm coalesced with meta-learning. In (Bao et al., 2019), DS is introduced to keep track of underlying word distributions across all available classes and to specify important lexical features for new classes.
Meta-learning: As a metric learning-based method, (Snell et al., 2017) suggested a deep neural network, called a prototype network (PROTO), through which class representations are composed using a learning similarity metric for members of the same class. In (Sung et al., 2018), similar to PROTO, a deep neural network, called a relation network, is proposed to learn a non-linear distance metric rather than the Euclidean distance. In addition, LEO (Rusu et al., 2018) learns a low-dimensional latent embedding of the model parameters such that the classifiers are generated from the latent space into which the tasks are mapped. Frog
Human Language Technologies, pages 99 - 106
GNN (Xu and Xiang, 2021) focuses on all query-support pairs and proposes a multi-perspective aggregation based graph neural network to explicitly reflect intra-class similarity and inter-class dissimilarity.
Few-shot text classification is a task in which a classifier must be adapted to accommodate new classes using only a few labeled examples. In the literature, this is called a  $C$ -way  $K$ -shot problem in which  $K$ -labeled examples are given for each of the  $C$  number of classes. In a meta-learning setting, tasks are divided into a meta-training set  $(S^{tr})$ , meta-validation set  $(S^{val})$ , and meta-test set  $(S^{test})$  as disjoint sets of classes.
3.2 Model-Agnostic Meta-Learning
Our proposed meta training strategy follows the overall procedure of optimization-based meta-learning (Finn et al., 2017). For a parametric model  $f_{\theta}$ , MAML seeks to find task-specific parameters  $\theta_{i}$  for any new task  $\tau_{i}$  sampled from a particular distribution of tasks. For a particular task  $\tau_{i} \sim p(\tau)$ , the task data  $\mathcal{D}_{\tau_i}$  consist of  $\mathcal{D}_{\tau_i}^{tr}$  and  $\mathcal{D}_{\tau_i}^{val}$  during the meta-training phase. MAML alternates between two update processes during meta-training: (1) task-adaptation and (2) meta-optimization.
Task adaptation (or inner update): Each task learner updates its own parameters through a gradient descent using the loss evaluated based on its own training data  $\mathcal{D}_{\tau_i}^{tr}$  with the initial parameter  $\theta_{m}$  given by the outer meta-optimization process. The task-adaptation process is formulated as in Equation 1.
$$
\theta_ {\tau_ {i}} ^ {\prime} \gets \theta_ {m} - \alpha \bigtriangledown_ {\theta_ {m}} \mathcal {L} _ {\tau_ {i}} \left(f _ {\theta_ {m}}, \mathcal {D} _ {\tau_ {i}} ^ {t r}\right), \quad (1)
$$
Meta-optimization (or outer update): The metalearner updates its parameters through a gradient descent using the loss evaluated by  $\mathcal{D}_{\tau_i}^{val}$  with respect to the task-specific parameters  $\theta_{\tau_i}'$ . The meta-optimization process is formulated as in Equation 2:
$$
\theta_ {m} \gets \theta_ {m} - \beta \bigtriangledown_ {\theta_ {m}} \sum_ {\tau_ {i} \sim p (\tau)} \mathcal {L} _ {\tau_ {i}} \left(f _ {\theta_ {\tau_ {i}} ^ {\prime}}, \mathcal {D} _ {\tau_ {i}} ^ {v a l}\right), (2)
$$
where  $\mathcal{L}_{\tau_i}$  denotes a loss function for a task  $\tau_{i}$ , and the inner and outer updates are applied through their own standard gradient descent with
Figure 1: The overall architecture of LEA.
fixed learning rates  $\alpha$  and  $\beta$ , respectively, which are given as hyperparameters.
In the meta-testing phase, the meta-learner provides the initial parameters for task-specific model learners. Subsequently, each task learner is individually tailored to find the optimal parameters  $\theta_{\tau_i}^{\prime}$  by applying the above task adaptation process. In this meta-testing, the dataset of task  $\tau_{i}$  is given as  $\mathcal{D}_{\tau_i} = \left(\mathcal{D}_{\tau_i}^{tr},\mathcal{D}_{\tau_i}^{te}\right)$ .
3.3 Pre-Trained Models
We conducted all experiments with BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as the underlying PTMs in the study. Given a text input, a dummy token (CLS) is added to the beginning of the input, and another token (SEP) is added to the end of a sentence. The PTMs end up with providing the corresponding embedding vectors (i.e., denoted as [CLS] and [SEP]) for the artificial tokens as well as embeddings for original tokens for the input text. For downstream classification tasks, the special embedding vector [CLS] is typically used to make a prediction as the representative of an text instance. In this study, the [CLS] vector plays an important role in probing the distinctive properties for an incoming task. In manufacturing a task-specific embedding, we especially utilize the token-level output embeddings of the individual tokens of the  $j$ th text instance under a particular task  $\tau_{i}$ , which we denote as  $H_{j}^{\tau_{i}} = [h_{j,1}^{\tau_{i}},\dots,h_{j,L}^{\tau_{i}}]$ . Likewise, the corresponding [CLS] embedding is denoted as  $c_{j}^{\tau_{i}}$ .
The overall architecture of LEA is shown in Figure 1. It represents our meta learning framework for the task-specific feature extraction. It is trained in an end-to-end manner using our proposed meta-learning strategy. The meta training alternates two
processes: (1) deriving all valid meta-attention aspects across tasks (namely, meta-optimization), and (2) choosing a task-specific subset from all the meta-attention aspects for each task (called, task adaptation). The high-level operation is described in Algorithm 1.
4.1 Meta Attention Aspects Dictionary
In this study, the meta-level knowledge dictionary maintains all attention aspects derived across tasks  $\tau_{i}\sim p(\tau)$ . The concept was inspired by (Lin et al., 2017). The meta-attention aspects in the dictionary are established throughout the meta-optimization process during which it seeks to learn how to attend according to the distribution of tasks. Herein, we define a matrix  $W_{A}\in \mathbb{R}^{A_{N}\times u}$  as the meta-level attention aspect dictionary. In addition,  $A_{N}$  and  $u$  are the total number of attention aspects and dimension of the attention aspect, respectively.
4.2 Top- $k$ : Attention Aspects Selection through Gate Network
When a novel task  $\tau_{i}$  is given, its related attention aspects, denoted by  $W_{A}^{\tau_{i}}$ , are selectively obtained by assigning the corresponding weights to members of the meta-level attention aspects  $W_{A}$  in the task-adaptation process. Here,  $W_{A}^{\tau_{i}} \in \mathbb{R}^{k \times u}$  indicates the selected  $k$  attention aspects of the task  $\tau_{i}$ .
Note that  $k$  and  $K$  are different in that the former is the number of topmost relevant attention aspects, whereas the latter, indicates as  $K$ -shot, refers to the number of samples in few-shot learning. To do so, we assess the relevance of the task among the meta-level attention aspects  $W_{A}$ . First, each task is fed into an encoding process, which is formulated as follows:
$$
e _ {n} ^ {\tau_ {i}} = \frac {1}{N K ^ {2}} \sum_ {k _ {n} = 1} ^ {K} \sum_ {m = 1} ^ {N} \sum_ {k _ {m} = 1} ^ {K} f _ {\theta_ {r}} \left(f _ {\theta_ {e}} \left(c _ {k _ {n}} ^ {\tau_ {i}}\right), f _ {\theta_ {e}} \left(c _ {k _ {m}} ^ {\tau_ {i}}\right)\right), \tag {3}
$$
where  $e_n^{\tau_i}$  is the representative embedding for the particular class  $n$  under a given task  $\tau_{i}$ ,  $f_{\theta_r}$  indicates the relation network (Sung et al., 2018), and  $f_{\theta_e}$  is an encoder network that transforms the delegate embedding [CLS] (denoted as  $c_j^{\tau_i}$  for the case of the  $j$ th text instance of a specific task  $\tau_{i}$ ) of a text instance in PTMs (Devlin et al., 2019; Lan et al., 2019; Liu et al., 2019). As a result, the class embedding  $e_n^{\tau_i}$  is enforced to encode the pairwise relationship with other classes.
Using the aforementioned class embedding, we attempt to selectively (i.e., top- $k$ ) collect task-specific attention aspects for a given task by employing a gating mechanism (Shazeer et al., 2017). The gating output vector is calculated through the following formulation:
$$
g _ {n} ^ {\tau_ {i}} = \operatorname {s o f t m a x} \left(G \left(e _ {n} ^ {\tau_ {i}}; W _ {g}, W _ {n}, k\right)\right), \tag {4}
$$
where  $g_{n}^{\tau_{i}}$  is the gating output vector whose number of dimensions must be the same as the size of the meta-attention-aspects dictionary. The gating process  $G$  produces a sparse output vector by being parameterized with  $\{W_{g}\in \mathbb{R}^{A_{N}\times A_{N}},W_{n}\in \mathbb{R}^{A_{N}\times A_{N}},k\}$ , where the remaining values except for the  $k$  elements are forced to become zeros, and the top- $k$  weights are finally generated through a softmax function.
As a result, we can extract the top- $k$  task-specific attention aspects for the task  $\tau_{i}$ . This is formulated as follows:
$$
W _ {A} ^ {\tau_ {i}} = \left(\left(W _ {A}\right) ^ {T} g _ {n} ^ {\tau_ {i}}\right) ^ {T}. \tag {5}
$$
4.3 Task-Specific Self-Attentive Document Embedding
Here, we perform the self-attentive feature extraction using the aforementioned top- $k$  task-specific attention aspects for a task. We then apply it into the generation of document embeddings for text
Table 1: Results of 5-way 1-shot classification.
20 Newsgroup	HuffPost	Reuters	RCV1	Amazon
MAML (Finn et al., 2017)	43.58%	35.27%	43.82%	36.69%	48.12%
PROTO (Snell et al., 2017)	34.78%	28.62%	46.78%	34.40%	36.42%
LEO (Rusu et al., 2018)	36.42%	28.75%	35.37%	32.26%	39.54%
INDUCTION (Geng et al., 2019)	43.04%	35.62%	42.73%	36.24%	36.33%
DS (Bao et al., 2019)	41.79%	25.52%	52.32%	44.35%	46.32%
Frog-GNN (Xu and Xiang, 2021)	-	54.1%	-	-	71.5%
LEA	BERTBASE	53.47%	48.43%	71.64%	51.96%
RoBERTABASE	45.97%	42.16%	63.2%	45.16%
fastText	54.07%	46.15%	69.01%	42.83%
Note: The highest performance in each dataset is highlighted in Bold.
Table 2: Results of 5-way 5-shot classification.
20 Newsgroup	HuffPost	Reuters	RCV1	Amazon
MAML (Finn et al., 2017)	52.73%	44.22%	56.96%	40.47%	63.71%
PROTO (Snell et al., 2017)	55.07%	45.56%	51.22%	44.05%	49.54%
LEO (Rusu et al., 2018)	52.17%	42.25%	54.07%	47.42%	52.47%
INDUCTION (Geng et al., 2019)	53.11%	44.22%	48.00%	45.76%	40.96%
DS (Bao et al., 2019)	52.5%	37.01%	80.80%	68.52%	70.43%
Frog-GNN (Xu and Xiang, 2021)	-	69.6%	-	-	83.6%
LEA	BERTBASE	65.88%	71.6%	83.07%	73.81%	82.69%
RoBERTaBASE	59.20%	68.35%	85.38%	69.08%	85.12%
fastText	60.18%	65.75%	89.01%	71.13%	83.51%
Note: The highest performance in each dataset is highlighted in Bold.
classification. For a text input, we utilize the corresponding embedding vectors for the individual tokens, which are denoted as  $H_{j}^{\tau_{i}} = [h_{j,1}^{\tau_{i}},\dots ,h_{j,L}^{\tau_{i}}]$  for the  $j$ th text example of the task  $\tau_{i}$ . This is formulated as follows:
$$
\mathcal {E} _ {j} ^ {\tau_ {i}} = W _ {A} ^ {\tau_ {i}} H _ {j} ^ {\tau_ {i}}, \tag {6}
$$
where  $\mathcal{E}_j^{\tau_i} \in \mathbb{R}^{k \times L}$  is the self-attentive document embedding of the  $j$ th input of the task  $\tau_i$ , and  $H_j^{\tau_i} \in \mathbb{R}^{u \times L}$  is a set of token embedding vectors for the  $j$ th instance with  $L$  tokens in the task  $\tau_i$ .
For the text classification, we sum  $\mathcal{E}_j^{\tau_i}$  columnwise and then feed it into a fully connected neural network (denoted as  $FC_{\theta_{\tau_i}'}$ ) with the parameters  $\theta_{\tau_i}'$ , which are optimized in the task-adaptation step to make the final predictions.
4.4 Meta-Training Objectives
As noted in Algorithm 1, LEA alternates the following two update steps: (1) task adaptation (or inner-update) and (2) meta-optimization (or outer-update). The former proceeds as follows:
$$
\theta_ {\tau_ {i}} ^ {\prime} \leftarrow \theta_ {\tau_ {i}} ^ {\prime} - \alpha \bigtriangledown_ {\theta_ {\tau_ {i}} ^ {\prime}} \mathcal {L} _ {\tau_ {i}} ^ {t r} \left(f _ {\theta_ {\tau_ {i}} ^ {\prime}}, \mathcal {E} _ {\tau_ {i}} ^ {t r}\right), \tag {7}
$$
where  $\theta_{\tau_i}^{\prime}$  indicates the task model parameters, and  $\mathcal{L}_{\tau_i}^{tr}$  is the classification loss by relying on  $\mathcal{E}_{\tau_i}^{tr}$  derived from  $\mathcal{D}_{\tau_i}^{tr}$ .
During the meta-optimization step, the groups of parameters  $\{W_A,W_g,W_n,\theta_m,\theta_e,\theta_r,\theta_a\}$  are trained in the outer loop with  $\mathcal{D}_{\tau_i}^{val}$ . This is formulated as follows:
$$
\phi \leftarrow \phi - \beta \nabla_ {\phi} \sum_ {\tau_ {i}} \mathcal {L} _ {\tau_ {i}} ^ {v a l} \left(f _ {\theta_ {\tau_ {i}} ^ {\prime}}, \mathcal {E} _ {\tau_ {i}} ^ {v a l},\right)) + \lambda \cdot \Omega \tag {8}
$$
Figure 2: The 5-way 5-shot prediction accuracy depending on the number of top- $k$  attention aspects.
where,  $\Omega$ , as a regularization, includes the term that encourages all attention aspects to have equal importance (Shazeer et al., 2017), and  $\lambda$  is its associated coefficient as usual.
We evaluated LEA on five text datasets — 20 Newsgroup(Lang, 1995), Huffpost headline(Misra and Grover, 2021), Reuters-21578(Lewis., 1997), RCV1(Lewis et al., 2004), and Amazon product reviews (He and McAuley, 2016) — and compared it with current state-of-art methods. We conducted two different experiments: 5-way 1-shot and 5-way 5-shot all over datasets. The details of the datasets are introduced in Appendix A.1.
In this experiment, we evaluate and compare LEA with six state-of-art methods as follows: Here, MAML (Finn et al., 2017) denotes the representative optimization-based meta-learning algorithm, PROTO (Snell et al., 2017) indicates the prototype network, LEO (Rusu et al., 2018) denotes the meta-learning algorithm using latent embedding optimization, INDUCTION indicates the induction network (Geng et al., 2019), DS (Bao et al., 2019) denotes few-shot text classification algorithm using the underlying word distributions, and Frog-GNN (Xu and Xiang, 2021) denotes the multi-perspective aggregation based graph neural network.
We performed all experiments on a frozen BERT<sub>BASE</sub> (Devlin et al., 2019) as a represent
(b)
Figure 3: t-SNE plot of task-specific embedding space after task adaptation. (a) Embedding space of seven top-level domains. (b) Same as (a) but highlighted by the four classes in 'recreation' domain. (c) Same as (a) but highlighted by the four classes in 'science' domain.
tative PTM for LEA and all baselines. As in LEA, DS is given the [CLS] embedding and the token embeddings from BERT's last layer, whereas the other algorithms used the [CLS] embedding of BERT. For the comparison with Frog-GNN, we referred to the reported results from (Xu and Xiang, 2021). We additionally applied LEA on RoBERTa<sub>BASE</sub> (Liu et al., 2019) and fastText (Bojanowski et al., 2017) to verify the applicability of LEA. All performance scores are reported as the average for three repetitions.
As shown in Table 1 and 2, LEA exhibits the competitive performance in both the 5-way 1-shot and 5-way 5-shot, compared to the state-of-the-arts for all the datasets. Namely, the results demonstrate that LEA quickly recognizes how to attend for new tasks using the established meta-attention aspects and provides a robust performance in few-shot text classification problems.
5.3 Hyperparameter Study: Effect of the Number of Top- $k$  Attention Aspects
We also investigate the impact of the number (i.e.,  $k$ ) of task-specific attention aspects. This specific study was conducted on the same frozen BERT $_{BASE}$  as the underlying PTM with the 5-way 5-shot experiment for the all datasets. We fixed the size of the meta attention aspects dictionary to 150 and measured the performances by gradually scaling the  $k$  up to 1, 10, 20, 30, 50, 75, 150. As shown in Figure 2, all the datasets exhibit their best performance when setting the top- $k$  attention aspects to 20. This empirical result indicates that each task derives its optimal document embedding
by referring only to the most relevant subset rather than exploiting all meta-level attention aspects.
5.4 Task-Specific Document Embedding Visualization
In addition, we plots the task-specific document embeddings and observe the relationships among classes on 20 Newsgroups dataset. To qualitatively characterize the task-specific document embedding space, we split 20 Newsgroup into seven top-level domains, that is, 'atheism', 'computer', 'for-sale', 'recreation', 'science', 'religion', and 'talk' and projected them via t-SNE as shown in Figure 3a. Figure 3b shows the relationships between the 'recreation' domain composed of four classes and the rest on the space. Figure 3c shows the relationships between the four classes of the 'science' domain and the others on the space. These plots demonstrate that LEA produces a structured task-specific embedding space after our task-adaptation step.
We hypothesized that a type of task-specific self-attentive mechanism might improve few-shot learning performance, especially when it is prohibitive to fine-tune a large-sized PTM. We have attempted to design a novel embedding transfer method for deriving a meta-level attention aspects dictionary to enable a new task to simply borrow the most relevant attention aspects from the dictionary. As a result, we proposed a novel meta-learning framework for the learning-to-attend and showed that LEA is an effective method that facilitates the utilization of large-sized PTMs in few-shot learning.
Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2019. Few-shot text classification with distributional signatures. In International Conference on Learning Representations.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR.
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3904-3913.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th international conference on world wide web, pages 507-517.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pages 331-339. Elsevier.