SlowGuess/NAACL_2022_OCR huggingface dataset:
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2019. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations.
David D. Lewis. 1997. Reuters-21578, distribution 1.0.
David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Rishabh Misra and Jigyasa Grover. 2021. *Sculpting Data for ML: The first act of Machine Learning*.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. 2018. Meta-learning with latent embedding optimization. In International Conference on Learning Representations.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. 2019. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 403-412.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208.
Shiyao Xu and Yang Xiang. 2021. Frog-gnn: Multiperspective aggregation based graph neural network for few-shot text classification. Expert Systems with Applications, 176:114795.
Table 3: Architecture details
Module Name	Architecture	Shape of (input, output)	The number of Params
Encoder (Eq.3, fθe)	linear	(768, 200)	153.6K
Relation Network (Eq.3, fθr)	2-layer MLP with ReLU	First layer: (2 × 200, 2 X200)
ReLU
Second layer: (2 × 200, 150)
ReLU	700K
Gating Network (Eq.4, fθr)	linear	(150, 150)	22.5K
Meta Attention Aspects (Eq.5, WA)	matrix	(150, 768)	115.2K
Task-Specific Attention Aspects (Eq.6, Εjτi)	linear	(20, 768)	15.36K
Task Classifier	1-layer MLP with ReLU	First layer: (768, 300)
ReLU
output layer: (300, 5)	231.9K
We introduce the datasets and the split (i.e., train/val/test) which had been maintained in our experiments.
20 Newsgroups is a collection of discourses in newsgroup posts for 20 topics (Lang, 1995).
Huffpost Headlines is a collection of news headlines published in the Huffington Post from 2012 to 2018 (Misra and Grover, 2021). It is composed of 41 topics.
Reuters-21578 is composed of documents that appeared on the Reuters newswire in 1987 (Lewis., 1997). In addition, we use the ApteMod version and discard documents with more than one label to avoid ambiguity, and thus 31 classes remain.
RCV-1 is a set of newswire stories published by Reuters journalists from 1996 to 1997 (Lewis et al., 2004) and comprises 71 topic classes.
Amazon data is a real-world dataset collected from Amazon.com as a set of customer reviews from 24 types of product categories(He and McAuley, 2016). Our goal is to match reviews to their own corresponding product categories.
To train and evaluate the models, we divided each of the aforementioned datasets into a meta-training set  $(S^{tr})$ , meta-validation set  $(S^{val})$ , and meta-test set  $(S^{test})$  as disjoint sets of classes within the experimental setting. In this work, we used the same split of classes as in (Bao et al., 2019) for the Huffpost headline(Misra and Grover, 2021), Reuters-21578(Lewis., 1997), and RCV-1(Lewis et al., 2004) datasets. Hence, the Huffpost headline is divided into 20, 5, and 16 disjoint classes for meta-training, validation, and test sets.
In terms of Reuters-21678, 15, 5, and 11 disjoint classes are used for meta-training/validation/test sets and 37, 10, and 24 disjoint classes for RCV-1. In Amazon product data, we split the data using rules in (Bao et al., 2019), and its training and
Table 4: Data Splitting
Dataset	# of tr. cls.	# of val. cls.	# of testCLS.
20 Newsgroup	10	5	5
HuffPost	20	5	16
Reuters	15	5	11
RCV-1	37	10	24
Amazon	10	5	9
Table 5: Hyperparameters for training process
Hyperparameters
meta-training set	# of tasks	8
# of queries	15
meta-validation set	# of tasks	15
# of queries	15
meta-test set	# of tasks	15
# of queries	15
α (learning rates in Eq.7)	1
β (learning rates in Eq.8)	0.001
λ (regularization weight in Eq.8)	0.0001
number of adaptation steps	40
validation sets are used for meta-training set. As a result, Amazon product data is divided into 15, and 9 disjoint classes for meta-training and test sets and meta-validation set is not used in Amazon product data. For the 20 Newsgroup dataset, we randomly selected 20 topic classes, and the metatraining set, meta-validation set, and meta-test set contained 10, 5, and 5 disjoint classes, respectively. We summarize the above information in Table 4.
We share the breakdown of LEA's implementation. In the encoding process of our experiments, the 768-dimensional [CLS] vector, which is of the same size of the output of the pre-trained BERT-base-uncased, is linearly transformed through  $f_{\theta_e}$  into a 300-dimensional vector. The relation network,  $f_{\theta_r}$  is composed of two-layers neural network with ReLU activation and input size is two times of encoder outputs and the size of output is the number of meta-attention-aspects, i.e., 150.
Smith Corona terminates stock sale pact. Smith Corona Corp said on Tuesday it has terminated its agreement for MaraFund Ltd to acquire 15 percent of the new common stock in the reorganized Smith Corona. Smith Corona claims that MaraFund did not meet its contractual obligations, but the termination of the deal does not alter the company's plan of reorganization it filed with Bankruptcy Court.	Smith Corona terminates stock sale pact. Smith Corona Corp said on Tuesday it has terminated its agreement for MaraFund Ltd to acquire 15 percent of the new common stock in the reorganized Smith Corona. Smith Corona claims that MaraFund did not meet its contractual obligations, but the termination of the deal does not alter the company's plan of reorganization it filed with Bankruptcy Court.	Smith Corona terminates stock sale pact. Smith Corona Corp said on Tuesday it has terminated its agreement for MaraFund Limited to acquire 15 percent of the new common stock in the reorganized Smith Corona. Smith Corona claims that MaraFund did not meet its contractual obligations, but the termination of the deal does not alter the company's plan of reorganization it filed with Bankruptcy Court.
Kun Young applies for receivership, South Korea's Kun Young	Kun Young applies for receivership, South Korea's Kun Young	Kun Young applies for receivership, South Korea's Kun Young
Construction Corp applied for court receivership to be protected from its creditors, a court official said on Tuesday.	Construction Corp applied for court receivership to be protected from its creditors, a court official said on Tuesday.	Construction Corp applied for court receivership to be protected from its creditors, a court official said on Tuesday.
Turkish markets closed Friday due to national day. Turkish markets closed Friday due to national day. Turkish markets, government offices and banks will be closed on Friday, the national Victory Day.	Turkish markets closed Friday due to national day. Turkish markets closed Friday due to national day. Turkish markets, government offices and banks will be closed on Friday, the national Victory Day.	Turkish markets closed Friday due to national day. Turkish markets closed Friday due to national day. Turkish markets, government offices and banks will be closed on Friday, the national Victory Day.
US Mint gains authority for platinum, gold coins. US Mint gains authority for platinum, gold coins. The U.S. Mint has authority, as part of the fiscal 1997 budget which took effect Tuesday, to issue a platinum coin and a second type of gold coin.	US Mint gains authority for platinum, gold coins. US Mint gains authority for platinum, gold coins. The U.S. Mint has authority, as part of the fiscal 1997 budget which took effect Tuesday, to issue a platinum coin and a second type of gold coin.	US Mint gains authority for platinum, gold coins. US Mint gains authority for platinum, gold coins. The U.S. Mint has authority, as part of the fiscal 1997 budget whichtook effect Tuesday, to issue a platinum coin and a second type of gold coin.
Figure 4: Visualization of attention weights generated by our model. The texts in (a) and (b) are different samples under a topic Corporate and Industrial. (c) and (d) are related to Markets and Economics.
The gating network,  $W_{g}$  is linear transformation and its size is the number of meta-attention-aspects. For each task classifier, that is,  $f_{\theta_{\tau_i}'}$ , it is designed as single-layer fully connected neural network. We set the size to 150 for the meta-attention-aspects dictionary, and importantly fixed the number of top- $k$  attention aspects to 20. Table 3 summarizes the above model parameters.
A.3 Training Details and Hyperparameter Tuning
In our work, we train all experiments on a single NVIDIA A100 32G GPU. During the meta-training process, we sampled four tasks with 15 queries from  $S^{tr}$ , and it leads to performing task adaptation four times per each meta-optimization update and early stop when the validation loss fails to improve for 20 steps. In validation and test process, we sampled 30 tasks with 15 queries from  $S^{val}$  and  $S^{test}$ , and only performed task adaptation using K-shots. After that, the performance of the adapted task model is obtained using queries. We used the Adam optimizer with learning rates of 0.1 and 0.001 in the inner and outer updates, that is,  $\alpha$  and  $\beta$  in, respectively. In addition, the coefficient  $\lambda$  of the regularization term was set as 0.0001. We summarize the hyperparameters in Table 5.
A.4 Case Study: Visualization of Attention Weights on Text
Herein, we visualize the heatmaps in some cases to investigate how to assign attention weights to text. Figure 4b demonstrates a termination of stock sale
pact, and Figure 4a shows a company growth in terms of consumer products. These were extracted under the Corporate and Industrial topic in the RCV-1 dataset and some seminal words such as "agreement", "contractual" and "receivership" are highlighted to appear in the topic. Figure 4c shows that the Turkish market was closed related to the Market topic, and its relevant words such as "Turkish," "markets," and "closed" are highly attended as expected. Figure 4d talks about the authority of platinum and gold coins under the Economics topic, and the words "US," "Mint," "authority," "gold," "platinum," and "coin" are hence highlighted. As shown in these cases, LEA properly captures important words under a certain topic and assigns attention weights to a given text.