SlowGuess/NAACL_2022_OCR huggingface dataset:
pre-trained language model fine-tuning. In International Conference on Learning Representations.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4921-4933, Online. Association for Computational Linguistics.
Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D Yoo. 2019. Edge-labeling graph neural network for few-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11-20.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv, abs/1907.11692.
Pau Rodríguez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. 2020. Embedding propagation: Smoother manifold for few-shot classification. In European Conference on Computer Vision, pages 121-138. Springer.
Victor Garcia Satorras and Joan Bruna Estrach. 2018. Few-shot learning with graph neural networks. In International Conference on Learning Representations.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. arXiv preprint arXiv:2103.11955.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.
Yaqing Wang, Abulikemu Abuduweili, Quanming Yao, and Dejing Dou. 2021a. Property-aware relation networks for few-shot molecular property prediction. In Advances in Neural Information Processing Systems, pages 17441-17454.
Yaqing Wang, Song Wang, Quanming Yao, and Dejing Dou. 2021b. Hierarchical heterogeneous graph representation learning for short text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3091-3101, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3):1-34.
Yi Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6365-6375, Online. Association for Computational Linguistics.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations.
Yanqiao Zhu, Weizhi Xu, Jinghao Zhang, Qiang Liu, Shu Wu, and Liang Wang. 2021. Deep graph structure learning for robust representations: A survey. arXiv preprint arXiv:2103.03036.
A Results on SuperGLUE Variant
In addition to PET (Schick and Schütze, 2021b), we also combine the proposed RGL with other prompt-
based tuning strategies. We take P-tuning $^4$  (Liu et al., 2021) as the representative for joint prompt and PLM tuning, and  $\mathsf{WARP}^5$  (Hambardzumyan et al., 2021) as the representative for prompt tuning with a fixed PLM. WARP is only evaluated on CB and RTE as in the original paper.
In the original papers, both P-tuning and WARP use SuperGLUE (Wang et al., 2019) variant proposed by Schick and Schütze (2021b) to evaluate the few-shot performance and use ALBERT $^6$  (Lan et al., 2020) as the PLM. We adopt their setting for fairness. Schick and Schütze (2021b) provide one training set which consists 32 samples per task and a testing set. Schick and Schütze (2021b) also use unlabeled samples, which are not used in this paper. Following Liu et al. (2021), a development set consisting of 32 samples per task are randomly drawn for model selection. As only one split is provided, we initialize the parameter with five random seeds and report the averaged results over five runs.
BoolQ(acc)	MultiRC(EM)	(F1a)	(WiC(acc)	WSC(acc)
P-tuning	75.2(5.2)	32.1(1.0)	74.9(1.9)	55.3(1.5)	80.8(2.5)
+RGL	77.4(0.8)	33.5(0.2)	75.6(1.2)	57.3(2.9)	81.7(1.0)
relative ↑	+2.2	+1.4	+0.7	+2.0	+0.9
CB(acc)	(F1)	RTE(acc)	COPA(acc)
P-tuning	87.5(3.0)	82.1(6.0)	74.7(1.0)	82.3(2.5)
+RGL	88.1(2.1)	84.2(2.3)	75.5(1.3)	83.7(5.1)
relative ↑	+0.6	+2.1	+0.7	+1.4
WARP	82.2(3.0)	77.5(7.2)	72.8(0.5)
+RGL	84.3(2.1)	80.5(4.7)	73.2(1.0)
relative ↑	+2.1	+3.0	+0.4
Table 2: Test performance obtained on SuperGLUE variant (Schick and Schütze, 2021b).
Table 2 shows the results obtained on SuperGLUE variants. The results show that RGL can consistently boost the performance when it is combined with P-tuning and WARP.
<sup>4</sup>https://github.com/THUDM/P-tuning.
$^{5}$ https://github.com/YerevaNN/warp.
$^{6}$ https://huggingface.co/albert-xxlarge-v2.