Comparing Prompt-Based and Standard Fine-Tuning for Urdu Text Classification
Authors: Faizad Ullah
Ubaid Azam
Ali Faheem
Faisal Kamiran
Asim Karim
Venue: EMNLP 2023 Findings (2023)
Keywords: Prompt-based Fine-tuning
Urdu Text Classification
Pre-trained Language Models

Abstract: Recent advancements in natural language processing have demonstrated the efficacy of pre-trained language models for various downstream tasks through prompt-based fine-tuning. In contrast to standard fine-tuning, which relies solely on labeled examples, prompt-based fine-tuning combines a few labeled examples (few shot) with guidance through prompts tailored for the specific language and task. For low-resource languages, where labeled examples are limited, prompt-based fine-tuning appears to be a promising alternative. In this paper, we compare prompt-based and standard fine-tuning for the popular task of text classification in Urdu and Roman Urdu languages. We conduct experiments using five datasets, covering different domains, and pre-trained multilingual transformers. The results reveal that significant improvement of up to 13% in accuracy is achieved by prompt-based fine-tuning over standard fine-tuning approaches. This suggests the potential of prompt-based fine-tuning as a valuable approach for low-resource languages with limited labeled data.

Comparing Prompt-Based and Standard Fine- Fine-Tuning for Urdu Text
Classification
Faizad Ullah', Ubaid Azam', Ali Faheem', Faisal Kamiran', and Asim Karim'
'Computer Science, Lahore University of Management Sciences (LUMS), Lahore, Pakistan Computer Science, Information Technology University (ITU), Lahore, Pakistan [faizad. ullah, 19030040, ali. faheem) @ llums.edu.pk faisal. ran@ii tu. edu. pk akarimēl .welllmsseeeeee
Abstract
emerged promising alternative for improving classification accuracy low-resource language contexts (An, 2023; Lee et al., 2022; Jin et al., 2022; Schucher et al., 2022; Wang et al., 2022). Prompt-based fine-tuning combines a small set of annotated examples (few shot) with carefully de- signed language-specift prompts tailored to the task. These prompts explicitly guide the models, providing crucial context and information for pre- cise predictions.
Recent advancements in natural language pro- cessing have demonstrated the efficacy of pre- trained language models for various down- stream tasks through prompt-based fine-tuning. In contrast to standard fine-tuning, which relies solely on labeled examples, prompt-hased fine- tuning combines a few labeled examples (few shot) with guidance through prompts tailored for the specific language and task. For low- resource languages, where labeled examples are limited, prompt-based fine-tuning appears be promising alternative. In this paper, we compare prompt-based and standard fine- tuning for the popular task of text classification in Urdu and Roman Urdu languages. We con- duct experiments using five datasets, covering different domains, and pre-trained multilingual transformers. The results reveal that signifi- cant improvement of up to 13% in accuracy is achieved by prompt-base fine-tuning over standard fine-tuning approaches. This suggests the potential of prompt-based finettnnig as valuable approach for low-resource languages with limited labeled data.
This paper presents an empirical evaluation of prompt-hased fine-tuning with traditional stan- dard fine-tuning for text classification in Urdu and Roman Urdu. The objective to determine whether prompt-based fine-tuning surpasses the performance of standard fine-tuning of multilin- gual PLMs in few shot setting. To do this, we conduct experiments five diverse datasets span- ning different domains and encompassing various classification tasks. These datasets are carefully selected to represent the challenges and nuances of Urdu and Roman Urdu text classification. Ad- ditionally, we utilize three pre-trained multilingual transformers (1) BERT-Multilingual (Devlin et al., 2019), (2) DistilBERT (Sanh et al., 2019). and (3) XLM-RoBERTa (Conneau et al., 2020). These transformers have been widely used and proven effective in various NLP tasks across multiple lan- guages. By incorporating numerous transformers, we aim to evaluate the robustness and generaliz- ability of coompt-based tie-ttunng across different architectures and language models.
1 Introduction
Recent advancements in natural language process- ing (NLP) have highlighted the efficacy of pre- trained language models (PLMs) in various down- stream tasks, including text classification. PLMs. such as (Conneau al., 2020; Devlin et al., 2019) and (Sanh et al., 2019), have revolutionized NLP by pre-training on extensive textual data to acquire language understanding and common knowledge. However, optimizing the performance of mod- els for various languages, especially low-resource languages like Urdu and Roman Urdu, presents challenges that need to be addressed. Standard fine-tuning usually requires large amounts of task- specific labeled examples adapt the parameters PLMS for robust performance. More recently, prompt-based fine-tuning (Gao et al., 2021) has
Our findings reveal a significant improvement of up to 13% in classification accuracy achieved by prompt-based fine-tuning over traditional ap- proaches. This improvement highlights the effec- tiveness of prompt-based methods in capturing the complex linguistic characteristies and nuances in- herent in Urdu texts and demonstrates its potential as promising alternative for low-resource lan- guages where limited labeled data is available. To

---

the best of our knowledge, there are no published works on prompt engineering or prompt-based fine- tuning for Urdu text classification. Our work com- paring prompt and standard fine-tuning in Urdu can seed further research in this direction.
tuning models on limited amounts of labeled data can introduce instability in execution and result in substantial performance variations depending on the choice of data splits (Zhang et al., 2021; Dodge et al., 2020). To generate robust results, we adopt a careful and comprchensive approach, as outlined below.
2 Literature Review
We aim to fine-tune pre-trained language model L (standard and prompt-based) on task D with label space y for Urdu and Roman Urdu text classification. Our goal is to develop effec- live learning strategies that generalize well to an unseen test set (affn grrat) ~ Diest In the few- shot setting, we have limited training examples per class. Let K denote the number of training exam- ples per class, and |Y| denote the total number of classes in the task. Thus, the few-shot training set Dirain consists of Ki K x |Y] examples, where K'ot
In the past few years, social media platforms have experienced an enormous surge in users. With the rise in digital media usage, there is an increasing demand for automated text classification in Urdu. Recently, various transfer learning and data augmentation approaches have been investigated for text classification, such as those described by (Banerjee et al., 2019; Azam et al., 2022;
González-Carvajal and Garrido-Merchán, 2020; Alam et al., 2023). These methods use pre-trained language models and fine-tune them smaller datasets to enhance their performance specific tasks. However, recent technique in natural lan- guage processing called prompt engineering has recently gained attention from researchers (Liu et al., 2023; Gao et al., 2021). Designing prompts to guide language models improves model predic- tions by effectively utilizing contextual information. This technique achieves better results with minimal data, including short learning zero-shot learning. Various types of prompts are used for text classifi- cation, namely Human Designed Prompts (Brown et al., 2020), Schema Prompts (Zhong et al., 2022), and Null Prompts (Logan IV et al., 2022).
Dtrain = { ( Yuran) )= We utilize a develop- ment set Daev to select the optimal model and tune hyper-parametern The size of Dev set is equal to the few-shot training set, i.e., |Devvl Dirain
Now, given the language model L, our process begins by converting the input Zin into token quence ž, which is then mapped to sequence of hidden vectors h2 RR by the language model L. For example, in a binary sentiment classification task, we can construct a prompt using the input x = Yeh jaga bohat pyari hai. The prompt formu- lation would be
proompt = [CLS] x Yeh [MASK] hai. [SEP]
While prompt engineering research has primar- ily focused on the English language, there has been recent work exploring its effectiveness in other lan- guages. For example, (Song et al., 2022) conducted research to determine if prompt engineering could improve text classification in Chinese. Their results showed that the use of prompts yielded positive re- sults. Similarly, (Seo et al., 2022) applied prompt engineering techniques to Korean and found that it improved performance for various text classifi- cation tasks, including topic and semantic classi- fication, even with few-shot learning. However, no known work has been done study prompt learning for the Urdu language.
In the literature, various templates are utilized for prompt-based classification tasks (Gao al., 2021). However, we find that "Yeh [MASK] hai." tem- plate is performing better for Urdu and Roman Urdu. We do not mention the results for other tem- plates due to lack of space in the paper.
The language model L is then responsible for determining whether it is more suitable to fill in the [MASK] position with "khubsurat" (beautiful) or "fazool" (useless) as depicted in Figure 1. This prompt-based methodology enables the model to autonomously complete prompts and make senti- ment/Classification predictions, enabling more effi- cient and accurate classification.
3 Experimental Design
For standard fine-tuning, we use the token se- quence
In this section, we describe the datasets and the experimental setup employed to compare the per- formance of coompt-bbsee and standard fine-tuning approaches for text classification in Urdu. Fine-
f fine-tune JCLS]Ych jaga bohat pyari hai
To construct the training and development sets for each dataset, we select labeled examples

---

khubsurat (label: positive) fazool
MLM
(label: negative)
head
CLS
label: positive
head
label: negative
MLM
(label: positive) (label: negative)
head
Label space Y
Label mapping M(Y)
[CLS] Yeh jaga bohat pyari hai, Yeh [MASK] hai.
[SEP)
[CLS]
Yeh jaga bohat pyari hai. [SEP]
Input
Template
[SEP] ملمي [CLS]
[SEP]
[MASK]
f [CLS]
Template
(a) Standard Fine-tuning
(b) Prompt-based Fine-tuning
Figure I: An illustration of (a) standard fine-tuning and (b) prompt-based fine-tuning for text classification in Urdu and Roman Urdu. The underlined text represents the task specifcc template, designed explicitly for Urdu and Roman Urdu, while the bold words highlight label words. The (Urdu/Roman Urdu text and their English translation) is given follows: (#6dge/-<=; "This place is very beautiful."", (f [MASK] It [MASK]), (Ych [MASK] hai It is [MASK]) (cott; beautiful), (U. useless)
from Diasi for each class, resulting in total of Y x K labeled examples, where Y represents the total number of classes (labels) in the dataset. In our experiments, we consider K 1,8, 16 and 32. We also refer this number splits. The re- maining examples from Dirain are reserved for the test set (with no labels). To ensure fair evalua- tion, we perform multiple rounds of testing. We select samples randomly from the unlabeled test set not used in each round's s training Dimin and de- velopment Dáv sets. This process is repeated five times, allowing for comprehensive evaluation of the models' performance across different test sets.
We used the Hugging Face library for stan- dard fine-tuning of models, while for prompt-based fine-tuning, we employed the OpenPrompt li- brary. OpenPrompt is an open-source framework designed explicitly for prompt learning, providing a comprehensive set of tools and resources for this approach. Our lie-ttnnig process (standard and prompt-based) utilizes a learning rate of 2c 5. The optimization method is AdamW (an Adam optimizer variant), and the loss function is Cross- Entropy Loss. The number of epochs for each train- ing 10. Our study utilizes three pre-trained multi- lingual transformers, namely (1) xlm-roberta-base. (2) bert-base base--alttllinne mutee-mmmliliilll-oooo ccase,, and (3) distilbert- ceee-ulttlinguallcass which are publicly avail- able on HuggingFace library.
We evaluate classification performance using curacy and macro F1-score. We report the mean and standard deviation of each measure over the five runs and for different splits. This approach allows us to draw robust conclusions regarding the effectiveness of prompt-based and standard fine- tuning approaches for text classification.
3.1 Datasets
Our study utilizes five distinct datasets that span different domains, including emotion and offen- sive language detection. Specifically, the Urdu Nastalique Emotions Dataset (UNED) (Bashir et al., 2023) consists of 1119 instances for emo- tion detection, featuring labels such as Neutral, Happy, Sad, Anger, Fear, and Love, The URDU OFFENSIVE DATASET (UOD) (Akhter et al., 2020) and Roman Urdu Dataset (RUD) (Akhter et al., 2020) play pivotal roles in offensive lan- guage detection, with instance counts of 2106 and 147116, respectively. These datasets employ Of-
Prompt-based fine-tuning involves using language-specific prompts tailored to Urdu and Roman Urdu to provide additional context to the models during training. This helps capture the intricacies of the languages and improve text classification performance, No prompts used in standard fine-tuning, and the models rely solely on labeled examples. Comparing these approaches allows us to assess the impact of prompts in leveraging speccfcc knowledge.
https://huggingffcccco http://hunnn.gghhhho//opppeeeeee

---

fensive and Non-Offensive labels. The Roman Urdu Emotion Detection Dataset (RUED) (Arshad et al., 2019), consisting of 2961 instances, facili- tates emotion detection with labels including Anger, Sad, Happy, and Neutral. The Roman Urdu Hate- Speech and Offensive Language Detection (RUH- SOLD) (Rizwan et al., 2020) dataset comprising 10012 instances revolves around hate speech and offensive language detection, utilizing labels such Hate Speech Offensive Language and Non- Offensive. RUHSOLD dataset originally used six labels Abusive/Offensivee Sexism, Religious Hate, Profane, and Normal. However, to make the tasks easier, we converted this multiclass problem to a binary class problem. The datasets selected for our experiments, however, represent both binary and multiclass text classification problems.
variation of accuracy with split size for UOD, RUD, RUED, RUHSOLD, and UNED datasets, respec- tively, under prompt-based and standard fine-tuning of the respective pessfrerfornnn models. In gen- eral, accuracy improves with an increase in split size. However, for prompt-based fine-tuned mod- els, the increase in accuracy was more significant compared to standard fine-tuned models. It is also worth noting that standard fine-tuning almost al- ways lags behind prompt-base based fine-tuning for all models and datasets for up to 32 splits, confirming that for the limited number labeled examples, prompt-based finettuning is preferable.
102
80
50
4 Results and Discussion
Saandrd Fine ane
Table shows the mean classification accuracy for prompt-based based and standard fine-tuned pre-trained language models on different datasets. Mean accu- racy is reported for splits of 4, 8, 16, and 32. In the demonstrated the
Prosphhassed
16
32
K-5pll
Figure 2: Accuracy comparison for Standard Prompt-based Fine-tuning for UOD, where K-Split de- notes the number instances per class.
UNED dataset, XL.MRoBBRR highest accuracy (after prompt-based fine-tuning) of 44.8% 16 splits, outperforming all other mod- cls. Similarly, for RUD dataset, fine-tuned XLM- RoBERTa remained the top performer with an ac- curacy of 84.8%. For the RUHSOLD dataset, fine- tuned RoBERa consistently dominated in performance across all splits, achieving the highest accuracy of 63.8% on 32 splits.
Standard Fine Premphhsssdd redise
On the RUED dataset, prompt-based fine-tuned XLM-RoBERTS outperformed other models with the highest accuracy of 35.6% for splits 32, How- ever, for splits 8 and 16, prompt fine-tuned Distil- BERT and HERT-MMlttilnnuaal models performed better, respectively. In the case of the UOD dataset, the results were more varied. Prompt-based fine- tuned XLM-RoBERT outperformed other models for splits 4 and 16. However, for splits 8 and 32, prompt fine-tuned BERT-Multilnnuaal yielded bet- ter results compared to the other models.
Figure 3: Accuracy comparison for Standard Prompt-based Fine- munittning for RUD dataset, where K- Split denotes the number of instances per class.
28
14
32
30
In our experiments, we consistently observed that prompt fine-tuned models outperformed stan- dard fine-tuned models achieving up to 13% abso- lute improvement and 5.11% average improvement in accuracy across all tasks. Additionally, it is noteworthy that as the data splits increased, the models' accuracy also increased, as evident from Figures 2, 3, 4, 5, and 6. These figures show the
and Pranuiisst Fimetas
20
E-Splt
Figure 4: Accuracy comparison for Standard vs. Prompt-based tuning for RUED dataset, where K K-Split denotes the number of instances per class.

---

UNED
UOD
RUD
RL HSOLD
RUED
16
52
4
8
16
32
BERT-M
21,4
26.8
$2.0
60.2
65.8
71.2
79.8
63.2
66.4
67.2
72.2
53.0
55.0
57.6
60.0
27.2
28.4
28.0
31.8
Standard
DisHIBERT
18.8
20.8
23.4
58.6
57.4
68.4
80.0
55.0
64.2
63.2
71.0
49.2
55.6
56.5
60.4
27.4
29.8
31.6
25.4
fine suning
XLM-R
19.0
27.4
74.6
66.6
75.2
54.2
56.2
57.8
60.2
19.4
23.0
26.8
26.6
BERT-M
23.2
27.4
36.2
60.4
68.2
77.4
89.2
63.8
70.6
70.6
77.6
54.2
53.8
58.4
61.8
26.8
29.0
32.2
30,4
Prompt based fine tuning
DistIBERT
24.4
25.6
32.8
57.6
64.2
72.6
81.4
65.2
70.8
68.4
75.6
52.8
54.8
38.4
60.8
27.6
30.0
31.8
33.4
X-M-R
27.4
31.8
84.8
59.0
64.0
63.8
29.0
28.4
31.0
38.6
Table 1: Mean accuracy comparison of standard and prompt-based fine-tuning for Urdu and Roman Urdu datasets using K = 4, 8, 16 and 32, where K represents the number of samples per class. The standard deviation of these experiments given in Appendix A.
ments using several multilingual pre-trained lan- guage models on different classification datasets. Regardless of the training set size or the specific classification task, prompt-based fine-tuning con- sistently outperforms standard fine-tuning, high- lighting its robustness and generalizability. The clear prediction superiority of prompt-based ap- proaches, coupled with its generally lower com- putational cost, makes it an attractive alternative to traditional fine-tuming methods for low oowrresource languages. The insights gained from this study can inspire future research and encourage the adoption of prompt-based techniques in other low-resource languages.
65
50
danded
20
Figure 5: Accuracy comparison for Standard= vs. Prompt-based Fine-tuning for RUHSOLD dataset, where K K-Split denotes the number of instances per class.
Limitations
25
is essential to note that the presented experiments and results are relevant to Urdu and Roman Urdu. Consequently, the generalizability of the findings to other languages remains to be determined. The effectiveness and performance of the approach in diverse linguistic contexts may vary significantly. Thus, care should be exercised when inferring the results to languages beyond Urdu and Roman Urdu. Another significant factor to consider is the reliance of prompt-based techniques on domain expertise. The success of prompt-based finc-tuning heavily hinges upon formulating appropriate prompts that adequately capture the desired semantic informa- tion. Effective prompts require deep understand- ing of the language, context, and the specific task. However, this process can introduce subjectivity and potential bias, as prompt design involves mak- ing subjective decisions and assumptions. These subjective elements may influence the performance of the approach and limit its objectivity in specific scenarios.
Saadard
Prompstesedd Fineaum
16
Figure 6: Accuracy comparison for Standard vs. Prompt-based Fine-tuning for UNED dataset, where K-Split denotes the number of instances per class.
We present additional experimental results the Appendices. Appendix A gives the standard devi- ations corresponding to the mean accuracy values reported in Table 1. Appendix B presents the mean F1-score of all the experiments on different models and datasets. Appendix C studies the impact of fine-tuning by comparing the performance of zero- shot (no fine-tuning) and 4-shot prompt-basee and standard fine-tuned models.
5 Conclusion
In this paper, we compare prompt-based fine-tuning with standard fine-tuning for Urdu and Roman Urdu text classification when restricted to dozens of training examples only. We perform experi-
The current findings provide valuable insights into the utility of prompt-based fine-tuning for low- resource languages and text classification. Future

---

studies should investigate the performance of this approach across a broader range of languages and tasks, considering different linguistic characteris- ties and data availability. It is also essential to acknowledge that the evaluation of the proposed upproach was focused solely on text classification. The applicabiliity and performance of prompt-based fine-tuning in other NLP tasks, such as named en- tity recognition, sentiment analysis, or machine translation, still need be explored. Therefore, caution should be exercised when attempting to generalize the findings to other NLP domains, as the effectiveness of prompt-base based fine-tuning may vary depending on the task and its linguistic prop- erties.
fensive Language for Urdu and Roman Urdu, LEEE Access, 0991213-12222
Shafiq Alam, Muhammad Sohaib Ayub, Sakshi Arora, and Muhammad Asad Khan. 2023, An Investigation of The Imputation Techniques for Missing Values in Ordinal Data Enhancing Clustering and Classifica- tion Analysis Validity. Decision Analytics Journal, page 100341.
Bo An. 2023. Prompt-based for Low-Resouree Tibetan Text Classification. ACM Transactions on Asian and Low-Resource Language Information Process- ing, 22(8).
Muhammad Umair Arshad, Muhammad Farrukh Bashir, Adil Majeed, Waseem Shahzad, and Mirza Omer Beg. 2019. Corpus for Emotion Detection on Roman Urdu. In 22nd IEEE International Multitopic Conference (INMIC), pages 1-6.
The limitations outlined in this study highlight the need for research and improvement in prompt- based fine-tuning. While the approach shows promise for low-resource languages and text clas- sification, its generalizability, subjectivity in a prompt design, limited task scope, and broader data scarcity challenges necessitate further investiga- tion and refinement. Addressing these limitations will enhance the applicability and effectiveness of prompt-based fine-tuning in diverse language set- tings and NLP tasks.
Ubaid Azam, Hammad Rizwan, and Asim Karim. 2022. Exploring data augmentation strategies for hate speech detection in Roman Urdu. In Pro- ceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4533-4531, Marseille, France. European Language Resources Association.
Siddhartha Banerjee, Cem Akkaya, Francisco Perez- Sorrosal, and Kostas Tsioutsiouliklis. 2019. Hierar- chical transfer learning for multi-label text classifica- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 6295 6300, Florence, Italy, Association for Compu- tational Linguistics.
Ethics Statement
Muhammad Farrukh Bashir, Abdul Rehman Javed, Muhammad Umair Arshad, Thippa Reddy Gadekallu, Waseem Shahzad, and Mirza Omer Beg. 2023. Context-aware Emotion Detection from Low- resource Urdu Language Using Deep Neural Net- work. ACM Transactions Asian and Low- Resource Language Information Processing, 22(5):1- 30.
We have carefully considered our study's s ethical implications and taken the measures into account. Data privacy and confidentiality were strictly main- tained throughout the rescarch process. We con- sciously tried to mitigate biases and subjectivity in prompt design and analysis. Our approach is designed to assist human decision-making rather than replace it, emphasizing the importance of hu- man involvement. We have given due consideration to the ethical aspects of based fine-tuning, including fairness and privacy issues. Our work contributes advancing NLP knowledge and bene- fits various stakeholders. Overall, this study aligns with established ethical standards and promotes the responsible application of prompt-basee fine- tuming in low-resource languages and text classifi- cation.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in Neural Information Process- ing Systems, 33:1877-1901.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
References
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of
Muhammad Pervez Akhter, Zheng Jiangbin, Irfan Raza Naqvi, Mohammed Abdelmajeed, and Muham- mad Tariq Sadiq. 2020. Automatic Detection of Of-

---

the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, A Distilled Ver- sion Of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:
Nathan Schucher, Siva Reddy, and Harm de Vries. 2022. The power of prompt tuning for low-re rew-resuuce seman- tic parsing, In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 148-156, Dublin, Ireland. Association for Computational Linguistics.
Jesse Dodge, Gabriel Ilharco. Roy Schwartz, Ali Farhadi, Hammaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight ini- tializations, data orders, and early stopping. arXiv preprint arXiv:2002.66:3
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 38 3616-383,, Online, Association for Computa- tional Linguistics.
Jachyung Sco, Hyeonseok Moon, Chanhee Lec, Sug- yeong Eo, Chanjun Park, Jihoon Kim, Changwoo Chun, and Heuiseok Lim. 2022. Plain Template In- sertion: Korean- Prreann-rrop---ee Based Engineering for Few- Shot Learners. IEEE Access, 10:107587-107599.
Chengyu Song, Taihua Shao, Kejing Lin, Dengfeng Liu, Siyuan Wang, and Honghui Chen. 2022. Investi- gating Prompi Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models. Applied Sciences, 12(21):11177
Santiago González- Gonzállz-Carvajal and Eduardo C Garrido- Merchán. 2020. Comparing BERT against traditional machine learning text classification. arXiv preprint 13012
Yufei Wang, Can Xu, Qingfeng Sun, Huang Hu. Chongyang Tao, Xiubo Geng, and Daxin Jiang. 2022. PromDA: Prompt-based data augmentation for low- resource NLU tasks. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4242- 4255, Dublin, Ireland. Association for Computational Linguistics.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. 2022. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2763-2775. Dublin, Ireland. Association for Computational Linguistics.
Dong-Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2022. Good examples make a faster learner: Simple demonstration-bon bbasdd learning for low-resource NER. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2687-2700, Dublin, Ireland. Association for Computational Linguistics.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Wein- berger, and Yoav Artzi. 2021. Revisiting Few-sample BERT Fine-tuning. In International Conference on Learning Representations.
Wanjun Zhong, Yifan Gao, Ning Ding, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. 2022. Improving Task Generalization via Unified Schema Prompt. arXiv preprint 2xXiv2208.00322
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre-
A Standard Deviation of Accuracy
Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 1-(5):1335.
Table 2 shows the standard deviation of accuracy over 5 runs for different approaches, models, and datasets. The standard deviation corresponds to the mean accuracy reported in Table 1.
Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Find- ings of the Association for Computational Lingwis- tics: ACL 2022, pages 2824 2233,2285,,1 Dublin, Ireland. Association for Computational I Linguistics.
B F1-score Comparison
Table 3 gives the mean F1-score for prompt-based and standard fine-tuning using different models and different datasets. It is observed that Fl-score results follow the same trend as that for accuracy results given in Table 1. Thus, despite significant class imbalance in some datasets the performance of text classification measured via accuracy and macro Fl-score is fairly consistent.
Hammad Rizwan, Muhammad Haroon Shakeel, and Asim Karim, 2020. Hate-speech and offensive lan- guage detection Roman Urdu. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2512- 2522, Online. Association for Computational Lin- guistics.

---

UNED
UOD
RUD
RUHSOLD
RUED
16
32
16
32
BERT-M
6.06
3,96
7.0
4.20
2.28
2.38
3.56
3.42
0.89
3.11
2.38
3.46
5.02
2.82
4.60
4.66
5.04
2.78
Standard
DisHIBERT
3.27
3.96
4.27
1,94
4.72
6.42
2.0
4.84
2.48
3.34
2.23
2.04
1.81
1.14
1.51
6.58
5.67
5.17
3.28
fine tuning
XLM-R
4.24
3.78
7.0
8.35
0.0
7.02
3.76
6.40
8.19
4.03
3.34
1.48
4.20
10.9
2.50
5.87
3.76
5.59
Prompi bused lino-tuning
BERT-M
1.72
6.65
DistiBERT
2,70
4.03
3.11
4,15
5.80
4.50
5.72
2.77
1.64
4.15
1.14
4.43
2.16
5.12
2.68
4.33
2.34
5.35
3.43
XLM-R
2.19
3.03
7.01
9.47
6.05
5.72
2.77
5.35
2.77
4.20
2.94
3.76
3.08
4.52
3.70
4.24
9.39
2.0
6.76
Table Standard deviation of classification accuracy across different datasets and pre-trained language models for Urdu and Roman Urdu datasets using K- = 4, 8, 16 and 32.
UOD
RUHSOLD
16
8
16
32
16
32
4
8
16
32
Standard fine tuning
BERT-M
16.8
25.2
27.8
57.4
64.8
712
79.8
62.0
65.6
67.0
72.0
46.6
52.0
56.4
59.5
23.0
25.4
27.0
29.2
DistIBERT
10.8
15.2
174
50.6
50.4
66.8
79.8
45.6
59.4
70.6
42.2
46.6
54.0
56.2
21.4
25.2
22.6
XLM-R
10.0
23.8
27.8
43.4
33
73.2
80.8
45.2
52.6
62.8
75.0
46.0
40.8
56.8
56.4
12.4
15.2
22.4
23,4
Prompt based
BERT M
20.4
23.4
32.8
59.2
67.6
76.0
89.2
63.4
69.8
70.0
77.4
52.6
51.4
54.4
61.8
22.8
25.0
28.0
26.8
DistiBERT
20.4
22.8
29
55.2
63.4
72.4
81.4
64.2
70.6
67.8
75.6
51.8
53.2
57.6
60.6
24.6
27.2
27.8
30.0
line tuuiinnnn
XLM-R R
25.6
28.8
45.0
63.6
66.0
79.6
87.8
66.4
76.8
79.8
84.8
55.8
61.8
63.4
26.8
24.4
30.0
30.4
Table 3: Mean Fl-score comparison of standard and prompt-based fine-tuning for Urdu and Roman Urdu datasets using K = 4, 8, and 32, where K represents the number of samples per class.
UNED
RUHSOLD
Zero-Shot
4-Shot
Zero-Shot
4-Shot
Zero-Shot
4-Shot
Zero-Shot
4-Shot
Zero-Shot
4-Shot
BERT-M
9.0
21.4
50.0
60.2
50.0
63.2
54.0
53.0
21.0
27.2
Standard
DistiBBRT
18.0
18.8
51.0
58.6
51.0
55.0
54.0
49.2
16.0
27.4
XI.M-R
14.0
19.0
49.0
54.4
50.0
55.0
$4.0
54.2
22.0
19.4
BERT-M
18.0
23.2
51.0
60.4
52.0
63.8
46.0
54.2
42.0
26.8
Prompt based
DistilBETT
19.0
24.4
51.0
57.6
52.0
65.2
48.0
52.8
45.0
27.6
XLM-R
16.0
27.4
51.0
65.8
50.0
67.8
48.0
57.2
46.0
29.0
Table 4:
Accuracy results for zero-shot and 4-Shot for various models on different datasets.
UNED
UOD
RUD
RUHSOLD
RUED
Zero-Shot
4-Shot
Zero-Shot
4-Shot
Zero-Shot
4-Shot
Zero-Shot
4-Shot
Zero-Shot
4-Shot
BERT-M
5.0
16.8
34.0
57.4
33.0
62.0
35.0
46.6
17.0
23.0
Standard
DistilBERT
5.0
10.8
34.0
50.6
40.0
45.6
40.0
42.2
9.0
22.0
XLM-R
4.0
10.0
33.0
43.4
33.0
45.2
35.0
46.0
12.0
12.4
BERT-M
6.0
20.4
34.0
59.2
46.0
63.4
44.0
52.6
19.0
22.8
Prompt based
DistiBBET
10.0
20.4
34.0
55.2
39.0
64.2
40.0
51.8
18.0
24.6
XLM-R
10.0
25.6
35.0
63.6
33.0
66.4
38.0
55.8
16.0
26.8
Table 5:
Fl-score comparison for zero-shot and 4-Shot for various models different datasets.
C
Accuracy and F1-Score Comparison for Zero-Shot and 4-Shot
behind those for 4-shot prompt-based fine-tuned predictions on all the datasets. other words, a per- formance boost is obtained by employing prompt- based fine-tuning even with 4 training examples. This underscores our argument that prompt-based fine-tuning holds significant potential in resource- constrained scenarios and is particularly relevant in the case of the Urdu language.
Tables and 5 present accuracy and Fl-score, re- spectively, for zero-shot and fine-tuned with four examples all five datasets. Results are shown for standard and prompt-based techniques using pre-trained language models. It is clear from the results that zero-shot prompt-basen predictions lag


Reviews:
[{"review_id": "iIj66vu8dZ", "reviewer": "Reviewer_XVEi", "summary": "", "questions": "1.  Can you provide the F1 result of Table 1?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reasons to reject  \nPoint 1: The evaluation metric, accuracy, in Table 1 is not convincing. For example, the test set of Roman Urdu Hate-Speech and Offensive Language Detection (RUHSOLD) data [1] is unbalanced. In this case, compared with the F1-score, the accuracy may ignore the minor class.\n\nResponse 1: We appreciate your comment regarding the class imbalance in RUHSOLD dataset, and the superiority of the F1-score in such scenarios. We have the F1-score results as well but decided to include the accuracy results in the short paper for two reasons: (1) the F1-score results show similar trends to accuracy results, and (2) most of the datasets we used have an even distribution of classes. Considering the short paper’s length limitations, we will provide the F1-score results in the Appendix. For your convenience, we are also presenting the F1-score table below. As you can see, a similar performance trend is observed in F1-score for all the datasets. \n\n|                          |            | UNED     |          |        | UOD      |          |          |          | RUD      |          |          |          | RUHSOLD  |        |          |          | RUED     |        |        |          |\n|--------------------------|------------|----------|----------|--------|----------|----------|----------|----------|----------|----------|----------|----------|----------|--------|----------|----------|----------|--------|--------|----------|\n|                          |            | 4        | 8        | 16     | 4        | 8        | 16       | 32       | 4        | 8        | 16       | 32       | 4        | 8      | 16       | 32       | 4        | 8      | 16     | 32       |\n| Standard fine-tuning     | BERT-M     | 16.8     | 25.2     | 27.8   | 57.4     | 64.8     | 71.2     | 79.8     | 62       | 65.6     | 67       | 72       | 46.6     | 52     | 56.4     | 59.6     | 23       | 25.4   | 27     | 29.2     |\n|                          | DistilBERT | 10.8     | 15.2     | 17.4   | 50.6     | 50.4     | 66.8     | 79.8     | 45.6     | 63       | 59.4     | 70.6     | 42.2     | 46.6   | 54       | 56.2     | 22       | 21.4   | 25.2   | 22.6     |\n|                          | XLM-R      | 10       | 23.8     | 27.8   | 43.4     | 33       | 73.2     | 80.8     | 45.2     | 52.6     | 62.8     | 75       | 46       | 40.8   | 56.8     | 56.4     | 12.4     | 15.2   | 22.4   | 23.4     |\n| Prompt based fine-tuning | BERT-M     | 20.4     | 23.4     | 32.8   | 59.2     | **67.6** | 76       | **89.2** | 63.4     | 69.8     | 70       | 77.4     | 52.6     | 51.4   | 54.4     | 61.8     | 22.8     | **25** | 28     | 26.8     |\n|                          | DistilBERT | 20.4     | 22.8     | 29.4   | 55.2     | 63.4     | 72.4     | 81.4     | 64.2     | 70.6     | 67.8     | 75.6     | 51.8     | 53.2   | 57.6     | 60.6     | 24.6     | 27.2   | 27.8   | 30       |\n|                          | XLM-R      | **25.6** | **28.8** | **45** | **63.6** | 66       | **79.6** | 87.8     | **66.4** | **76.8** | **79.8** | **84.8** | **55.8** | **57** | **61.8** | **63.4** | **26.8** | 24.4   | **30** | **30.4** |\n\n\nPoint 2: Previous works have proposed the prompt-based approach, and the paper does not provide much insight in terms of methodology. [1] https://aclanthology.org/2020.emnlp-main.197.pdf \n\nResponse 2: Our primary focus in the short paper is the comparison of two competing paradigms for utilizing large language models (LLMs). We were motivated by the fact that low-resource languages do not have large amounts of training data for different scenarios, and standard fine-tuning of LLMs can be computationally and financially more expensive than that for prompt-based fine-tuning of the same LLMs. As such, our main contribution is the demonstration that prompt-based fine-tuning can in fact outperform standard fine-tuning for different text classification tasks in a popular, yet under-resourced, language Urdu.  \n\nWe have utilized the RUHSOLD dataset introduced in Rizwan et al. (2020). They focus on hate speech and offensive language classification in Roman Urdu only, while we consider several text classification applications in Urdu and Roman Urdu. Our motivation and contribution, as highlighted above, is also different from theirs’.   \n\nQuestions For the Authors: \n\nPoint 3: Can you provide the F1 result of Table 1? \n\nResponse 3: Please see Response 1 above.  \n\nPoint 4: Typos Grammar Style and Presentation Improvements: \n\nResponse 4: Your thorough review is much appreciated. We will correct these typos and grammatical mistakes in the final version of the paper. \n\nPoint 5: Can you provide a parallel English translation for the Urdu text in Figure 1? \n\nResponse 5: We will include the parallel English translation for the Urdu text featured in Figure 1 in its caption. The translation is also provided below for your reference. \n\nUrdu text: \"Yeh jaga bohat pyari hai. yeh [MASK] hai.\" \n\nEnglish: \"This place is good. It is [MASK].\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper may be interesting to the Urdu language research community, considering that the Urdu language has less research and available resources than the English language.", "reasons_to_reject": "1. The evaluation metric, accuracy, in Table 1 is not convincing. For example, the test set of Roman Urdu Hate-Speech and Offensive Language Detection (RUHSOLD) data [1] is unbalanced. In this case, compared with the F1 score, the accuracy may ignore the minor class. \n\n2. Previous works have proposed the prompt-based approach, and the paper does not provide much insight in terms of methodology. \n\n[1] https://aclanthology.org/2020.emnlp-main.197.pdf", "questions_to_authors": "1.  Can you provide the F1 result of Table 1?", "paper_topic_and_main_contributions": "The paper compares the prompt-based and standard fine-tuning for Urdu text classification. More specifically, three pre-trained LMs have been evaluated on five Urdu text classification datasets. The experimental result shows that XLM-R model together with prompt-based tuning obtains the best classification performance.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Can you provide a parallel English translation for the Urdu text in Figure 1?"}, "extra_text": {}}, {"review_id": "HOmOPbVZ2K", "reviewer": "Reviewer_18m6", "summary": "", "questions": "A: Please provide the corresponding English to the example sentences and templates (LL 148).", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Rebuttal Points:\n\nPoint 1: Just reporting the results with known methods, without proposing new or better methods for prompting. \n\nResponse 1: The primary focus of this short paper is to examine and contrast two distinct learning paradigms for Urdu text classification: prompt-based and Standard Fine-Tuning. The use of prompts is a relatively new technique that can be valuable for situations where training data is limited. Furthermore, there is little exploration into how these techniques function within the context of the Urdu language. To the best of our knowledge, there are no published works on prompt engineering or prompt-based fine-tuning for Urdu text classification. Our work comparing prompt and standard fine-tuning in Urdu can seed further research in this direction.  \n\nPlease see Response 2 to Reviewer 1 as well. Thanks!\n\nPoint 2: The conclusion that prompt-based tuning is better is not surprising at all, considering the standard fine-tuning was trained on very small data. As done in Chandio 2022 (see missing references), normally the traditional fine-tuning requires at least thousands of data. \n\nResponse 2: Our objective is to determine which modern-day learning paradigm performs better when only a limited amount of training data is available. This is a constraint common in low-resource language processing. Obviously, when large amounts of training data are available standard fine-tuning should be preferred. \n\nAs highlighted by Reviewer 3 (Point 7), it would be useful to determine the number of training examples after which standard fine-tuning surpasses prompt-based fine-tuning. In our experiments, where training data is constrained to a few dozen examples, this did not occur.  \n\nPoint 3: The observation on the graphs (Fig 2 to 6), that more sample helps higher accuracy, is trivial. \n\nResponse 3: We present these figures to emphasize the significance of the observation that increasing sample size improves text classification accuracy in the context of the low-resource language Urdu. In other words, this observation reinforces the fundamental understanding that larger sample sizes generally contribute to enhanced accuracy, but when we have scarce data, Prompt engineering can be better than standard techniques. \n\nPoint 4: Five data sets are tested, but the overall differences between Persian and Roman alphabets are not discussed. As the characteristics of Urdu, the differences in writing should be qualitatively and quantitatively compared. \n\nResponse 4: Thank you for your valuable feedback. The study of different scripts of Urdu and their implication on prompt-based text classification warrants separate treatment and is beyond the scope of the current short paper. Nonetheless, we will highlight this issue in the final version and suggest it as a future direction. \n\nQuestions For the Authors: \n\nPoint 5: Please provide the corresponding English to the example sentences and templates (LL 148). \n\nResponse 5: Thank you for your suggestion. Please see Response 5 to Reviewer 1.  \n\nPoint 6: Missing References: Chandio et al. 2022, Sentiment Analysis of Roman Urdu on E-Commerce Reviews Using Machine Learning “https://www.techscience.com/CMES/v131n3/47397 This paper handles Urdu sentiment analysis.” The task is three-way classification, so it is harder than binary classification. \n\nResponse 6: Thank you for your suggestion; we will add this reference to the final version of the paper.  \nTo clarify, our evaluation datasets include the task of sentiment classification and represent both binary and multiclass text classification tasks.  \n\nPoint 7: Typos Grammar Style and Presentation Improvements: \"as shown in 1\" in LL200 should be \"Table 1 \\& 2\", I guess. \n\nResponse 7: Thank you. You are right, we will correct this mistake in the final version of the paper. \n\nPoint 8: \"Ye\" should be \"Yeh\" in Figure 1 (b)? \n\nResponse 8: Thank you for the thorough review. This was a typo, and we will correct this in the final version of the paper. \n\nPoint 9: LL231: \"35.6\\% for splits 4 and 32\" is not accurate - 35.6\\% is just for 32. \n\nResponse 9: Thank you. You are right, we will correct this typo in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Handled Urdu language that has not been well studied.\n* Provided exhaustive experimental results for five data sets.", "reasons_to_reject": "* Just reporting the results with known methods, without proposing new or better methods for prompting.\n* The conclusion that prompt based tuning is better is not surprising at all, considering the standard fine-tuning was trained on very small data.  As done in Chandio 2022 (see missing references), normally the traditional fine-tuning requires at least thousands of data.\n* The observation on the graphs (Fig 2 to 6), the more sample helps higher accuracy, is trivial.\n* Five data sets are tested, but the overall differences between Persian and Roman alphabets are not discussed.  As the characteristics of Urdu, the difference of writing should be qualitatively and quantitatively compared.", "questions_to_authors": "A: Please provide the corresponding English to the example sentences and templates (LL 148).", "paper_topic_and_main_contributions": "This paper compares two approaches to Urdu binary sentiment classification: traditional supervised fine-tuning and prompt-based few-short learning.  The authors reported the results on five datasets written in Persian and Roman alphabets and three pre-trained language models.\nExperimental results show that the prompt-based approach consistently outperformed the supervised method.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* \"as shown in 1\" in LL200 should be \"Table 1 & 2\", I guess.\n* \"Ye\" should be \"Yeh\" in Figure 1 (b)?\n* LL231: \"35.6% for splits 4 and 32\" is not accurate - 35.6% is just for 32.\n"}, "extra_text": {"Missing_References": "Chandio et al. 2022, \"Sentiment Analysis of Roman Urdu on E-Commerce Reviews Using Machine Learning\" https://www.techscience.com/CMES/v131n3/47397\nThis paper handles Urdu sentiment analysis.  The task is three-way classification so it is harder than binary classification."}}, {"review_id": "OwLgqom9Kl", "reviewer": "Reviewer_pxrY", "summary": "", "questions": "A. Is there any special reason or motivation behind the choice of Urdu?\n\nB. In Figure 1, what does the label mapping look like in each dataset used in the experiment? The prompt-based fine-tuning method requires defining the label mapping that maps target labels to words in the model’s vocabulary for each dataset. This mapping benefits subsequent research in this direction if available and should be considered as a contribution of the paper since the mapping used in the experiment has already been shown to work well in the paper.\n\nC. How are the prompt-based few-shot results compared to those in a zero-shot setting? The prompt-based method can be evaluated in a zero-shot setting without any fine-tuning simply by taking a word with the highest probability among words in the label mapping at the [MASK] position. The comparison with the zero-shot results is needed to verify the effectiveness and necessity of fine-tuning in the prompt-based method.\n\nD. When does the standard fine-tuning method outperform the prompt-based method as the number of samples per class, namely K, increases? In general, if a sufficient amount of labeled data is available, the standard fine-tuning method works quite well and could outperform the prompt-based method. Given that the datasets used in the paper have enough samples to test if the paper does not adopt the few-shot setting, it is recommended to analyze when the standard fine-tuning method outperforms the prompt-based method.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Point 1: While it is important to address low-resource languages, it is unclear for the choice of Urdu language among many other low-resource languages such as Swahili and Nepali. \n\nResponse 1: Please see Response 4 below. \n\n\nPoint 2: The main argument stated in the paper (in Lines 021-024 and 079-082) is somewhat overclaimed. The paper states that prompt-based fine-tuning allows the models to capture Urdu’s unique linguistic characteristics and nuances, while it seems to lack the experiment and analysis to support this claim. It is necessary to qualitatively and/or quantitatively show what type of Urdu’s unique linguistic characteristics and nuances are captured by the model in addition to reporting the overall accuracy of the datasets. If it is difficult to fit additional experiments in the body of the short paper, it would be preferable to rewrite or tone down the argument. \n\nResponse 2: Thank you for your feedback. We will revise the statement to accurately represent our findings and their implications for prompt-based methods for Urdu and Roman Urdu. The statement would be updated as: \"The improved classification accuracy achieved by prompt-based fine-tuning over standard fine-tuning suggests its potential as a valuable approach for low-resource languages with limited labelled data.\" \n\n \n\nPoint 3: A description of the experiment, especially for the fine-tuning procedure, is missing, which affects reproducibility. For reproducibility, at least the following information should be provided: the source of the pre-trained language model and the hyper-parameters such as batch size, learning rate, number of epochs, optimization method, etc. As for the source of the pre-trained language model, there are multiple options, even for a single model. For example, BERT-Multilingual has two variants provided by Hugging Face (bert-base-multilingual-uncased and bert-base-multilingual-cased) while XLM-R offers two different sizes (xlm-roberta-base with 279M parameters and xlm-roberta-large with 561M parameters). Different variants of models could produce different results. \n\nResponse 3: We appreciate your valuable feedback regarding the need for clarity in our experimental procedures. In response, we provide the missing information: our fine-tuning process utilizes a learning rate of 2e-5. The optimization method is AdamW (an Adam optimizer variant), and the loss function is Cross-Entropy Loss. The number of epochs for each training is 10. Our study utilizes three pre-trained multilingual transformers, namely (1) xlm-roberta-base, (2) bert-base-multilingual-cased, and (3) distilbert-base-multilingual-cased, which are publicly available on HuggingFace library.  \n\nWe provided the link to the HugginFace library in the manuscript; however, your suggestion to add additional details will improve the paper’s reproducibility.  \n\n \n\nQuestions For the Authors: \n\nPoint 4: Is there any special reason or motivation behind the choice of Urdu? \n\nResponse 4: We appreciate the reviewer's question about the choice of Urdu over other languages like Swahili and Nepali. The decision to focus on Urdu is primarily motivated by our deep familiarity and native-level proficiency in Urdu. This familiarity enables us to make well-informed decisions and conduct accurate analyses in Urdu, including the important issue of prompt design, which would not be the case with other languages like Swahili and Nepali. Additionally, we recognize Urdu's distinct right-to-left writing system and regional diversity. This distinctive feature brings specific challenges in terms of processing and analysis. While we acknowledge the importance of other low-resource languages, our familiarity with Urdu's linguistic intricacies and distinctive writing system makes it a logical choice for our research emphasis. \n\n \n\nPoint 5: In Figure 1, what does the label mapping look like in each dataset used in the experiment? The prompt-based fine-tuning method requires defining the label mapping that maps target labels to words in the model’s vocabulary for each dataset. This mapping benefits subsequent research in this direction if available and should be considered as a contribution to the paper since the mapping used in the experiment has already been shown to work well in the paper. \n\nResponse 5: We appreciate your valuable suggestion. In Figure 1, the label mapping for each dataset used in the experiment is not explicitly depicted due to the numerous datasets spanning different domains, such as sentiment analysis and hate speech, that are employed in this work. However, as you pointed out, we did indeed establish label mappings for the prompt-based fine-tuning method, and we will add the list of labels in the Appendix of the paper. These mappings were created by associating target labels with words from the model's vocabulary. \n\n \n\nPoint 6: How are the prompt-based few-shot results compared to those in a zero-shot setting? The prompt-based method can be evaluated in a zero-shot setting without any fine-tuning simply by taking a word with the highest probability among words in the label mapping at the [MASK] position. Comparison with the zero-shot results is needed to verify the effectiveness and necessity of fine-tuning in the prompt-based method. \n\nResponse 6: Thank you for the insightful comment. Our primary focus in the short paper is to evaluate fine-tuning with limited amounts of data. Obviously, zero-shot predictions will fare poorly as compared to few-shot fine-tuned predictions. Nonetheless, we will include zero-shot results in the final version of the paper.     \n\n \n\nPoint 7: When does the standard fine-tuning method outperform the prompt-based method as the number of samples per class, namely K, increases? In general, if a sufficient amount of labelled data is available, the standard fine-tuning method works quite well and could outperform the prompt-based method. Given that the datasets used in the paper have enough samples to test if the paper does not adopt the few-shot setting, it is recommended to analyze when the standard fine-tuning method outperforms the prompt-based method. \n\nResponse 7: We appreciate your thoughtful feedback. We refrain from evaluating with large numbers of examples because we want to mimic the scenario where only a limited number of examples for a given task are available. This is quite common for low-resource languages. Furthermore, it is worth noting that prompt-based fine-tuning is often more economical than standard fine-tuning on many platforms.  \n\nPlease also see Response 2 to Reviewer 2.  \n\n \n\nPoint 8: Typos Grammar Style and Presentation Improvements: In Line 160, the word “and” between 16 and 32 should not be in italics, In Line 200, in 1 -> in Table 1, In Lines 213 and 215, The Table 1 -> Table 1 \n\nResponse 8: Thank you for pointing out these typos. We will fix them in the final version. \n\n \n\nPoint 9: The brief description of the datasets used for the experiment (e.g., the number of examples in train/dev/test sets, the total number of target classes, and the characteristics of the texts, such as the source or domain of the texts) should be added to Appendix for the readers of the paper to better understand the tasks used for the evaluation. \n\nResponse 9: Thank you for highlighting this aspect. We acknowledge the importance of providing datasets and experiment details. Our study utilizes five distinct datasets that span different domains, including emotion and offensive language detection. Specifically, the UNED dataset consists of 1119 instances for emotion detection, featuring labels such as Neutral, Happy, Sad, Anger, Fear, and Love. The UOD and RUD datasets play pivotal roles in offensive language detection, with instance counts of 2106 and 147116, respectively. These datasets employ Offensive and Non-Offensive labels. The RUED dataset, consisting of 2961 instances, facilitates emotion detection with labels including Anger, Sad, Happy, and Neutral. The RUHSOLD dataset, comprising 10012 instances, revolves around hate speech and offensive language detection, utilizing labels such as Hate Speech / Offensive Language and Non-Offensive. RUHSOLD dataset originally used six labels Abusive/Offensive, Sexism, Religious Hate, Profane, and Normal. However, to make the tasks easier, we converted this multiclass problem to a binary class problem. \n\nTo generate robust evaluations, we employ a 5-fold cross-validation approach. This involved the creation of five samples for each dataset, using different splits (4, 8, 16, and 32) for training and development, whereas the remaining for testing set. The importance of non-repetition of training samples within training splits is diligently maintained.  \n\nAs recommended, we will include these details in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper studies the feasibility of a prompt-based fine-tuning method in Urdu, which is considered a low-resource language.\n2. The paper experiments with three pre-trained language models and ensures that the outperformance of the prompt-based method over the standard fine-tuning method is consistent within these LMs.\n3. The paper discusses its limitations from various aspects.", "reasons_to_reject": "1. While it is important to address low-resource languages, it is unclear for the choice of Urdu language among many other low-resource languages such as Swahili and Nepali.\n2. The main argument stated in the paper (in Lines 021-024 and 079-082) is somewhat overclaimed.\n    - The paper states that prompt-based fine-tuning allows the models to capture Urdu’s unique linguistic characteristics and nuances, while it seems to lack the experiment and analysis to support this claim. It is necessary to qualitatively and/or quantitatively show what type of Urdu’s unique linguistic characteristics and nuances are captured by the model in addition to reporting the overall accuracy on the datasets. If it is difficult to fit additional experiments in the body of the short paper, it should be preferable to rewrite or tone down the argument.\n3. A description of the experiment, especially for the fine-tuning procedure, is missing, which affects reproducibility.\n    - For reproducibility, at least the following information should be provided: the source of the pre-trained language model and the hyper-parameters such as batch size, learning rate, number of epochs, optimization method, etc. As for the source of the pre-trained language model, there are multiple options, even for a single model. For example, BERT-Multilingual has two variants provided by Hugging Face (`bert-base-multilingual-uncased` and `bert-base-multilingual-cased`) while XLM-R offers two different sizes (`xlm-roberta-base` with 279M parameters and `xlm-roberta-large` with 561M parameters). Different variants of models could produce different results.", "questions_to_authors": "A. Is there any special reason or motivation behind the choice of Urdu?\n\nB. In Figure 1, what does the label mapping look like in each dataset used in the experiment? The prompt-based fine-tuning method requires defining the label mapping that maps target labels to words in the model’s vocabulary for each dataset. This mapping benefits subsequent research in this direction if available and should be considered as a contribution of the paper since the mapping used in the experiment has already been shown to work well in the paper.\n\nC. How are the prompt-based few-shot results compared to those in a zero-shot setting? The prompt-based method can be evaluated in a zero-shot setting without any fine-tuning simply by taking a word with the highest probability among words in the label mapping at the [MASK] position. The comparison with the zero-shot results is needed to verify the effectiveness and necessity of fine-tuning in the prompt-based method.\n\nD. When does the standard fine-tuning method outperform the prompt-based method as the number of samples per class, namely K, increases? In general, if a sufficient amount of labeled data is available, the standard fine-tuning method works quite well and could outperform the prompt-based method. Given that the datasets used in the paper have enough samples to test if the paper does not adopt the few-shot setting, it is recommended to analyze when the standard fine-tuning method outperforms the prompt-based method.", "paper_topic_and_main_contributions": "This paper tackles Urdu text classification and makes a comparison between a prompt-based and standard fine-tuning method on Urdu text classification in a few-shot setting (K=4, 8, 16, and 32) using five datasets. These datasets contain both Urdu and Roman Urdu characters and consist of emotion detection and offensive language detection tasks. The paper verifies the superiority of the prompt-based method over the standard fine-tuning method with three pre-trained multilingual language models (i.e., BERT-Multilingual, DistilBERT, XLM-RoBERTa).\n\nUpdated: I have read the author rebuttal and adjusted the scores given its type.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Line 160, the word “and” between 16 and 32 should not be in italics\n2. In Line 200, in 1 -> in Table 1\n3. In Lines 213 and 215, The Table 1 -> Table 1\n4. The brief description of the datasets used for the experiment (e.g., the number of examples in train/dev/test sets, the total number of target classes, and the characteristics of the texts such as the source or domain of the texts) should be added to Appendix for the readers of the paper to better understand the tasks used for the evaluation"}}]

Author rebuttal:

Meta-review: This paper compares prompt-based fine-tuning with standard fine-tuning for 5 text classification tasks from Urdu and Roman Urdu languages. These datasets cover emotion detection and offensive language detection tasks. The paper uses 3 pretrained LLMs and shows that prompt-based fine-tuning outperforms standard fine-tuning when using a few examples (4, 8, 16, 32 per class). 

As reviewers mentioned, this work presents a valuable contribution focusing on Urdu as a low-resource language. During the discussion period, the authors highlighted that the presented comparison (prompt-based fine-tuning vs. standard fine-tuning) is important when fine-tuning with a very small amount of data which is in line with the paper's focus on Urdu language. In this context, experimental results are sound and consistent across 5 datasets and 3 LLMs.  

Addressing feedback from the reviewers, the authors provided additional experimental results during the discussion period and acknowledged the suggestions from reviewers. Concretely, incorporating the additional results, rephrasing the main claim, adding further details on the dataset and fine-tuning hyperparameters will further improve the paper.
Decision: Accept-Findings