Word-level Prefix/Suffix Sense Detection: A Case Study on Negation Sense with Few-shot Learning
Year: 2023

Abstract:
Morphological analysis is an important research issue in the field of natural language processing. In this study, we propose a context-free morphological analysis task, namely word-level prefix/suffix sense detection, which deals with the ambiguity of sense expressed by prefix/suffix. To research this novel task, we first annotate a corpus with prefixes/suffixes expressing negation (e.g., il, un-, -less) and then propose a novel fewshot learning approach that applies an inputaugmentation prompt to a token-replaced detection pre-training model. Empirical studies demonstrate the effectiveness of the proposed approach to word-level prefix/suffix negation sense detection.1

heading: 1 Introduction
text: Morphological analysis mainly refers to processing a word into a lemma (root) and a well-defined morphological tag (Anglin et al., 1993; Haspelmath and Sims, 2013; Morita et al., 2015; Nicolai and Kondrak, 2017; Deacon et al., 2017; Ganesh et al., 2019). For instance, through morphological analysis, the word “unhappy” will be divided into a lemma “happy” and a negation sense prefix tag “un-”. Morphological analysis has played an important role in natural language processing (NLP) and it has been applied to many downstream tasks such as spelling checking (Aduriz et al., 1993; Oflazer, 1995; Sénéchal and Kearnan, 2007; Levesque et al., 2021) and machine translation (Lee, 2004; Habash, 2007; Toutanova et al., 2008; Belinkov et al., 2017).
One major challenge in morphological analysis is that prefixes/suffixes are sometimes ambiguous. For instance, in English, the prefix “un” often means a meaning “not”, i.e., a negation
∗*Corresponding author 1https://github.com/mengmeng233/Word-level-Prefix-
Suffix-Sense-Detection
sense. However, not all words with the prefix “un” have a negation sense, such as “unanimous” and “unpick”. Besides, the substring “un-” sometimes does not appear as a prefix in some words, such as “universe” and “unique”. In this study, we directly address the above challenge by proposing a novel morphological analysis task, namely word-level prefix/suffix negation sense detection, which aims to detect whether a substring in a word is a prefix/suffix and meanwhile takes a specific pre-defined morphological sense. As a preliminary study, we focus on negative prefixes/suffixes. In many languages, one way to make a negative expression is to add a negative prefix/suffix to a word. For instance, in English, il-, im-, un-, and -less are some popular negative prefixes/suffixes.
One straightforward approach to prefix/suffix negation sense detection is to build a dictionary that covers all words with the prefixes/suffixes expressing such a sense. However, this is unrealistic because there are always many newly-emerging words due to non-standard language usage or incorrect spelling in some informal texts like Twitter. Therefore, we address the task of word-level prefix/suffix negation sense detection in a computational way.
Specifically, to further reduce the annotation cost, we propose a few-shot learning approach by employing the token-replaced detection model as our basic prompt-learning model due to its excellent performance in few-shot learning (Li et al., 2022). Furthermore, we propose a novel prompt, namely input-augmentation prompt, which relies only on the input word. As illustrated in Fig.1(c), for the input word is “unhappy”, the prompt, “ unhappy It is not happy”, is used to predict whether the word “not” is original or replaced so as to determine whether the input word is a negation word or not, where the substring “happy” is generated by removing the potential prefix (i.e., un-) from the input word. The de-
7651
sign of our input-augmentation prompt can avoid one major shortcoming of existing few-shot learning approaches, i.e., the selection of labels (e.g., two labels, “positive” and “negative” in Fig.1a) or the selection of label description words (e.g., “negative positive” in Fig.1b) has a big impact on learner performance (Jiang et al., 2020; Gao et al., 2020; Li et al., 2022). Moreover, our empirical studies also demonstrate that our approach achieves much better performances than the existing few-shot learning approaches.

heading: 2 Related work
text: Morphological analysis aims to learn about the morphological structure of a given word form, and in general, there are four specific tasks: morphological tagging (i.e., assigning some pre-defined morphological tags to a word in a sentence) (Müller et al., 2013; Labeau et al., 2015; Cotterell and Heigold, 2017; Conforti et al., 2018; Malaviya et al., 2019), lemmatization (i.e., converting a word in a sentence into the normalized form) (Plisson et al., 2004; Chrupała, 2006; Jongejan and Dalianis, 2009; Straková et al., 2014; Bergmanis and Goldwater, 2018), morphological segmentation (i.e., judging whether the substring in a word could be segmented as a prefix/suffix) (Ruokolainen et al.,
2013, 2016; Goldsmith et al., 2017; Cotterell et al., 2019), and morphological disambiguation (i.e., assigning a correct morphological segmentation to a word by leveraging the context) (Hakkani-Tür et al., 2002; Yildiz et al., 2016; Cotterell et al., 2018; Wiedemann et al., 2019).
Compared to the above tasks, our work has at least three different aspects. First, our task is a combination of morphological tagging and morphological segmentation. Second, our task is word-level, i.e., the input contains only a single word without context, which leads to the inapplicability of previous approaches based on contextual information. Third, we propose a novel few-shot learning approach to our task. To the best of our knowledge, this is the first attempt of studying fewshot learning in morphological analysis.

heading: 3 Corpus Generation
text: We use six prefixes, i.e., un-, im-, in-, il-, irand dis- as negation prefixes and two suffixes, i.e., -less and -free as negation suffixes to collect words from two resources, i.e., the ninth edition of Oxford Advanced Learner′s Dictionary (AS et al., 2005) and 1.6 million English Tweeter data collected by Go et al. (2015). In summary, we obtain 2,717 and 6,671 words with negation prefixes/suffixes from the Oxford dictionary and
tweeter data, respectively. Then, we randomly select 3,000 words and annotates such words as our corpus. Specifically, we assign two annotators to annotate each word into two categories, i.e., positive and negative. The Kappa consistency check value of the human annotation is 0.87. Moreover, for words with different sense annotations, we assign another annotator to make a final decision. Table 1 shows the data statistics of the corpus.

heading: 4 Methodology
text: Problem statement: The prefix/suffix negation sense detection task can be formulated as follows. Let Dl = {w, y} be labeled data, where w is the input word and y is a label in {positive, negative}. Our approach aims to provide a few-shot learner for such a detection task. Approach overview: As shown in Figure 1(c), a prompt-based learner, which is based on a pretrained token-replaced detection model and an input-augmentation prompt, is built for the prefix/suffix negation sense detection task. The goal of a pre-trained token-replaced detection model (e.g., ELECTRA) is to predict whether a token in the input string is replaced or not.
Approach specification: First, an inputaugmentation prompt xprompt is constructed for an input word w, as follows.
xprompt = w It is not w, (1)
where “It is not” is a template, and w is a substring of the input word w without the prefix/suffix, such as w = “happy” for w = “unhappy”.
Second, prompt x = [w1, w2, ..., wn] is fed into the encoder in the discriminator of the pre-trained token-replaced detection model to obtain an output sequence y = [y1, y2, ..., yn], where wi is the ith word in the prompt, and yi is the prediction label (either original or replaced) for word wi, indicating whether the word is original or replaced.
Finally, we map the label set of the pre-trained token-replaced detection model to the label set of our task, with the following formulas.
P (“negative”|xprompt) = P (y“not” = original) (2) and
P (“positive”|xprompt) = P (y“not” = replaced), (3)
where y“not” denotes the label corresponding to the word “not” in the input-augmentation prompt as shown in formula (1). For instance, suppose that the input word is “unhappy”, we first obtain the input-augmentation prompt “unhappy It is not happy” and then use the pretrained token-replaced model to predict whether the word “not” in the prompt is original or replaced. If the prediction result is original, we conclude that the input word “unhappy” is a negative word.
In the training phase of our few-shot learning setting, only a few prompt samples, together with their labels are used to update the parameters in the discriminator of the pre-trained token-replaced detection model. It is important to note that our approach reuses the pre-trained parameters in the pretrained token-replaced detection model and does not use any other new parameters.

heading: 5 Experiments
text: Data setting: 2,000 samples are randomly selected from the human-annotated corpus. First, 400 samples are selected as test data, including 200 for each class. Then, we follow the evaluation protocol of Li et al. (2022) by running 5 experiments with 5 different training and development splits. In each split, 16 training samples (i.e., 8 samples in each class) and 16 development samples (i.e., 8 samples in each class) are selected in few-shot learning. In fully-supervised learning, 1,400 training samples and 200 development samples are used. Evaluation Metrics: Standard Macro-F1, Accuracy, F1 for negative samples (1-F1), and F1 for positive samples (0-F1) are used to evaluate the performance. Model Settings: We employ ELECTRA as the pre-trained token-replaced detection model. The weight_decay is 2e-3, the maximum length is set to 64, and the remaining hyper-parameters are obtained by searching.
We implement the following approaches for comparison: (1) Finetuning-RoBERTa (Liu et al., 2019): Based on the fine-tuning approach and RoBERTalarge model, the prediction label is obtained by mapping the “[CLS]” token to label space. (2) Finetuning-ELECTRA (Clark et al., 2020): It is similar to finetuning-RoBERTa except that the ELECTRA-large model is used. (3) Prompt-RoBERTa (Gao et al., 2020): It is a discrete prompt learning approach based on RoBERTa-large, as shown in Figure 1(a), where the prompt is “ w it is [mask] ”, and the prediction label is obtained by the filling of “[mask]” (either “negative” or “positive”). (4) Prompt-ELECTRA (Li et al., 2022): It is a discrete prompt learning approach based on ELECTRA-large, as shown in Figure 1(b), where the prompt is “ w is a negative positive word ”. (5) Warp (Hambardzumyan et al., 2021): It is a continuous prompt learning approach, in which the best prompt template is obtained by searching in the (continuous) embedding space. Moreover, the template is learned using adversarial refactoring. (6) DART (Zhang et al., 2021): It is a continuous prompt learning approach, in which the search for the best prompt template is based on backpropagation. (7) P-tuning-v2 (Liu et al., 2021): It is a continuous prompt learning approach, in which the search for the best prompt is based on a prefixed-tuned multi-layer prompt. (8) Fully-supervised Learning: 1,400 training and 200 development samples are used to re-train the ELECTRA-large model.
Table 2 shows the performances of different approaches, from which we can see that : (1)
Our approach significantly outperforms the fullysupervised learning and fine-tuning approaches, which proves the effectiveness of our few-shot learner. (2) Our approach performs much better than other prompt-based learners, e.g., obtaining 8.6% increase on Macro-F1 when compared with Prompt-ELECTRA. The improvement confirms the effectiveness of our input-augmentation prompt. (3) Our approach, using only 16 training and 16 development samples, almost performs equivalent to the fully-supervised learning approach with 1,400 training and 200 development samples.
An error analysis is made for our approach, which shows two main error causes: (1) the input word w or its substring w has multiple meanings, such as “hapless” vs. “hap”, and “disembarkation” vs. “embarkation”. (2) the meaning of w and w is irrelevant, such as “dispossession” vs. “possession”, and “ingot” vs. “got”. This indicates that more efforts are needed for our prefix/suffix negation sense detection.

heading: 6 Conclusion
text: In this study, we propose a novel word-level morphological analysis task, namely prefix/suffix sense detection, and make a case study on negation sense. We provide an annotated corpus for the prefix/suffix negation sense detection, and then propose a novel few-shot learning approach, which uses an input-augmentation prompt and a pretrained token-replaced detection model to effectively make the negation sense detection. Empirical studies show that our approach performs much better than other approaches in the few-shot scenario, such as using only 16 training samples.
Limitations
The limiation of this work is that we only consider one type of prefixes/suffixes, i.e., negative prefixes/suffixes. In our future work, we would like to work on other types of prefix/suffix sense detection tasks, such as prefix/suffix sense detection on occupation. For instance, in English, there are many suffixes such as -or, -er, and -ee, which mean a person with a certain occupation.

heading: Acknowledgement
text: We thank the reviewers for their insightful comments and suggestions. This work was supported by the NSFC grant (No.62076176), and the General Research Fund (GRF) project sponsored by the Research Grants Council Hong Kong (Project No.15611021).

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations.
7 A2. Did you discuss any potential risks of your work? At present, we haven’t found any potential risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3, 5.
3 B1. Did you cite the creators of artifacts you used? 3, 5.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3, 5.
C 3 Did you run computational experiments? 5
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We report the model sizes used in all our work, which run on the same GPU.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In our future work, we would like to work on other types of prefix/suffix sense detection tasks, such as prefix/suffix sense detection on occupation. For instance, in English, there are many suffixes such as -or, -er, and -ee, which mean a person with a certain occupation