Definition Generation for Automatically Induced Semantic Frame (2024)

Abstract: In a semantic frame resource such as FrameNet, the definition sentence of a frame is essential for humans to understand the meaning of the frame intuitively. Recently, several attempts have been made to induce semantic frames from large corpora, but the cost of creating the definition sentences for such frames is significant. In this paper, we address a new task of generating frame definitions from a set of frame-evoking words. Specifically, given a cluster of frame-evoking words and associated exemplars induced as the same semantic frame, we utilize a large language model to generate frame definitions. We demonstrate that incorporating frame element reasoning as chain-ofthought can enhance the inclusion of correct frame elements in the generated definitions.

heading: 1 Introduction
text: Semantic frames are conceptual structures that describe specific types of situations or events, and FrameNet (Baker et al., 1998) is one of the representative semantic frame resources. Each frame in FrameNet comprises a frame name, a definition, a set of core and non-core frame elements (FEs), a set of frame-evoking words (lexical units), and exemplars in which lexical units appear. Table 1 exhibits CUTTING frame in FrameNet as an example.
While FrameNet has expanded to include multiple languages, such as Spanish (Subirats and Sato, 2004) and Japanese (Ohara et al., 2004), it remains absent in many low-resource languages. There may also be cases where frame knowledge of different granularity or specific to a particular domain is required. Accordingly, new frame resources need to be developed in some cases, but the costly annotation process hinders their construction. To address this issue, a number of efforts have been made to automate the process of building frames from large corpora. For instance, the frame induction task aims to automatically group frame-evoking
words (typically verbs), along with their associated exemplars, on the basis of the semantic frame they evoke. The mainstream methods are mainly based on contextualized word embeddings (QasemiZadeh et al., 2019; Yamada et al., 2021b) such as BERT (Devlin et al., 2019). These methods leverage the observation that words evoking the same semantic frame tend to appear in similar contexts, resulting in their embeddings being grouped into the same cluster (Yamada et al., 2021a, 2023).
However, while the frame induction task provides clusters of frames, it lacks interpretability because definitions of these clusters are not provided. To make frame resources intuitive and understandable to humans, it is desirable to analyze each cluster and assign a definition sentence that describes the frame, but manually adding definition sentences is labor-intensive and time-consuming. Therefore, in this study, we address a new task of generating frame definitions from a set of frameevoking words.

heading: 2 Frame Definition Generation
text: Our formulation of the semantic frame definition generation task is as follows: given a set of frameevoking words and associated exemplars clustered as one frame, our objective is to produce a cohesive natural language definition that accurately captures
11112
the essence of the frame they evoke.1 Figure 1 illustrates our frame definition generation task.
We employ a large language model as backbone of the definition generator for its strong performance across various natural language generation (NLG) tasks (Zhang et al., 2023). Specifically, we employ Llama2-70b-chat2, an open-source large language model that has demonstrated strong performance among large language models (Touvron et al., 2023). The task of the definition generator is to map the clusters of frame-evoking words and exemplars to their respective definitions.

heading: 2.1 In-context Learning
text: Recent studies have shown that large language models have strong few-shot performance on a wide range of downstream tasks, known as in-context learning (ICL) (Brown et al., 2020). In standard ICL, the LLM is prompted with a set of inputoutput pairs, termed demonstrations, to output new predictions. In our approach, we utilize frameevoking words and exemplars as input queries, presenting the LLM with query-definition pairs to learn the underlying mapping and generate corresponding definitions. Leveraging ICL enables the integration of frame knowledge into LLMs using a limited number of demonstrations, making our method more practical in low-resource language scenarios. Moreover, ICL does not require parameter adaptation, thereby significantly reducing computational costs compared to other task adaptation methods such as fine-tuning.
1As our primary focus lies in frame definition generation rather than frame induction task, we extract data directly from the ground truth semantic frame dataset and input them into definition generator.
2https://huggingface.co/meta-llama
Frame elements provide contexts for understanding the relationships between various concepts within the frame. By identifying frame elements, people can better interpret and infer information related to the frame. Related study shows that including Chain-of-Thought (CoT) in the prompt can benefit the ICL performance (Wei et al., 2022). Aiming to increase the inclusion of correct frame elements, we added frame element reasoning between query and definition to construct demonstrations. By generating frame elements and frame definition subsequently, LLM is encouraged to improve the incorporation of accurate frame elements within the generated definitions.

heading: 2.2 Prompt for Frame Definition Generation
text: The framework for the prompt of frame definition generation is illustrated in Figure 2, consisting of three main components: task instruction, demonstrations, and new query. The task instruction provides a concise description of the task objective. Demonstrations are key to the prompts and we evaluate two strategies for selecting demonstrations, as detailed below.
Random demonstrations We use this as our main setting, where demonstrations are randomly extracted from the training set consisting of ground truth semantic frames. Note that once extracted, the demonstrations are fixed when mapping different clusters into definitions. Due to the random extraction of data from FrameNet, performance fluctuations occur due to randomness. To address this, we
conduct multiple experiments using diverse sets of demonstrations to provide a comprehensive assessment of their overall performance.
Similar demonstrations It has been shown that demonstrations that are similar with new query can benefit ICL performance in many tasks (Liu et al., 2022). Given the cluster of one frame, we encode all exemplars within the cluster using a sentenceencoder3, then we take average of their exemplar embeddings as the frame embedding. We search for similar demonstrations by measuring the cosine similarity between the target frame embedding and other candidates within the training set, and extract the nearest neighbors.

heading: 3 Experiment

heading: 3.1 Dataset
text: In this work, we extract frame data from the publicly available Berkeley FrameNet 1.7 dataset4. We use frames evoked by verbs, splitting them into an 80 percent training set (513 out of 641 verbevoking frames) and a 20 percent test set with the remaining 128 frames. Note that for random demonstrations setting, we extract only a small subset of the training data rather than utilizing the entirety. While in similar demonstrations and finetuning settings, we utilize the entire training dataset. For frame elements, we mainly focus on core frame elements, without which the frame cannot be instantiated.

heading: 3.2 Evaluation
text: Evaluation of definition generation tasks typically relies on standard NLG metrics such as BLEU, Rouge, or BERTScore. However, assessing frame generation tasks demands diversity, where traditional reference-based metrics often fall short. To address this, our evaluation method, represented as Def-Eval, employs GPT-4 to evaluate the generated frame definitions rated on a 1-5 scale. As shown in Figure 3, the prompt consists of task instruction, evaluation criteria, reference definition and generated definition. To validate the efficacy of our evaluation method, we manually annotate 30 frame definitions and assess their correlation with the aforementioned metrics. Results in Table 2 demonstrate that our evaluation approach exhibits the highest correlation with human judgement.
3https://huggingface.co/distilbert/ distilbert-base-uncased
4https://framenet.icsi.berkeley.edu/
Additionally, we leverage an auxiliary evaluation method called FE-Eval to assess the inclusion of frame elements. This involves extracting and concatenating frame elements, followed by calculating BERTScore between the concatenated elements and the reference.

heading: 3.3 Experimental Settings
text: We utilize Llama2-70b-chat as the backbone of definition generator and set the parameters as follows: temperature=0.1, max_new_tokens=1024, repetition_penalty=1.2, top_p=0.9, and top_k=50. We explore the proposed method in three groups of experiments, represented as follows.
• Impact of Demonstration Components: In this group of experiments we test the impact of demonstration components. Since frameevoking word list and frame definition are indispensable for the prompt, we experiment with the effect of exemplars and frame elements by removing them from prompts. We randomly choose 3 demonstrations (3-shot) from FrameNet and report their Def-Eval and FE-Eval scroes.
• Impact of Demonstration Quantity: We experiment with varying numbers of demonstrations: 0 (zero-shot), 1, 3, 5, and 10. For each number of demonstrations, we experiment with various demonstration sets and report their average and standard deviation values.
• Method Selection Strategy: In this experiment, we examine the strategy for selecting methods for generating definitions, considering the variability in FrameNet data availability across different languages.

heading: 3.4 Results
text: Impact of Demonstration Components As illustrated in Table 3, employing frame element reasoning as Chain-of-Thought yields similar DefEval scores, while significantly enhancing FE-Eval scores. This suggests that this process assists the language model in identifying and incorporating the correct frame elements into the frame definitions. Moreover, we observe that excluding exemplars has minimal effect on performance. We attribute this to the pretraining of LLMs on extensive raw corpora, enabling them to interpret words without exemplar contexts. In subsequent experiments, we adopt the Word list + FEs demonstration setting in the following experiments.
Impact of Demonstration Quantity As Figure 4 shows, it is evident that compared to zero-shot scenarios, adding demonstrations can significantly enhance performance, highlighting the effectiveness of ICL. As the number of demonstrations increases, the average Def-Eval score also improves. However, The score peaks and experiences a slight decline, indicating that too many demonstrations may have a negative effect on the performance. Furthermore, we note a consistent decline in the variance value of Def-Eval scores, suggesting that as the number of demonstrations increases, the definition generator tends to produce more stable definitions.
Mehod Selection Strategy We utilize the five demonstration setting which yielded the best and stable results in the previous experiment. Since the availability of FrameNet data varies across different languages, we also report the ground truth labels required for each strategy (represented as Num). Because fine-tuning and ICL are two popular task-adaptation strategies for language models, we also evaluate fine-tuning’s effectiveness in the context of frame definition generation task. We adopt QRoLA (Dettmers et al., 2024) to finetune LLM.
Table 4 presents the Def-Eval and Num values for various methods. To clarify the Def-Eval scores, we have included generated definition examples of different frames in the Appendix A. It is evident that ICL, with only five ground truth demonstrations, achieves a significant improvement compared to the zero-shot setting. This implies that in constructing FrameNet data for low-resource languages, researchers need only annotate a minimal number of ground truth demonstrations to define all induced frames. Leveraging semantically similar frames as demonstrations can enhance performance, yet necessitates the entire training dataset, rendering it more viable in contexts with abundant FrameNet data. Fine-tuning yields similar results with ICL, but also demands the entire training dataset. Moreover, it is less efficient due
to its higher computational resource requirements for adapting LLM to our frame generation task.

heading: 4 Related Work
text: Definition generation, also named definition modelling, was initially formulated to generate a readable word definition based on word embeddings (Noraset et al., 2017). Initially, its primary objective was to assess word embeddings’ effectiveness, later evolving to generating word definitions in context. Mickus et al.; Ni and Wang treat the definition generation as sequence-to-sequence task. Given a target a word highlighted in the exemplar, they utilize transformer-based model to generate word definition based on contextual information (Giulianelli et al., 2023). Unlike definition generation task aimed at generating individual definitions for each word, the frame definition generation task involves defining semantic frame evoked by multiple words.

heading: 5 Conclusion and Future Work
text: We introduce semantic frame definition generation tasks and employ ICL to generate definitions for induced semantic frames. We explore diverse demonstration formats through experiments on FrameNet. Results indicate that even with a limited number of ground truth demonstrations, our approach yields promising performance. Our future endeavors will involve experimenting with various LLMs to assess their efficacy. Additionally, although we currently leverage ground truth demonstrations from FrameNet, the clustering of induced frames may introduce noise. Investigating the impact of the noise on performance will be one of our focus in the future work.
Limitations
Our investigation focused on the Llama2-70b-chat model, without delving into other language models, potentially limiting the generalizability of our research findings. In future work, we plan to explore our method with various language models. Additionally, the study only examined the English language without exploring the efficacy across other languages.

heading: Acknowledgements
text: This work was supported by JST FOREST Program, Grant Number JPMJFR216N.

heading: A Generated definitions
text: In our evaluation method, we have ensured that the criteria for each point were included in the prompt, providing a reference for assessment, as illustrated in Figure 3. For instance, the majority of our results fell within the range of 3 to 4. Therefore, one could interpret the generated definitions as falling between 4: The two definitions are mostly equivalent, but some unimportant details differ, and 3: The two definitions are roughly equivalent, but some important information differs/missing. To offer a more comprehensive understanding of the generated definitions and their corresponding Def-Eval scores, we present examples of three frames in Tables 5, 6, and 7.

heading: Future Work
text: Our future endeavors will involve experimenting with various LLMs to assess their efficacy. Additionally, although we currently leverage ground truth demonstrations from FrameNet, the clustering of induced frames may introduce noise. Investigating the impact of the noise on performance will be one of our focus in the future work. Limitations
Our investigation focused on the Llama2-70b-chat model, without delving into other language models, potentially limiting the generalizability of our research findings. In future work, we plan to explore our method with various language models. Additionally, the study only examined the English language without exploring the efficacy across other languages