Annotating FrameNet via Structure-Conditioned Language Generation (2024)

Abstract: Despite the remarkable generative capabilities of language models in producing naturalistic language, their effectiveness on explicit manipulation and generation of linguistic structures remain understudied. In this paper, we investigate the task of generating new sentences preserving a given semantic structure, following the FrameNet formalism. We propose a framework to produce novel frame-semantically annotated sentences following an overgenerateand-filter approach. Our results show that conditioning on rich, explicit semantic information tends to produce generations with high human acceptance, under both prompting and finetuning. Our generated frame-semantic structured annotations are effective at training data augmentation for frame-semantic role labeling in low-resource settings; however, we do not see benefits under higher resource settings. Our study concludes that while generating highquality, semantically rich data might be within reach, the downstream utility of such generations remains to be seen, highlighting the outstanding challenges with automating linguistic annotation tasks.1

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated unprecedented capabilities in generating naturalistic language. These successes hint at LMs’ implicit capabilities to “understand” language; but are they capable of processing explicit symbolic structures in order to generate language consistent with the structures? Not only would this help us understand the depth of LLMs’ linguistic capabilities but would also serve to efficiently and cheaply expand existing sources of linguistic structure annotation. In this work, we investigate the abilities
1Our code is available at https://github.com/ X-F-Cui/FrameNet-Conditional-Generation.
of LLMs to generate annotations for one such resource of linguistic structure: FrameNet, a lexical database grounded in the theory of frame semantics (Fillmore, 1985; Ruppenhofer et al., 2016). We propose an approach for language generation conditioned on frame-semantic structure such that the generation (i) is consistent with the frame structure, (ii) is acceptable by humans and (ii) is useful for a downstream task, namely frame-semantic role labeling (Gildea and Jurafsky, 2000b).
681
Our framework for generating frame-semantic annotations leverages both the FrameNet hierarchy and LLMs’ generative capabilities to transfer annotations from existing sentences to new examples. Specifically, we introduce frame structureconditioned language generation, focused on specific spans in the sentence such that the resulting sentence follows the given frame structure and is also acceptable to humans. Overall, we follow an overgenerate-and-filter pipeline, to ensure semantic consistency of the resulting annotations. Our framework is outlined in Figure 1.
Our intrinsic evaluation, via both human judgment and automated metrics, show that the generated sentences preserve the intended framesemantic structure more faithfully compared to existing approaches (Pancholy et al., 2021). As an extrinsic evaluation, we use our generations to augment the training data for frame-semantic role labeling: identifying and classifying spans in the sentence corresponding to FrameNet frames. Under a low-resource setting, our generation annotations tend to be effective for training data augmentation for frame-semantic role labeling. However, these trends do not translate to a high-resource setting; these findings are consistent with observations from others who have reported challenges in leveraging LLMs for semantic parsing tasks, such as constituency parsing (Bai et al., 2023), dependency parsing (Lin et al., 2023), and abstract meaning representation parsing (Ettinger et al., 2023). Our findings prompt further investigation into the role of LLMs in semantic structured prediction.

heading: 2 FrameNet and Extensions
text: Frame semantics theory (Gildea and Jurafsky, 2000a) posits that understanding a word requires access to a semantic frame—a conceptual structure that represents situations, objects, or actions, providing context to the meaning of words or phrases. Frame elements (FEs) are the roles involved in a frame, describing a certain aspect of the frame. A Lexical Unit (LU) is a pairing of tokens (specifically a word lemma and its part of speech) and their evoked frames. As illustrated in Figure 1, the token “disciplined” evokes the LU discipline.v, which is associated with the frame REWARDS_AND_PUNISHMENT, with FEs including Time, Evaluee, and Reason. Grounded in frame semantics theory, FrameNet (Ruppenhofer et al., 2006) is a lexical database, featuring sentences that
are annotated by linguistic experts according to frame semantics. Within FrameNet, the majority of sentences are annotated with a focus on a specific LU within each sentence, which is referred to as lexicographic data; Figure 1 shows such an instance. A subset of FrameNet’s annotations consider all LUs within a sentence; these are called full-text data; Figure 1 does not consider other LUs such as grow.v or break.v.
FrameNet has defined 1,224 frames, covering 13,640 lexical units.The FrameNet hierarchy also links FEs using 10,725 relations. However, of the 13,640 identified LUs, only 62% have associated annotations. Our approach seeks to automatically generate annotated examples for the remaining 38% of the LUs, towards increasing coverage in FrameNet without laborious manual annotation.
Sister LU Replacement Pancholy et al. (2021) propose a solution to FrameNet’s coverage problem using an intuitive approach: since LUs within the same frame tend to share similar annotation structures, they substitute one LU (the target LU) with another (a sister LU) to yield a new sentence. This replacement approach only considers LUs with the same POS tag to preserve the semantics of the original sentence; for instance, in Figure 1, we replace the sister LU discipline.v with the target LU reward.v. However, due to the nuanced semantic differences between the two LUs, the specific content of the FE spans in the original sentence may no longer be consistent with the target LU in the new sentence. Indeed Pancholy et al. (2021) report such semantic mismatches as their primary weakness.
To overcome this very weakness, our work proposes leveraging LLMs to generate FE spans that better align with the target LU, as described subsequently. For the rest of this work, we focus solely on verb LUs, where initial experiments showed that the inconsistency problem was the most severe. Details of FrameNet’s LU distribution by POS tags, along with examples of non-verb LU replacements can be found in Appendix A.

heading: 3 Generating FrameNet Annotations via Frame-Semantic Conditioning
text: We propose an approach to automate the expansion of FrameNet annotations by generating new annotations with language models. Given sister LUreplaced annotations (§2; Pancholy et al., 2021), we select FE spans which are likely to be semantically inconsistent (§3.1), generate new sentences
with replacement spans by conditioning on framesemantic structure information (§3.2) and finally filter inconsistent generations (§3.3).

heading: 3.1 Selecting Candidate FEs for Generation
text: We identify the FEs which often result in semantic inconsistencies, in order to generate replacements of the spans corresponding to such FEs. Our selection takes into account the FE type, its ancestry under FrameNet, and the span’s syntactic phrase type. Preliminary analyses, detailed in Appendix B, help us narrow the criteria as below:
1. FE Type Criterion: The FE span to be generated must belong to a core FE type, i.e., the essential FEs that are necessary to fully understand the meaning of a frame. 2. Ancestor Criterion: The FE should not possess Agent or Self-mover ancestors. 3. Phrase Type Criterion: The FE’s phrase type should be a prepositional phrase.
Qualitative analyses revealed that it suffices to meet criterion (1) while satisfying either (2) or (3). For instance, in Figure 1, under REWARDS_AND _PUNISHMENTS, only the FEs Evaluee and Reason are core (and satisfy (2)) while Time is not; thus we only select the last two FE spans for generation.

heading: 3.2 Generating Semantically Consistent Spans
text: We generate semantically consistent FE spans for selected candidate FEs via two approaches: finetuning a T5-large model (Raffel et al., 2019) and prompting GPT-4 Turbo, following Mishra et al. (2021). In each case, we condition the generation on different degrees of semantic information: No Conditioning We generate FE spans without conditioning on any semantic labels. FE-Conditioning The generation is conditioned on the type of FE span to be generated. Frame+FE-Conditioning The generation is conditioned on both the frame and the FE type.
The above process produces new sentences with generated FE spans designed to align better with the target LU, thereby preserving the original frame-semantic structure. However, despite the vastly improved generative capabilities of language models, they are still prone to making errors, thus not guaranteeing the semantic consistency we aim for. Hence, we adopt an overgenerate-and-filter approach (Langkilde and Knight, 1998; Walker et al., 2001): generate multiple candidates and aggressively filter out those that are semantically inconsistent. Details on fine-tuning T5 and prompting
GPT-4 are provided in Appendix C.

heading: 3.3 Filtering Inconsistent Generations
text: We design a filter to ensure that the generated sentences preserve the same semantics as the expert annotations from the original sentence. This requires the new FE spans to maintain the same FE type as the original. We propose a new metric FE fidelity, which checks how often the generated spans have the same FE type as the original. To determine the FE type of the generated spans, we train an FE type classifier on FrameNet by finetuning SpanBERT, the state-of-the-art model for span classification (Joshi et al., 2019).2 We use a strict filtering criterion: remove all generations where the FE classifier detects even a single FE type inconsistency, i.e. only retain instances with perfect FE fidelity.

heading: 3.4 Intrinsic Evaluation of Generations
text: We evaluate our generated frame-semantic annotations against those from Pancholy et al. (2021), before and after filtering (§3.3). We consider three metrics: perplexity under Llama-2-7B (Touvron et al., 2023) for overall fluency, FE fidelity, and human acceptance. We randomly sampled 1000 LUs without annotations under FrameNet and used our generation framework to generate one instance each for these LUs. For human acceptability, we perform fine-grained manual evaluation on 200 examples sampled from the generated instances.3 We deem an example acceptable if the FE spans semantically align with the target LU and preserve the FE role definitions under FrameNet. We provide a qualitative analysis of generated examples in Appendix E.
Results in Table 1 shows that our filtering approach—designed for perfect FE fidelity— improves performance under the other two metrics. Compared to rule-based generations from Pancholy et al. (2021), our filtered generations fare better under both perplexity and human acceptability, indicating improved fluency and semantic consistency. Most importantly, models incorporating semantic information, i.e., FE-conditioned and Frame+FE-
2Our SpanBERT FE classifier attains 95% accuracy on the standard FrameNet 1.7 splits; see Appendix D for details.
3Human evaluation is mainly conducted by the first author of this work. These annotations were validated by two independent volunteers unfamiliar with generated data evaluating the same examples from GPT-4 | Frame+FE, where the ratings differ by only 1% from our primary ratings. This suggests a consistent rating quality across different observers.
conditioned models, achieve higher human acceptance and generally lower perplexity compared to their no-conditioning counterparts, signifying that semantic cues improve both fluency and semantic consistency. Even before filtering, FE fidelity increases with the amount of semantic conditioning, indicating the benefits of structure-based conditioning. We also provide reference-based evaluation in Appendix F.

heading: 4 Augmenting Data for Frame-SRL
text: Beyond improving FrameNet coverage, we investigate the extrinsic utility of our generations as training data to improve the frame-SRL task, which involves identifying and classifying FE spans in sentences for a given frame-LU pair. Here, we consider a modified Frame-SRL task, which considers gold-standard frames and LUs, following Pancholy et al. (2021). This remains a challenging task even for powerful models like GPT-4, which achieves a test F1 score of only 0.228 in contrast to Lin et al. (2021)’s state-of-the-art F1 score of 0.722. For experimental ease, we fine-tune a SpanBERT model on FrameNet’s full-text data as our parser4 and avoid using existing parsers due to their reliance on weaker, non-Transformer architectures (Swayamdipta et al., 2017), complex problem formulation (Lin et al., 2021), or need for extra frame and FE information (Zheng et al., 2022).
As a pilot study, we prioritize augmenting the training data with verb LUs with F1 scores below 0.75 on average. This serves as an oracle augmenter targeting the lowest-performing LUs in the test set. For the generation of augmented data, we use our top-performing models within T5 and GPT-4 models according to human evaluation: T5 | FE and GPT-4 | Frame+FE models. Of 2,295
4This parser obtains an F1 score of 0.677, see Table 2.
LUs present in the test data, 370 were selected for augmentation, resulting in 5,631 generated instances. After filtering, we retain 4,596 instances from GPT-4 | Frame+FE and 4,638 instances from T5 | FE. Additional experiments using different augmentation strategies on subsets of FrameNet are in Appendix G.
Table 2 shows the Frame-SRL performance, with and without data augmentation on all LUs and on only the augmented LUs. Despite the successes with human acceptance and perplexity, our generations exhibit marginal improvement on overall performance, and even hurt the performance on the augmented LUs. We hypothesize that this stagnation in performance stems from two factors: (1) the phenomenon of diminishing returns experienced by our Frame-SRL parser, and (2) the limited diversity in augmented data. Apart from the newly generated FE spans, the generated sentences closely resemble the original, thereby unable to introduce novel signals for frame-SRL; see subsection G.3 and Appendix H for more experiments on generation diversity. We speculate that Pancholy et al. (2021)’s success with data augmentation despite using only sister LU replacement might be attributed to use of a weaker parser (Swayamdipta et al., 2017), which left more room for improvement.

heading: 4.1 Augmenting Under Low-Resource Setting
text: To further investigate our failure to improve frameSRL performance via data augmentation, we simulate a low-resource scenario and conduct experiments using increasing proportions of FrameNet training data under three settings: (1) training our SRL parser with full-text data, (2) training our SRL parser with both full-text and lexicographic data (which contains 10x more instances), and (3) training an existing frame semantic parser (Lin et al., 2021)5 with full-text data, to control for the use of our specific parser.
Figure 2 shows that parsers across all three settings exhibit diminishing returns, especially on the second setting, which utilizes the largest training set. This indicates that there seems to be little room for improvement in frame-SRL, even with human annotated data.
Following our learning curves, we further evaluate the utility of our generations without the influence of diminishing returns, by performing data augmentation in a low-resource setting. Specifically, we augment 25% of the full-text training data with an additional 6.25% of data generated using our method. As demonstrated in Figure 2, the performance of the model in this scenario not only exceeds that of the 25% dataset without augmentation but the results of the 25% dataset augmented with 6.25% of human-annotated data. This show-
5Lin et al. (2021) break frame-SRL into three subsequent sub-tasks: target identification, frame identification, and SRL, contributing to worse overall performance.
cases the high utility of our generations for targeted data augmentation in a low-resource setting.

heading: 5 Related Work
text: Data Augmentation for FrameNet While FrameNet annotations are expert annotated for the highest quality, this also limits their scalability. In an effort to improve FrameNet’s LU coverage, Pavlick et al. (2015) proposes increasing the LU vocabulary via automatic paraphrasing and crowdworker verification, without expanding the lexicographic annotations. Others address this limitation by generating annotations through lexical substitution (Anwar et al., 2023) and predicate replacement (Pancholy et al., 2021); neither leverages the generative capabilities of LLMs, however.
Controlled Generation Other works have explored using semantic controls for generation tasks. Ou et al. (2021) propose FrameNet-structured constraints to generate sentences to help with a story completion task. Ross et al. (2021) studied controlled generation given target semantic attributes defined within PropBank, somewhat coarse-grained compared to FrameNet. Similarly, Ye et al. (2024) employ the rewriting capabilities of LLMs to generate semantically coherent sentences that preserve named entities for the Named Entity Recognition task. Guo et al. (2022) introduced GENIUS, a novel sketch-based language model pre-training approach aimed at reconstructing text based on keywords or sketches, though not semantic structures; this limits its effectiveness in capturing the full context.

heading: 6 Conclusion
text: Our study provides insights into the successes and failures of LLMs in manipulating FrameNet’s linguistic structures. When conditioned on semantic information, LLMs show improved capability in producing semantically annotated sentences, indicating the value of linguistic structure in language generation. Under a low-resource setting, our generated annotations prove effective for augmenting training data for frame-SRL. Nevertheless, this success does not translate to a high-resource setting, echoing challenges reported in applying LLMs to other flavors of semantics (Bai et al., 2023; Lin et al., 2023; Ettinger et al., 2023). These outcomes underline the need for further exploration into how LLMs can be more effectively employed in automating linguistic structure annotation.

heading: Acknowledgements
text: We thank the anonymous reviewers and area chairs for valuable feedback. This work benefited from several fruitful discussions with Nathan Schneider, Miriam R. L. Petruck, Jena Hwang, and many folks from the USC-NLP group. We thank Ziyu He for providing additional human evaluation on generated annotations. This research was partly supported by the Allen Institute for AI and an Intel Rising Stars Award.

heading: Limitations
text: While our work contributes valuable insights into LLMs’ capabilities towards semantic structureconditioned generation, we acknowledge certain limitations. First, our research is exclusively centered on the English language. This focus restricts the generalizability of our findings to other languages, which likely present unique linguistic structures with associated semantic complexity. The exploration of LLMs’ capabilities in linguistic structures manipulation and generation in languages other than English remains an open direction for future research.
Moreover, we do not consider the full complexity of the frame semantic role labeling task, which also considers target and frame identification. Even for the argument identification task, we use an oracle augmentation strategy. Despite this relaxed assumption, the generations had limited improvement in performance, except in low-resource settings, where targeted data augmentation proved more effective. This indicates potential for improvement in scenarios with limited annotated data but highlights the need for further research in diverse and complex settings.

heading: Ethics Statement
text: We recognize the inherent ethical considerations associated with utilizing and generating data via language models. A primary concern is the potential presence of sensitive, private, or offensive content within the FrameNet corpus and our generated data. In light of these concerns, we carefully scrutinize the generated sentences during the manual analysis of the 200 generated examples and do not find such harmful content. Moving forward, we are committed to ensuring ethical handling of data used in our research and promoting responsible use of dataset and language models.

heading: A FrameNet Statistics

heading: A.1 Distribution of Lexical Units
text: Table 3 illustrates a breakdown of FrameNet corpus categorized by the POS tags of the LUs. Specifically, we report the number of instances and the average count of candidate FEs per sentence, corresponding to LUs of each POS category. The two predominant categories are verb (v) LUs and noun (n) LUs, with verb LUs exhibiting a higher average of candidate FE spans per sentence compared to noun LUs.

heading: A.2 Replacement of non-verb LUs
text: Table 4 shows several examples of non-verb LU replacement, where the resulting sentences mostly preserve semantic consistency. Given the extensive number of annotated verb LUs available for LU replacement and candidate FEs per sentence for masking and subsequent structure-conditioned generation, our generation methodology is primarily applied to verb LUs.

heading: A.3 Full-Text and Lexicographic Data
text: Table 5 shows the distribution of the training, development, and test datasets following standard splits on FrameNet 1.7 from prior work (Kshirsagar et al., 2015; Swayamdipta et al., 2017; Peng et al., 2018; Zheng et al., 2022). Both the development and test datasets consist exclusively of full-text data, whereas any lexicographic data, when utilized, is solely included within the training dataset. Since our generation approach is designed to produce lexicographic instances annotated for a single LU, when augmenting fulltext data (§4), we break down each fulltext example by annotated LUs and process them individually as multiple lexicographic examples.

heading: Dataset Split Size

heading: B Details on Candidate FEs Selection
text: There are three criteria for determining a candidate FE span, i.e., FE Type Criterion, Ancestor Criterion, and Phrase Type Criterion. In preliminary experiments, we have conducted manual analysis on the compatibility of FE spans with replacement LUs on 50 example generations. As demonstrated through the sentence in Figure 1, the FE Type criterion can effectively eliminate non-core FE that do not need to be masked, i.e., "Growing up" of FE type Time. Also, the Phrase Type Criterion can identify the candidate FE "for breaking the rules", which is a prepositional phrase. Moreover, we find that FEs of Agent or Self-mover type describes a human subject, which is typically independent of
the LU evoked in the sentence. Since FE types within the same hierarchy tree share similar properties, we exclude FEs of Agent and Self-mover types, as well as any FEs having ancestors of these types, from our masking process, as illustrated in Table 6.

heading: C Details on Span Generation

heading: C.1 T5-large Fine-Tuning
text: During the fine-tuning process of T5-large, we incorporate semantic information using special tokens, which is demonstrated in Table 7 through the example sentence in Figure 1. T5 models are finetuned on full-text data and lexicographic data in FrameNet for 5 epochs with a learning rate of 1e-4 and an AdamW (Loshchilov and Hutter, 2017) optimizer of weight decay 0.01. The training process takes around 3 hours on 4 NVIDIA RTX A6000 GPUs.

heading: C.2 GPT-4 Few-shot Prompting
text: When instructing GPT-4 models to generate FE spans, we provide the task title, definition, specific instructions, and examples of input/output pairs along with explanations for each output, as demonstrated in Table 8.

heading: D FE Classifier Training Details
text: Our classifier operates on the principle of classifying one FE span at a time. In cases where multiple FE spans are present within a single sentence, we split these into distinct instances for individual processing. For each instance, we introduce special tokens—<LU_START> and <LU_END>—around the
LU, and <FE_START> and <FE_END> around the FE span. Additionally, the name of the evoked frame is appended to the end of the sentence. To train our classifier to effectively discern valid FE spans from invalid ones, we augment training data with instances where randomly selected word spans are labeled as “Not an FE”, constituting approximately 10% of the training data. The FE classifier is finetuned on full-text data and lexicographic data for 20 epochs with a learning rate of 2e-5 and an AdamW optimizer with weight decay 0.01. The training process takes around 4 hours on 4 NVIDIA RTX A6000 GPUs.

heading: E Human evaluation of generated examples
text: We perform fine-grained manual analysis on 200 generated sentences to evaluate the quality of model generations based on two criteria: (1) sentence-level semantic coherence and (2) preservation of original FE types. We present 10 example sentences from the overall 200 in Table 9.
F Intrinsic Evaluation on FrameNet Test Data
To evaluate the quality of generated sentences on reference-based metrics such as ROUGE (Lin, 2004) and BARTScore (Yuan et al., 2021), we perform §3.1 and §3.2 on the test split of FrameNet 1.7 with verb LUs. As observed in Table 10, the T5 | FE model surpasses others in ROUGE scores, signifying superior word-level precision, while GPT-4 achieves the highest BARTScore, indicat-
ing its generated sentences most closely match the gold-standard FE spans in terms of meaning. For reference-free metrics, GPT-4 | FE performs well in both log perplexity and FE fidelity, showcasing its ability to produce the most fluent and semantically coherent generations.

heading: G More on Augmentation Experiments

heading: G.1 Experiments using Non-oracle Augmentation Strategy
text: To evaluate the robustness and generalizability of our model under realistic conditions, we employed an augmentation strategy similar to that used by Pancholy et al. (2021). Specifically, we remove all annotated sentences of 150 randomly selected verb LUs from the full text training data and train our baseline parser using the remaining training data. Our full model was trained on instances of the 150 verb LUs re-generated by our framework along with the data used to train the baseline model. As a result, the test F1 scores for the baseline model and full model were 0.689 and 0.690, respectively, which echos the lack of significant improvement using the oracle augmentation strategy.

heading: G.2 Experiments on Verb-only Subset
text: Since our generation method mainly focuses on augmenting verb LUs, we conduct additional augmentation experiments using a subset of FrameNet that includes only verb LU instances. To ensure model performance on a subset of data, we incorporate lexicographic data with verb LUs into our
training set, resulting in a training set enriched with 80.2k examples, a development set comprising approximately 600 examples, and a test set containing about 2k examples. We experimented with different augmentation percentages both with and without filtering, as shown in Table 11. We use an oracle augmenter to augment LUs inversely proportional to their F1 scores from the unaugmented experiments. To expand coverage on more LUs during augmentation, we augment all LUs rather than limiting to those with F1 scores below 0.75. Although the improvements are marginal, the outcome from filtered augmentations is generally better than those from their unfiltered counterparts.

heading: G.3 Experiments on Multiple Candidate Generations
text: In the main experiments conducted in this paper, we generated one instance for each LU-sentence pair. However, instances could be filtered out due to inconsistent FE spans, which could hurt generation diversity. To address this, we further experimented with generating three candidate instances for each LU-sentence pair to improve generation coverage.
Specifically, we augmented the full-text training data by 25% under both the 1-candidate and 3-candidate settings. However, as shown in Table 12, generating three candidates did not lead to performance improvements in the F1 score. This suggests that simply increasing the number of generated candidates may not be sufficient to enhance
generation diversity. Future work may need to explore more effective strategies to improve the diversity of generated data.

heading: H Effect of Filtering on Generation Diversity
text: To examine the effect of filtering on the diversity of generated data, we have conducted experiments to compute the Self-BLEU scores to measure diversity for the same 1,000 instances discussed in §3.4. A lower Self-BLEU score indicates higher diversity, as it signifies less overlap within the generated texts. As demonstrated in Table 13, the diversity of the generated candidates increases after applying the filter, even surpassing the diversity of the original instances created by humans. This substantiates the effectiveness of our filtering process in
enhancing the variability and quality of the generated sentences.

heading: Future Work
text: Future work may need to explore more effective strategies to improve the diversity of generated data