Uniform Meaning Representation Parsing as a Pipelined Approach (2024)

Abstract: Uniform Meaning Representation (UMR) is the next phase of semantic formalism following Abstract Meaning Representation (AMR), with added focus on inter-sentential relations allowing the representational scope of UMR to cover a full document. This, in turn, greatly increases the complexity of its parsing task with the additional requirement of capturing documentlevel linguistic phenomena such as coreference, modal and temporal dependencies. In order to establish a strong baseline despite the small size of recently released UMR v1.0 corpus, we introduce a pipeline model that does not require any training. At the core of our method is a two-track strategy of obtaining UMR’s sentence and document graphs separately, with the document-level triples being compiled at the token level and the sentence graph being converted from AMR graphs. By leveraging alignment between AMR and its sentence, we are able to generate the first automatic English UMR parses.

heading: 1 Introduction
text: While the end-to-end deep learning methods based on transformers (Vaswani et al., 2017; Devlin et al., 2019; Radford et al., 2019) helped usher in an era of Large Language Models (LLM) with outstanding results especially in the practical domains of Natural Language Processing (NLP), they also brought about significant advances in the performance of Abstract Meaning Representation (AMR) parsing. Once thought extremely challenging due to its inherently multi-tasking nature, AMR parsing with its adoption of transformer architecture (Bevilacqua et al., 2021; Lee et al., 2022a; Vasylenko et al., 2023) has since matured to a point where its automatic parses feature in various downstream applications (Bonial et al., 2020; Mansouri et al., 2023; Wang et al., 2023), often as a meaningful companion to the Pre-trained Language Models (PLM) llke T5 (Raffel et al., 2019) or BART (Lewis et al.,
2020). This trend serves to highlight the enduring interest of the community in leveraging symbolic meaning representations not only for the computational benefit in boosting the model performance but also as a way to better understand how a model seems to ‘understand’ language.
However, AMR by design is limited to the representational scope of a single sentence. Although efforts have been made to bring together multiple AMRs into a single unified structure (O’Gorman et al., 2018; Naseem et al., 2022), additional annotations across different sentences remain largely confined to coreference and implicit role labeling.

heading: Uniform Meaning Representation
text: In contrast, Uniform Meaning Representation (UMR) (Van Gysel et al., 2021) begins by inheriting AMR’s focus on predicate-argument structure in its sentence-level representation and further adds semantic coverage for aspect, scope, person and number for cross-lingual compatibility (Flanigan et al., 2022; Bonn et al., 2023b). In addition, UMR introduces new document-level triples which cover linguistic phenomena such as coreference, modal and temporal dependencies (Vigus et al., 2019; Zhang and Xue, 2018a; Yao et al., 2022) that potentially go beyond sentence boundaries.
Figure 1 provides an example of UMR annotation for a sample document of two sentences:
1. Kim left to join the others.
2. “They are probably eating,” she said.
At the top is an abstract ROOT node, whose immediate children AUTHOR (author of the text) and DCT (document creation time) serve as sub-roots of modal and temporal dependencies respectively. These abstract nodes are highlighted in lightblue.
A modal dependency graph (MDG), shown as a series of red edges in the figure, captures the epistemic certainty and polarity with which
40
the sources (formally known as conceivers) view another conceivers or events (Yao et al., 2021; Van Gysel et al., 2021). In the example, the Author knows with full certainty that Kim already left (:full-affirmative edge from the Author to s1l:leave-02), while Kim expresses uncertainty in her belief that They are eating at the moment (:partial-affirmative edge from s2p:person to s2e:eat-01). Since the Author presumably knew about Kim’s state of mind, a :full-affirmative modal relation between these two sources is finally established.
On the other hand, a temporal dependency graph (TDG) represents the temporal relations between events and time expressions such as DCT (Zhang and Xue, 2018b). The past tense of the main predicates left and said in the above example provides a strong indication of the actions having taken place before DCT, hence its two :before outgoing edges to s1l:leave-02 and s2e:eat-01. Further-
more, the chain of events dictates that Kim could not have possibly joined the others without having first left. This is annotated with the :after edge from s1l:leave-2 to s1j:join-04, which adds the temporal aspect to the :purpose relation that already exists between the two. Following the green edges in the figure reveals the temporal graph in its entirety.
Finally, the two sentences are further linked via the participation of same entities: Kim and the others. Their presence in the second sentence solely as pronouns, she and they, requires the context from the first sentence for anaphora resolution. These clusters of coreferent entities are highlighted as the same colored nodes in the figure connected by :same-entity edges between (1) s1p:person and s2p:person, and (2) s1p2:person and s2p2:person.
It is also worth noting the core differences between UMR sentence graphs and AMRs despite
their striking similarities. One of the notable discrepancies is the addition of :aspect annotations in UMR, visualized as purple edges in Figure 1, representing the internal state of an eventive concept as it relates to its status as an on-going, finished or habitual event, or simply a state with no changes over the course of action, or something else1 (Donatelli et al., 2018, 2019). In the figure, Kim having left and said had already come to an end (“performance”), whereas eating is presumably still an on-going process at the time of writing (“activity”).
Finally, pronouns in AMRs are replaced with generic person nodes with :refer-person edges denoting first, second or third point of view. Generic, non-named entities, including pronouns, are further annotated for their plurality with :refer-number relations, as seen with the blue outgoing edges from variables s1p2:person, s2p:person and s2p2:person in Figure 1.

heading: UMR Parsing
text: While these new features help expand the representational scope of UMR to include a full document, they come at a great cost to the parsing complexity. In addition to the sentence graph generation, a parser would have to produce an additional document-level structure whose scope generally encompasses multiple sentences. Since the triples in the document graph need to be grounded in the context of the sentence graphs (Figure 1), the parsing task effectively revolves around a series of pairwise relation classifications between sentence graph nodes that have been abstracted away from their source tokens, much like AMRs. This is further complicated by the limited number of publicly available annotations in the recently released UMR
1See umr-guidelines for the full lattice of aspectual values.
v1.0 corpus2 (Bonn et al., 2023a, 2024) . In light of these challenges, we propose to settle for a more tractable version of the problem that does not require any training. Our approach adopts a two-track strategy of obtaining sentence and document graphs separately. This is possible if we obtain the document-level triples at the token level, i.e., between the source tokens, not between the sentence graph nodes. By leveraging models individually trained for each of the document-level parsing tasks, we can set up a pipeline that compiles a list of document-level triples without any training on the limited UMR corpus. At the same time, we rely on off-the-shelf AMR parsers to first generate AMR, which is then subsequently converted into the UMR sentence graph using linguistically motivated heuristics. The final step involves the alignment of source tokens in the document-level triples to their corresponding nodes in the sentence graph, resulting in the final UMR structure.
The performance of our pipelined model is evaluated against the entire English section of the UMR v1.0 corpus, using a recently introduced AnCast++3 whose details are provided in Section 5. We report the highest comprehensive macro F1 score at 61.5, establishing a strong baseline for future improvement. The code is available at https://github.com/umr4nlp/umrlib.

heading: 2 UMR-v1.0 Corpus
text: UMR v1.0 corpus consists of documents annotated in 6 languages: Arapaho, Chinese, Cocama-Cocamilla, English, Navajo, and Sanapaná4. This work focuses only on 5 English documents, whose summary statistics are
2https://lindat.mff.cuni.cz/repository/xmlui/ handle/11234/1-5198
3https://github.com/sxndqc/ancast 4https://umr4nlp.github.io/web/data.html shows
the number of annotations for all 6 languages.
given in Table 1. The entire newsire domain (english_umr-0001, english_umr-0003, and english_umr-0005) overlaps with the LDC’s latest release of AMR R3 corpus LDC2020T025 (Knight et al., 2021). Each sentence receives 2 core layers of annotations: (1) sentence graph and (2) document-level triples involving at least one local variable from its sentence graph.

heading: Corpus Preprocessing
text: The corpus exhibits a few labeling inconsistencies. For instance, there are 12 occurrences of :AFF abbreviated modal relation label in addition to the more established :full-affirmative at 324. We attribute these and other similar occurrences to be simple errors and apply a cleanup to ensure labeling consistency across all of the annotations, e.g., :AFF replaced with :full-affirmative.
In addition, the :modal-strength relation (sometimes abbreviated as :modstr) is used as a shorthand to annotate a modal triple within a UMR sentence graph, although modal triples typically belong to a document-level annotation. In order to facilitate correct evaluation in our parsing experiments as required by AnCast++, these embedded modal triples are relocated from the sentence graph to its document-level annotation. It should be noted that this operation does not modify the content of the original annotation. We report parsing performance results with and without these procedures.

heading: 3 Model Description
text: In this section, we provide a detailed description of each of the models that makes up our pipeline. The entire flowchart is depicted in Figure 2.
5https://catalog.ldc.upenn.edu/LDC2020T02

heading: 3.1 AMR Parsing
text: AMR parsing aims to transform text into AMR where the meaning of a sentence is encoded in a single-rooted, directed and acyclic graph, as partially seen with the two sentence graphs in Figure 1 rooted by variables s1l and s2s whose black edges reveal the predicate-argument structure of each sentence. Due to its graphical nature, previous parsing methods often adopted graph methods such as finding the maximum spanning AMR graph (Flanigan et al., 2014, 2016), while others exploited the structural similarity between AMR and a dependency graph by applying a series of actions to transform the dependency graph into AMR in a transition-based framework (Wang et al., 2015, 2016; Wang and Xue, 2017). These approaches were largely superseded by larger models that began to pivot around various deep learning-based approaches (Foland and Martin, 2017; Lyu and Titov, 2018; Cai and Lam, 2020), culminating in the adoption of transformers (Bevilacqua et al., 2021). The subsequent advancements in AMR parsing relied on pretrained language models to consume and predict linearized AMRs (Chen et al., 2022; Bai et al., 2022; Yu and Gildea, 2022; Vasylenko et al., 2023), and the linearized representation of AMRs further opened up the possibility of a transitionbased approach where a sequence of transductions are interpreted graphically to incrementally build towards the final AMR graph (Zhou et al., 2021b,a; Drozdov et al., 2022).
Given the efficacy of transformers-based AMR parsers, along with the unmistakable similarity of AMR to the UMR sentence graph, it is only natural to choose AMR parsing as a starting point of the pipeline. We experiment with four AMR parsers: LeakDistill (Vasylenko et al., 2023), SPRING (Bevilacqua et al., 2021), AMRBART (Bai et al., 2022) and IBM Transition Parser (Zhou et al., 2021b,a; Lee et al., 2022b; Drozdov et al., 2022). Maximum Bayes Smatch Ensemble (MBSE) (Lee et al., 2022b) is additionally used to ensemble best performing parsers for further improvement. Experiments using BLINK (Ledell Wu, 2020) entity linker for Wikification did not improve the model performance and is thus omitted in our experimental setup. Finally, we run LEAMR (Blodgett and Schneider, 2021) to produce sentence-AMR alignment for subsequent use in AMR-to-UMR conversion. Appendix A provides more details on the setup used in our experiments.

heading: 3.2 AMR-to-UMR Conversion
text: Once an AMR parse is obtained, we apply heuristics for in-place conversion to the UMR sentence graph based on the mapping methodology described in Bonn et al. (2023b) and UMR guidelines6. We notice a few minor discrepancies between the methodology and some of the annotations in UMR v1.0; for instance, the guidelines advocates for :ref-person label whereas the corpus prefers :refer-person. In cases like this, we choose to follow the corpus for consistent parsing evaluation. A more recent work on AMR-to-UMR conversion provides fine-grained, nondeterministic mapping strategies based on the graph context (Post et al., 2024) but was not consulted for this work.
One of the practical challenges in AMR-toUMR conversion is the :aspect edge creation task for events. Its heavily context-dependent nature makes it difficult to reliably determine its child node label—i.e., aspectual value—via heuristics. For this reason, we seek the help of Universal Dependency-style syntactic analysis from UDPipe v2 (Straka, 2018) whose UD features such as Tense and Verbform provide limited but helpful insights. The distribution of the aspect labels from the corpus is shown in Table 3.
Another important aspect of conversion is handling of the non-named entities including the pronouns. Their ubiquitous presence makes it a highpriority sub-task, and here again we rely on UD features from which we are able to infer the plurality of any generic entity.
Table 2 provides the overall results with AMR parsers and subsequent in-place conversion to
6https://github.com/umr4nlp/umr-guidelines/ blob/master/guidelines.md
UMR sentence graph, using Smatch (Cai and Knight, 2013) and AnCast7 (Sun and Xue, 2024). AnCast is a recently introduced metric for evaluating graph-based meaning representations whose alignment strategy differs from the hill-climbing heuristics of Smatch by first identifying anchor nodes based on content similarity, and then iteratively propagating alignment throughout the neighborhood. It finally computes the labeled relation F1 score which measures the degree of matching for concepts and relations. This value represents the overall metric of AnCast and is reported in Table 2.

heading: 3.3 Modal Dependency Parsing
text: Modal dependency parsing (MDP) aims to construct a hierarchical structure representing the epistemic strength (full, neutral and partial) and polarity (affirmative and negative) of conceivers as related to other conceivers or events (Yao et al., 2021; Van Gysel et al., 2019). Largely based on FactBank (Saurí and Pustejovsky, 2009), UMR modal dependency sub-structure combines 3 modal strengths with 2 polarities as shown in Table 4. As
7not to be confused with AnCast++ whose details are provided in Section 5.
seen with Figure 3, the resulting graph typically involves heavy traffic through the Author who displays confidence or doubt in various statements s/he commits to in writing.
In practice, MDP consists of two different stages. First, the conceivers and events must be identified; then, each event or conceiver must be paired with the most appropriate parent in the text in a newlycreated modal triple whose label needs to be predicted. In our experiments, we use a prompt-based model described in Yao et al. (2022), where the two tasks are trained end-to-end in a joint manner based on language model priming. Table 4 shows
the distribution of modal labels in the UMR English corpus.

heading: 3.4 Temporal Dependency Parsing
text: In a similar vein to MDP, temporal dependency parsing (TDP) is the task of identifying a documentlevel graph whose nodes are time expressions (timex) and events, and edges represent the temporal relations between them. Specifically, an event first searches for its referent timex that is the most specific (i.e., closest) temporal anchor (Pustejovsky and Stubbs, 2011) in whose absence it settles for another event that can provide the most specific temporal context (Zhang and Xue, 2018b; Yao et al., 2020). Figure 4 depicts a temporal dependency graph for the sample document of two sentences.
TDP also consists of 2 stages. The timex and event identification is performed first, followed by edge generation between the identified nodes. For stage 1 we use the neural ranking model described in Yao et al. (2020) based on Zhang and Xue (2018a) and Ross et al. (2020) that extracts timex and events by labeling the appropriate span in the text8. Then we turn to the parser from Yao et al. (2023) which interprets TDP as a textual entailment (NLI) task in which the temporal relation is verbalized into text, requiring the model to infer entailment probability. Table 5 shows the distribu-
8We observed higher performance when TDP stage 1 output is augmented with events from MDP stage 1.
tion of temporal labels in the corpus.

heading: 3.5 Coreference
text: UMR supports two types of coreference—event and entity—which form disjoint clusters. Both may additionally participate in the :subset-of relationship. Table 6 provides the number of coreference labels in the corpus.

heading: Event Coreference
text: For cross-sentence event clustering, our pipeline relies on Cross-Document Coreference Resolution (CDLM) described in Cattan et al. (2021), which is pre-trained to include multiple documents by leveraging global attention. Although the model is designed with cross-document context in mind, we limit the global range to a single document. Since it requires event candidates be provided as input, we re-use the events identified in MDP stage 1.

heading: Entity Coreference
text: For entities we use wl-coref (Dobrovolskii, 2021) and caw-coref (D’Oosterlinck et al., 2023) which attempt to build a coreference link between individual words.

heading: 3.6 Context Grounding via Alignment
text: So far, the pipeline has produced two distinct structures—a sentence graph as a result of AMR-toUMR conversion, and document-level triples from
MDP, TDP and coreference—that are seemingly independent from each other. This is because the sentence graph is generated by transforming an AMR parse whose nodes have been abstracted away from their source tokens, whereas the document-level triples obtained from MDP, TDP and coreference are expressed as between these source tokens.
In order to bring these structures together, the final step of our pipelined approach involves the use of the alignment between the sentence graph and the source sentence provided by LEAMR9 to map the tokens in document-level triples to the corresponding nodes in the UMR sentence graph. This effectively means transferring the context of the document-level triples from the source sentence to the UMR sentence graph, and only after this stage do these structures demonstrate cohesion as required for UMR.

heading: 4 Experiments
text: We follow the flowchart in Figure 2 to generate UMR parses. Appendix A provides details on the experimental setup. Our model is evaluated against all of the English section of UMR v1.0 corpus. In order to cope with the input length limitation of some of the pipeline models, english_umr-0004 is split into smaller fragments each of which is treated as a separate document. The intermediate results for the split data are pieced together at the end into a single document for evaluation. Table 7 shows the experimental results using AnCast++ evaluation which we introduce in the next section.

heading: 5 Evaluation
text: Currently, there is no published work that can evaluate the performance of a UMR parser. To this end, we first provide Smatch scores for the sentence graphs evaluation in Table 2. Since the UMR sentence graphs resemble AMRs, Smatch can continue to provide a meaningful and comparable evaluation score during the transition towards UMR.
For the full UMR evaluation we adopt AnCast++10, a recently introduced open-source evaluation toolkit for UMR that provides an aggregated metric of Sentence, Modal, Temporal and Coreference scores. The Sentence graph evaluation is based on AnCast and is claimed to be highly correlated with Smatch despite differences in the align-
9LEAMR provides AMR-to-sentence alignment, which is preserved during the in-place conversion.
10https://github.com/sxndqc/ancast
ment strategy (Sun and Xue, 2024). While the Modal score is based on the number of overlaps in the modal triples owing to its inherently tree structure, Temporal and Coreference scores require finding the transitive closures via Depth-First Search (DFS) in order to identify clusters of nodes and relations, from which precision and recall measures are computed in terms of closed sets as follows:
p =
∑ ri∈R(|ri| × ∑ kj∈K rel(ri∩kj) rel(ri)
) ∑
rz∈R |rz|
r =
∑ ki∈K(|ki| × ∑ rj∈R rel(ki∩rj) rel(ki)
) ∑
kz∈K |kz| where ki and ri are node clusters in key (gold) and response (prediction) graphs, and rel(ki) and rel(ri) are the reference and deducted links respectively. This approach builds on Setzer et al. (2005) and Link-based Entity-Aware (LEA) metric (Moosavi and Strube, 2016; Moosavi, 2020).

heading: 6 Error Analysis
text: As a pipeline model, our parser is prone to error propagation when generating document-level triples. This is especially true with the event identification phase in MDP and TDP stage 1, where the identified event candidates are subsequently considered for the modal and temporal dependency edge generation as well as cross-sentence event coreference. Naturally, any event that goes undetected is non-recoverable in the subsequent pipeline. This is further compounded by the fact that the generated triples ultimately need to be aligned to the appropriate UMR sentence sub-graph but may be un-aligned or mis-aligned, resulting in low performance on the document-level parsing tasks. Nevertheless, MDP appears to show comparatively stronger performance because MDG is inherently
a tree unlike TDG and coreference clusters, with most of traffic consolidated around the Author.
The parser is also unable to guarantee 100% coverage of UMR as it is unable produce certain labels such as “Habitual” aspectual value and “:partial-negative” modal label. Another prominent example is “:subset-of” coreference label which makes up a sizable portion of coreference labels (Table 6), and its lack thereof carries significant repercussions for overall parsing performance. This is to be expected as none of the models are directly trained on the UMR-style of annotations, and it remains a major source of error in our experiments.
The corpus itself shows highly varied annotation styles across different documents. For instance, English UMR documents 1, 2 and 4 consistently annotate :modal relation from ROOT to AUTHOR, although its presence is implied in every document and is not strictly necessary—a view taken in documents 3 and 5. english_umr-0005 further stands out as what initially appears to be a news article abruptly turns into a dialogue, leading to subsequent sentence graphs being wrapped under (s / say-01 :ARG0 (p / person) :ARG1 ...) ‘phantom’ outer sub-graph. This explains the comparatively low score for the document.

heading: 7 Conclusions
text: This paper presents the first published UMR parsing model evaluated against UMR v1.0 English corpus using AnCast++. We describe our pipelined approach to cope with the shortage of publicly available UMR data so that no training on the UMR corpus is necessary. Our experimental results at 61.5 macro F1 establishes a strong baseline for future improvement. The proposed parser is suitable for modular upgrade by optimizing individual models, which we plan to visit in future work.

heading: Limitations
text: Due to the small number of UMR data available for evaluation, current parsing result is not yet stable. UMR English dataset further shows highly skewed distribution of number of sentences per document— as small as 2 for english_umr-0002 and over 140 for english_umr-0004 which is not taken into account by AnCast++. Increased number of UMR annotations will partially mitigate this issue.
The proposed UMR parser uses sub-models trained in English and is unable to parse any other languages. To apply this model in a cross-lingual setting depends on the availability of models such as temporal dependency parser being trained either multi-lingually or on non-English datasets.
Since the pipeline consists of multiple models each of which may require a different set of dependencies, the parser is difficult to set up for use in practice. We therefore provide a WebUI version of our parser which serves as a one-stop interface to interact with every component in the pipeline.

heading: Acknowledgments
text: This work is supported by grants from the CNS Division of National Science Foundation (Awards no: NSF_2213804) entitled “Building a Broad Infrastructure for Uniform Meaning Representations”. Any opinions, findings and conclusions or recommendations expressed in this material do not necessarily reflect the views of NSF.

heading: A Experimental Setup
text: Experiments were run on NVIDIA RTX 3090.

heading: AMR Parser
text: We found 4-way and 5-way MBSE models to produce the highest Smatch and AnCast scores on UMR sentence graphs evaluation (Table 2). We were also able to obtain the highest AnCast++ scores on full UMR evaluation using 5-way MBSE (Table 7). These include:
1. LeakDistill trained on AMR R311.
2. SPRING trained on AMR R3.
3. Struct-BART trained on AMR R3 and parsed using ensemble of 3 seeds: 42, 43, and 44.
11Checkpoint ‘best-smatch_checkpoint_12_0.8534’ is available upon request to the authors
4. AMRBART 3.0 trained on AMR R3.
5. AMRBART 2.0 trained on AMR R2 (not part of 4-way MBSE).
We do not run the BLINK entity linking system in our pipeline.

heading: Modal Dependency Parsing
text: mdp-prompt (Yao et al., 2022) is the prompt-based modal dependency parser trained on publicly available English modal dependency dataset12 (Yao et al., 2021). We exactly follow the training configurations described in the paper for English.

heading: Temporal Dependency Parsing
text: Unlike MDP where a single parser can perform stage 1 and stage 2 jointly, we train two separate models since the best stage 2 parser does not produce stage 1 outputs.

heading: TDP Stage 1
text: To identify events and timex, we use the XLMRoberta (Conneau et al., 2020) based ranking model (Yao et al., 2020) whose source code is not publicly available but is similar to that of mdpprompt.
The model is trained on publicly available English temporal dependency dataset13 for 30 epochs with learning rate of 2e-5 and max sequence length of 128. The model processes a long document by splitting it into smaller segments before encoding each with the language model. When doing so, we allow the model to apply segmentation by letting each overlap with one another. These procedures are in accordance with what is described in the paper.
In practice, the identified events are merged with those found by mdp-prompt, leading to better results. Finally, the merged events also serve as inputs to CDLM for event coreference.

heading: TDP Stage 2
text: thyme-tdg (Yao et al., 2023) is trained following the model implementation details as specified for the general-domain experiments, but we allow training to last for 10 epochs rather than 3. We use seed 42 for data preparation as well as model training.
12https://github.com/Jryao/modal_dependency/ tree/main/data
13https://github.com/Jryao/temporal_dependency_ graphs_crowdsourcing/tree/master/tdg_data
In practice, we find that the ranking model (Yao et al., 2020) should also be trained for stage 2 eventto-time and event-to-event edge generation task, whose outputs are then fed to thyme-tdg. In both scenarios the hyperparameters remain the same as described in the paper.

heading: Coreference
text: CDLM for event coreference is trained on ECB+ corpus14 (Cybulska and Vossen, 2014). For wlcoref and caw-coref, we use the Roberta (Liu et al., 2019) based pre-trained models publicly available at their respective Github repositories. In our experiments, using wl-coref led to higher AnCast++ scores.
14https://www.newsreader-project.eu/results/ data/the-ecb-corpus/

heading: Future Work
text: 5 macro F1 establishes a strong baseline for future improvement. The proposed parser is suitable for modular upgrade by optimizing individual models, which we plan to visit in future work