LREC 2020 WORKSHOP
Language Resources and Evaluation Conference
11–16 May 2020

International FrameNet Workshop 2020
Towards a Global, Multilingual FrameNet

PROCEEDINGS

Edited by:
Tiago T. Torrent, Collin F. Baker, Oliver Czulo, Kyoko Ohara andMiriam R. L. PetruckProceedings of the LRECInternational FrameNet Workshop 2020:
Towards a Global, Multilingual FrameNet

Edited by: Tiago T. Torrent, Collin F. Baker, Oliver Czulo, Kyoko Ohara and Miriam R. L. Petruck

ISBN: 979-10-95546-58-0
EAN: 9791095546580

For more information:
European Language Resources Association (ELRA)
9 rue des Cordelières
75013, Paris
Francehttp://www.elra.info
Email: lrec@elda.org

c European Language Resources Association (ELRA)

These Workshop Proceedings are licensed under a Creative Commons
Attribution-NonCommercial 4.0 International Licenseii

Introduction
This workshop continues the series of International FrameNet Workshops, based on frame semantics
and FrameNets around the world, including meetings in Berkeley, California in 2013 (sponsored by
the Swedish FrameNet group), in Juiz de Fora, Brazil in 2016 (sponsored by FrameNet Brasil), andin Miyazaki, Japan in 2018 (in conjunction with the LREC conference there). The last of these wasspecifically designed to bring together two sometimes separate groups of researchers, those concentratingon frame semantics and those concentrating on construction grammar, which Charles J. Fillmore andmany of the same colleagues developed in parallel over a period of several decades. The call for papersof the current conference emphasized that the workshop would welcome studies of both theoretical andpractical issues, and we are fortunate to have strong contributions of both types, sometimes within asingle paper. A number of the papers arise out of the parallel annotation of Ken Robinson’s TED talk "Doschools kill creativity?". This very popular talk has been translated (or at least subtitled) in a wide varietyof languages. The attendees at the International FrameNet Workshop in 2016 decided that different
groups would undertake Frame Semantic annotation for it, based on Berkeley FrameNet Release 1.7 asshared standard. These efforts have raised a number of important theoretical issues while also helping
new FrameNet projects to get underway in several languages.
Three papers deal with expansion of FrameNets to new languages and content areas:• Gargett and Leung describe the building of a new FrameNet for Emirati Arabic, and how they wereable to use novel combinations of existing databases and tools to create an Emirati Arabic corpus
and begin annotating it semiautomatically with frames and frame elements derived mainly fromBerkeley FrameNet. They report encouraging numbers for inter-annotator agreement and goodprospects for further automatic processing.
• Giouli et al. describe participating in the shared annotation task on the TED talk and the issues
they encountered in trying to use the Berkeley FrameNet frames for Modern Greek. Some of theseare simply gaps in the BFN lexical coverage, others are interesting issues related to differences
between English and Greek.
• L’Homme et al. are building a multilingual frame semantic resource for the specialized
terminology relating to the environment. They have created more than 100 new frames for thispurpose and "slightly modified" 40 more in the process; they began with French and expanded
to English, Spanish, Portuguese, and Chinese recently. They propose a general technique for
FrameNet expansion in technical domains.
Two papers deal with how frame semantics applies across languages and what relation it has todifferences in constructions across languages:• Ohara concentrates on constructions, comparing constructions in English and in Japanese, also
based on examples from the shared annotation task. In this study, she expands on earlier work
comparing frames and constructions between English and Japanese children’s books, and points
out that the Japanese version of the TED talk is not exactly intended as a complete translation of
the English version.
• Baker and Lorenzi describe a variety of techniques for aligning frames between FrameNets indifferent languages. For this purpose, the authors have created a tool to visualize the alignments
and compare the techniques used, an effort that should aid in testing different combinations ofalignment methods.
iii

Three papers deal with practical application of frame semantics to other NLP tasks:• Marzinotto projects FrameNet annotation from English to French using two cutting-edge NLP
methods, attention-based machine translation and parsing based on BERT embeddings, using
data from two French frame semantic resources, ASFALDA and CALOR. The study found thatprojecting annotation in this way slightly improves the performance of the parser.
• Lee et al. report on the creation of a database of Mandarin Chinese verbs, and how it was used
to analyze the occurrences of verbs in Chinese language textbooks used in China across grades
1-12. They found that the distinction between core and non-core frame elements helps to predictthe reading difficulty of the texts.
• Gruzitis et al., in a surprising change of direction, start from the Latvian FrameNet corpus, which
also has a universal dependency layer and semi-automatically create a complete set of Propbank
annotation for the same corpus; this process was found to be much easier than a mapping from
Propbank to FrameNet would have been.
Four papers deal with connecting frame semantics to other types of meaning:• Czulo et al. remind us that from early in the development of frame semantics, Fillmore conceivedof the theory as encompassing both lexical frames where meanings are tied to specific lexical items
and pragmatic frames (called "scenes" at that point) tied to the situational context of usage, rather
than lexical items. The paper offers an update on the concept, with examples from the sharedannotation task.
• Postma et al. describe a new approach to annotating both semantic frames and real-worldreferences (based on Wikidata) on the same texts, creating rich possibilities for future work
analyzing text coherence and variations in framing of news events. The authors have developed atool which both assists and constrains the annotator in this process.
• Remijnse and Minnema also deal with real-world references, but for the problem of constructing
narratives from news reports. The authors found that pragmatically inferred frames were neededto achieve coherence across documents.
• Belcavello et al. discuss a novel tool they have created for annotating multimodal data withsemantic frames. The paper provides an example worked in detail about how the text, audio, andvideo can all be annotated somewhat differently, with attention to how they are actually temporally
related in the data – here, a TV series about travel to different countries. The paper also illustrates
how a highly elaborated set of ternary qualia relations can help to connect the annotations of thedifferent modalities.
The Organizing Committee would like to thank the authors for submitting so many substantial papers andthe reviewers for helping to select and improve those included here. We would also like to thank LREC
for scheduling a full day for this workshop, so that we could cover the full range of relevant topics. As thispreface is being written, the pandemic caused by COVID-19 makes it uncertain whether the workshop
originally planned for May 16, 2020 in Marseilles in conjunction with the LREC conference will simply
be postponed or will be converted into an on-line meeting. Still, we are eager to press ahead with the
publication of the proceedings, so that these exciting papers can receive the attention they deserve.
Collin Baker, for the Organizing Committee
iv

Organizers:
Tiago Timponi Torrent, FrameNet Brasil, Federal University of Juiz de Fora, Brazil
Collin Baker, FrameNet, International Computer Science Institute, USA
Oliver Czulo, Department of Translation Studies, Universität Leipzig, Germany
Kyoko Ohara, Japanese FrameNet, Keio University, Japan
Miriam R. L. Petruck, FrameNet, International Computer Science Institute, USA

Program Committee:
Hans Boas, University of Texas, Austin
Dana Dannells, Gothenburg University
Gerard de Melo, Rutgers University
Ellen Dodge, Google Inc.
Jerome Feldman, International Computer Science Institute
Maucha Gamonal, University of Leipzig
Laurent Gautier, University of Bourgogne
Normunds Gruzitis, University of Latvia
Karin Friberg Heppin, Apple Inc.
Richard Johansson, Gothenburg University
Dimitrios Kokkinakis, Gothenburg University
Marie-Claude L’Homme, University of Montréal
Russell Lee-Goldman, Google Inc.
Ely Matos, Federal University of Juiz de Fora
Sebastian Padó, University of Stuttgart
Michael Roth, University of Stuttgart
Josef Ruppenhofer, Institute for the German Language (IDS)Nathan Schneider, Georgetown University
Simon Varga, University of Mainz
Alexander Ziem, Heinrich-Heine University Düsseldorfv

Table of Contents
Beyond Lexical Semantics: Notes on Pragmatic Frames
Oliver Czulo, Alexander Ziem and Tiago Timponi Torrent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Finding Corresponding Constructions in English and Japanese in a TED Talk Parallel Corpus using
Frames-and-Constructions Analysis
Kyoko Ohara . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Towards Reference-Aware FrameNet Annotation
Levi Remijnse and Gosse Minnema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Frame-Based Annotation of Multimodal Corpora: Tracking (A)Synchronies in Meaning Construction
Frederico Belcavello, Marcelo Viridiano, Alexandre Diniz da Costa, Ely Matos and Tiago Timponi
Torrent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Combining Conceptual and Referential Annotation to Study Variation in Framing
Marten Postma, Levi Remijnse, Filip Ilievski, Antske Fokkens, Sam Titarsolej and Piek Vossen 31
FrameNet Annotations Alignment using Attention-based Machine Translation
Gabriel Marzinotto . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Greek within the Global FrameNet Initiative: Challenges and Conclusions so far
Voula Giouli, Vera Pilitsidou and Hephaestion Christopoulos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Using Verb Frames for Text Difficulty Assessment
John Lee, Meichun Liu and Tianyuan Cai . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Deriving a PropBank Corpus from Parallel FrameNet and UD Corpora‘
Normunds Gruzitis, Roberts Dargis,
Laura Rituma, Gunta Nešpore-Bērzkalne and Baiba Saulite63
Building the Emirati Arabic FrameNet
Andrew Gargett and Tommi Leung . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Exploring Crosslinguistic Frame Alignment
Collin Baker and Arthur Lorenzi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Building Multilingual Specialized Resources Based on FrameNet: Application to the Field of the Environment
Marie-Claude L’ Homme, Benoît Robichaud and Carlos Subirats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85viConference Program
Saturday, May 16, 2020
09:00–09:20

Opening Session

09:00–09:20

Welcome and introduction to the workshop
Collin F. Baker

09:20–11:00

Oral Session 1: Theoretical Papers

09:20–09:45

Beyond Lexical Semantics: Notes on Pragmatic Frames
Oliver Czulo, Alexander Ziem and Tiago Timponi Torrent

09:45–10:10

Finding Corresponding Constructions in English and Japanese in a TED Talk Parallel Corpus using Frames-and-Constructions Analysis
Kyoko Ohara

10:10–10:35

Towards Reference-Aware FrameNet Annotation
Levi Remijnse and Gosse Minnema

10:35–11:00

Frame-Based Annotation of Multimodal Corpora: Tracking (A)Synchronies in
Meaning Construction
Frederico Belcavello, Marcelo Viridiano, Alexandre Diniz da Costa, Ely Matos andTiago Timponi Torrent

11:00–11:30

Coffee Breakvii

Saturday, May 16, 2020 (continued)
11:30–13:00

Poster Session

11:30–13:00

Combining Conceptual and Referential Annotation to Study Variation in Framing
Marten Postma, Levi Remijnse, Filip Ilievski, Antske Fokkens, Sam Titarsolej andPiek Vossen

11:30–13:00

FrameNet Annotations Alignment using Attention-based Machine Translation
Gabriel Marzinotto

11:30–13:00

Greek within the Global FrameNet Initiative: Challenges and Conclusions so far
Voula Giouli, Vera Pilitsidou and Hephaestion Christopoulos

11:30–13:00

Using Verb Frames for Text Difficulty Assessment
John Lee, Meichun Liu and Tianyuan Cai

11:30–13:00

Deriving a PropBank Corpus from Parallel FrameNet and UD Corpora‘
Normunds Gruzitis, Roberts Dargis,
Laura Rituma, Gunta Nešpore-Bērzkalne andBaiba Saulite

11:30–13:00

Building the Emirati Arabic FrameNet
Andrew Gargett and Tommi Leung

13:00–14:00

Lunch Breakviii

Saturday, May 16, 2020 (continued)
14:00–14:50

Oral Session 2: Practical Papers

14:00–14:25

Exploring Crosslinguistic Frame Alignment
Collin Baker and Arthur Lorenzi

14:25–14:50

Building Multilingual Specialized Resources Based on FrameNet: Application to
the Field of the Environment
Marie-Claude L’ Homme, Benoît Robichaud and Carlos Subirats

14:50–15:50

Group Discussion Session

14:50–15:50

Group Discussion Session 1: Fostering Collaboration in Global FrameNet
Tiago Torrent

14:50–15:50

Group Discussion Session 2: Alignment Issues in FrameNet
Collin F. Baker

15:50–16:10

Coffee Break

16:10–17:00

Closing Session

16:10–17:00

Planning MeetingixProceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 1–7
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Beyond lexical semantics: notes on pragmatic frames
Oliver Czulo, Alexander Ziem, Tiago Timponi Torrent
Universität Leipzig, Heinrich-Heine Universität Düsseldorf, Federal University of Juiz de Foraczulo@uni-leipzig.de​, ​ziem@phil.hhu.de​, ​tiago.torrent@ufjf.edu.br
Abstract

FrameNets as an incarnation of frame semantics have been set up to deal with lexicographic issues (cf. Fillmore and Baker 2010,
among others). They are thus concerned with lexical units (LUs) and conceptual structures which categorize these together. These
lexically-evoked frames, however, generally do not reflect pragmatic properties of constructions (LUs and other types of non-lexicalconstructions), such as expressing illocutions or establishing relations between speaker and hearer. From the viewpoint of amultilingual annotation effort, the Global FrameNet Shared Annotation Task, we discuss two phenomena, greetings and tag questions,highlighting the necessity both to investigate the role between construction and frame annotation and to develop pragmatic frames (andconstructions) related to different facets of social interaction and situation-bound usage restrictions that are not explicitly lexicalized.
Keywords:​ frame semantics, construction grammar, pragmatic frames, tag questions, greetings, pragmatics, multilinguality

1.

standards and applications of FrameNet data. The sharedannotation task is devised such that comparable texts ororiginals with their translations from different genres may
be annotated in multiple languages. The comparison
between these annotations should highlight various
differences such as between the conceptual structures offrames in two languages or the structure of certain parts ofa framenet (e. g. the types of relations between a set offrames). Moreover, the resulting data can be used inapplications, such as designing machine translation
metrics (Czulo et al. 2019) and multilingual annotation
projection3. Nonetheless, the kinds of analyses andapplications that can be derived from the data in theshared annotation task are restricted to the existence ofsome lexically-specified material evoking a given frame.

Introduction

The past two decades have witnessed the development offramenets1 for several languages (Borin et al. 2010;
Burchardt et al. 2006; Ohara et al. 2004; Subirats
Rüggeberg and Petruck 2003; Torrent and Ellsworth
2013, and others). Relying more or less on the original
Berkeley FrameNet infrastructure and data (Baker andSato 2003), those initiatives have built independent
resources whose alignment is currently being pursued
under the Multilingual FrameNet Project (Gilardi and
Baker 2018). Because the resulting resources are very
diverse in nature, in 2016, during the International
FrameNet Workshop held in Juiz de Fora, representativesof various framenet projects involved in the multilingualalignment initiative agreed on engaging in a sharedannotation task so as to assess the complexity of thedifferences found between each language-specificresource.

In this paper, building on this restriction, we make thecase for the oftentimes neglected pragmatic nature ofmany frames. Particularly, we analyse greetings and tag
questions as instances of multi-word expressions evoking
a frame. We show that these units do not refer to entity-,
state-, attribute-, relation-, or event-related frames, butrather bear pragmatic value. Through the comparison of
examples from English, Brazilian Portuguese andGerman, we illustrate why the annotation of pragmatic
properties is informative on a cross-linguistic level:
Framesandconstructions associated with aconventionalized pragmatic meaning do not need tocoincide in form, but may be linked to each other throughtheir pragmatic meaning.

In the following, we address one of these attested
differences, namely linguistically encoded pragmatic
information. Just like in constructionist analyses (Cappelle
2017), frame-based approaches to linguistic meaning tend
to neglect conventionalized pragmatic properties (as an
exception, see Blyth and Koike 2014). However, as we
will argue, they are essential to a plethora of frames andconstructions.
The starting point of our investigation are results obtainedin the Global FrameNet2 Shared Annotation Task (Torrent
et al. 2018) in which an original text and its translations
into other languages are all annotated for the frames inBerkeley FrameNet Data Release 1.7. The annotation
teams were allowed to create Lexical Units (LUs) for theirlanguages, but not to change, or create new, frames.
Hence, mismatches between frames and the LUs
associated with them are parameterized in the system forfurther analysis. Among the relevant questions are thecomparability and alignment of frames, annotation

We begin by explaining why pragmatic frames should be
addressed and included in any framenet and constructicon
approach (Section 2). We do this by example of tag
questions in English which highlights that pragmatic
frames need not be linked to specific lexemes. In Section
3, we present examples on greetings and tag questions
from the Shared Annotation Task in English, Brazilian
Portuguese and German, discussing formal differences
and how these could be bridged by introducing pragmatic
frames in framenet-like annotation efforts. In Section 4,
we sketch out basic aspects (potentially) included in thedescription of pragmatic frames, assuming that theseshared aspects (in terms of family resemblances) motivate

1

In the following, we use capital letters (“FrameNet”) for
referring to specific projects like, for example, to Berkeley
FrameNet; in contrast, “framenet” is reserved for cases where noreference to a specific project is made.
2​https://www.globalframenet.org/

3

1

See http://www.ufjf.br/framenetbr-eng/summer-of-code/pragmatic frames as a separate group of frames. We closewith some suggestions as to the annotation and further
empirical investigations on pragmatic frames.

2.

In order to describe the frame-evoking potential of tag
questions, single-word LUs cannot function asframe-evoking elements because they alone are not able toaccount for the meaning of the constructions they occur in(see Torrent et al. 2014 for a discussion of criteria toassign frame meaning to lexical items versus
constructions). At first glance, there seems to be a rather
simple solution to take account of tag questions within theconstruction-and-frame approach presented here. The
frame-evoking power of the constructions cannot be
traced back to single lexical elements but must rather be
assigned to the phrase as a whole. In other words, tag
questions are multi-word expressions that evoke frames ina holistic rather than a compositional fashion, in that thebuilding blocks of the expressions cannot be consideredunits carrying frame-semantic information on their own.

Why pragmatic frames? The case of tag
questions in English

The notion of pragmatic frame - including phenomena
sometimes also subsumed under the notion of“interactional” or “interactive frame” (Blyth and Koike
2014) - goes back to Goffman's early work on verbal
interactions, building on what he has called “interactionalframes” (Goffman 1961, 1967, 1974). Moreover,Fillmore's (1977, 1985) early terminological division
between scenes and frames - only the latter being evokedby linguistic material - also made room for the existenceof frames whose nature is different from that captured bya lexicographic analysis (for a discussion of Fillmore’s
notion of scene see Ziem 2014a, pp. 188–195).

Tag questions are multi-word expressions because they do
not feature schematic CEs that can be filled in a
productive or even semi-productive way (see Clausner
and Croft 1997 for different degrees of productivity).
Instead, different languages may provide specificinventories of tag questions, very much in parallel tosubstantive idioms (in the sense of Fillmore et al. 1988, p.
505f.) that take a variety of forms. This inventory must betreated just like LUs, that is, one-word units that evoke aframe. In this view, tag questions are LUs in that theyevoke a frame in their own right.

Therefore, before diving into the kind of phenomena
whose analyses motivated the notes in this paper, we must
point out that the case for pragmatic frames presented here
is restricted in two ways. First, it should not be interpreted
as a claim towards the revision of any theory, neither
Frame Semantics (Fillmore 1982) nor Frame Analysis(Goffman 1974), since both of them already had room forthis type of frames in their frameworks. Second, it should
not be taken as embracing every aspect of what falls under
the umbrella of pragmatics in linguistics. Most
importantly, following the Fillmore tradition, our notionof “pragmatic frames” crucially differs from the one
introduced by Rohlfing and colleagues (2016) in that it
only relates to pragmatic information conventionallyattached to linguistic expressions and ​not ​to ad hoc
inferences or framing activities in individual
communicative experiences. Also, we do not intend toaddress frames as linguistic ‘devices’ that help explain
linguistic phenomena of any kind, such as resolutions of(associative) anaphora (Ziem 2014b). Instead, our main
point is that, in a framenet-like annotation setting, such asthat provided by the Global FrameNet Shared Annotation
Task or the one derived from constructicon-building
efforts (Benjamin Lyngfelt et al. 2018), we should also
take account of conventionalized pragmatic affordancesand requirements that can be associated to the text units
being annotated.

However, it is anything but trivial to specify the frames
evoked. In contrast to fully, or partially, schematicconstructions, they cannot be said to evoke frames thatmirror, at least partially, the valence of a lexical item thatmight or might not be part of the construct. Rather, it
seems that they evoke frames that do not even consist offrame elements (FEs), usually defined as semantic roles
that abstract away from the specific semantics of
instances. Following Fillmore’s proposal (1982, p. 117), it
seems as if they do not evoke linguistic but interactionalframes. Tag questions, in this regard, are not only
substantive idioms but also what Fillmore, Kay andO’Connor (1988, p. 506) call idioms with a pragmatic
point. However, the concept of interactional frame is far
from being well-explored. From the standpoint of aconstruction-and-frame analysis, Ohara (2018) points outthat a lot of questions still remain unanswered; it is neither
clear what interactional frames really are nor, morecrucially, how a FrameNet approach may address them; asa matter of fact, so far interactional frames are almost
completely missing in FrameNet, the exceptions including
the ​Attention_getting frame, as pointed out by
Ohara (2018, p. 158). Therefore, our analysis also remains
somewhat incomplete. However, aside from Japanese
FrameNet, other initiatives, such as the German
4
FrameNet, consider interactional frames highly relevant .
Moreover, the Global FrameNet Shared Annotation Task
has created an opportunity to the discussion of suchframes in a multilingual setting, as we discuss next.

One example of such associations can be found in tag
questions. Tag questions take the form of short questions
mostly attached to a main declarative clause, such as in“You’re coming to dinner with us, aren’t you?”. They do
not necessarily require a response; in terms of Searle’s
speech act theory (1969), their illocutionary force lies
elsewhere. Tag questions are said to be multi-functional,
depending on the context they may serve as signals foremphasising, hedging, reinsurance, maintenance ofintersubjectivity, among others (Columbus 2010; König
2017). Tag questions thus do not denote an entity (ofwhatever kind), they rather fulfill an interactive function
in a communication between at least two parties.

42

For more details, see www.german-framenet.de, last accessed:
October 1, 2019.

3.

Pragmatic frames in the Shared
Annotation Task

(1)

a. Good morning.
b. Bom dia.

The first text to currently be jointly annotated in the
Global FrameNet Shared Annotation Task is the mostviewed TED talk, given by Sir Ken Robinson with thetitle “Do schools kill creativity” (2006) which at the timeof writing has been viewed more than 64 million times.
The close captioning of the 20-minute talk in English
contains 267 sentences. These subtitles have beentranslated to 63 languages by TED community members.

In the annotation effort, the English (1a) and Brazilian
Portuguese (1b) variants remain unannotated. It is only inthe case of the German annotation, that the greeting wasannotated with frames, given in (2) :For the annotation, members of the English, Brazilian
Portuguese and German annotation teams worked with the
1.7 release of Berkeley FrameNet. While this raises anumber of questions as to cross-linguistic applicability offrames and framenets, this decision greatly facilitated thecomparison of the annotations. The first thirty annotatedand aligned sentences were fed into an evaluation system
which is designed to automatically compute a similarity
measure between a pair of sentences for machinetranslation evaluation purposes, based on semantic frames(Czulo et al. 2019).

Obviously, however, the frame annotation of the lexemes
in itself with the frames ​Desirability andCalendric_unit does not point to (2) being agreeting. The annotation in (2) was originally provided
with the goal of presenting an analysis of the semantic
part of the constructional pattern licensing a greeting,much like proposed in (Fillmore et al. 2012). Note,however, that such an approach to (2) does not take us far
either; the greeting formula instantiated in (2) does notallow for substantial internal variation, though
premodification is possible in specific registers (e.g.,
Schönen guten Morgen! ​‘Top of the morning!’, lit.
“Beautiful good morning). Thus, it seems to be more
adequate to treat ​Guten Morgen as multi-word unit
(MWU; Ruppenhofer et al. 2016, p. 21), similar to other
MWUs already included in Berkeley FrameNet, such as
good idea.n evoking the ​Desirable_event frame in
expressions like ​X is a good idea​.

(2)

What stood out in the analysis were a few sentences
which, contrary to our intuition, would show a lack ofsimilarity. Most notably this concerned the first two
sentences between all three languages, with no frame
overlap and no similarity at all. The reason was quickly
identified: the first two sentences contained formulae ofgreetings which had not been annotated at all for English
and Brazilian Portuguese, but with lexical frames inGerman (see Section 3.1). Still, it was clear that theGerman annotation was not an adequate description of thesentences, as it did not reflect the role of the expression in
the sentence of being a greeting formula. Up to now, theFrameNet database does not include a ​Greeting frame,or any other frame that matches the pragmatic value of theunit under consideration.

].

_​UNIT​

ALENDRIC​

Thus, pragmatic frames substantially differ from semantic
frames in that they define situational settings as usage
parameters; these settings resemble much more
non-linguistic “scenes” (Fillmore 1985) attached to the
MWU used than valency-based frames as incorporated inBerkeley FrameNet and the lexicographically oriented
initiatives for other languages that derived from it.
Looking at examples (1) and (2), we can state that theinternal frame structure for the greeting at hand is thesame in all three languages. As a different case, let us lookat a construction which is more variable across languages
and allows different interpretations in (3):Greetings5

The TED talk does not dive right into the topic but begins
with some remarks by the speaker the function of which
can at this point roughly be characterized as establishing aconnection with the audience of the talk. This isspearheaded by a greeting, with the English and Brazilian
Portuguese versions as in (1).

(3)

5

For a more detailed analysis see also (Czulo et al. submitted),on which some of the following notes are based.

] [Morgen​C​
morning

ESIRABILITY​

What an adequate semantic-pragmatic representation of
the MWUs in (1-2) should include is the meaning of theexpressive speech act, that is, the fact that it's a greeting.
This situation type includes (a) the involvement of a
speaker interacting with an addressee by means of the
MWU, (b) time specification of the uttered MWU(roughly: before noon), and (c), by default, thesupposition that it is the first encounter of speaker andaddressee in a given time span. These specifications comewith the frame evoked without materializing themselves
as frame elements that are instantiated by parts of the
MWU or the other linguistic material surrounding it.

Similarly, sentences with tag questions lead the evaluation
system to calculate larger differences than anticipated, as
they had been annotated again in German, but not in theother two languages. Especially with regard to those,
however, Section 3.2 explains why annotating pragmatic
frames is desirable and informative here: Tag questions
vary in form between these languages, even as regards thelexical material used to form them, but in terms of anevaluation, they should be seen as (pragmatically)equivalent expressions. If pragmatic frames existed for tag
questions, an automatic evaluation measure would benefit
from more precisely assessing the similarity of two
sentences in two languages.

3.1[Guten​D​Good

3

a. How are you?
b. Como estão?
How be.3.PERS
c. Wie geht es Ihnen?
How goit you.3.PERS.DATquestions. LUs such as ​have.v and ​ter.v cannot be said to
evoke a frame that refers to any kind of ‘assurance’ thatthe speaker wants to express. Moreover, the Brazilian
construction can also feature the ver ​ser ​('be') in the tag,instead of repeating the auxiliary used in the main clause.
Even though the German LU ​wahr.a (‘true’) points to thisdirection, it still fails to trigger a specific frame thatprovides the pragmatically relevant information required.

While the English and Brazilian Portuguese versions are
only superficially different (with a pro-drop in theBrazilian version), the German version uses a different
verbal lexeme: ​gehen ‘go’6. When taken literally, thiswould evoke a different frame than the verb ​be in English
or Brazilian Portuguese, but this annotation of a ‘literaldefault’ meaning would not reflect the pragmatics behindthe formula. The annotation of an interactional frame
would indicate that there is a greeting with the function to
open an interaction. Moreover, if interpreted literally in
the given situation and answered with something like
Terrible, my dog just died,​ in this kind of interactional
scenario, (3a-c) would not be successful.

Another case in point relates to the tag questions
exemplified in (5).
(5)

As Bunt and colleagues (2010, p. 2549) point out, though,assigning a singular function to a linguistic item could run“into the problem that the same linguistic form can often
be used to express different communicative functions.” In
the case of greetings like ​How are you​, a literalinterpretation besides the function of opening aninteraction can easily be imagined, such as a doctor
greeting a patient, where the patient is actually expected
to answer according to the question. Still, primarily
pragmatic items such as greetings may be among theeasiest to detect in texts for purposes of annotation,
notwithstanding other questions of interpretation.
Moreover, as it is already the case for polysemous
lexemes in any FrameNet, expressions such as ​How are
you could be easily associate to different sets of frames pragmatic or not.

Tag questions

In the annotated sample text, we find three instances oftag questions. (4) exemplifies one of them.
(4)

a.
b.

c.

It's been great, hasn't it?
Tem
sido
have.PRES.3SG
be.PART
não tem?
no have.PRES.3SG
Es wargroßartig,
It be.PST.3SG great
nicht wahr?
not trueótimo,
great

4.

Suggestions for a basic structure ofpragmatic frames

Conventionalized pragmatic frames are viewed here as
situation- or genre-bound concepts specified by a set of
conditions defining adequate uses at peculiar occasions.
These frames, like greetings and tag questions, helporganise interaction between two or more parties rather
than relating to conceptual structures representing objects,
attributes, relations, states or events. These parties can be
individuals or groups with members being humans or
ascribed human-likeness (real or imagined entities such as
AI machines, fairies, aliens etc.).

The tag questions in English and Brazilian Portuguese,
illustrated in (4a-b), feature finite forms of the auxiliaries
have and ​ter (‘have’), respectively. In contrast, theGerman tag question is realized in the form of a negatedadjective ​nicht wahr (lit. ‘not true’). It is obvious that aframe-based annotation of the respective LUs isinsufficient to capture the pragmatic function of the tag

Presumably, this commonality of pragmatic frames andthe conceptual systematicities resulting therefrom groups
these frames in terms of family resemblances (forsuggestions on operationalisation see Ziem 2014a, pp.

6

Brazilian Portuguese also admits a variant of this construction
with the verb ​ir​ 'go'.

I mean, Sirena last night was a marvel,
wasn't she?
b.
Sirenaontem
a noite foi
Sirenayesterday at nightbe.PST.3SG
uma maravilha, não foi?
one wonder
no be.PST.3SG
c.
Sirena gestern Abend war
Sirena yesterday evening be.PST.3SG
wunderbar, nichtwahr?
marvellous not true

Again, English and Brazilian Portuguese include theauxiliaries ​be.v and ​ser.v (‘to be’) whereas the German tag
question is equivalent to the one introduced in (4). Just
like the auxiliaries in (4), the verbs in the English andBrazilian Portuguese instances cannot be said to evoke aframe that point to the pragmatics of tag questions(Columbus 2010). To be successful, we need an integrated
frame-and-construction approach that also accounts forboth the clausal form (sub-aux inverted clause, whereapplicable) underlying tag questions andcontext
requirements to be met. The latter includes not only (a)the existence of a pre-established referent to which thepersonal pronoun anaphorically refers, be it a person(“haven’t ​you”​ ) or an entity or any kind of propositionallyexpressed state of affair (“hasn’t it”); (b) also, the tense of
the verb included in a tag question must be consistent
with the context, more precisely with previous uses oftenses. In the next section, we provide a proposal as tohow such an approach may be implemented in practicalterms.

An answer of how such interpretations could be modelled
can be found in the framework of Systemic-Functional
Linguistics (Halliday 1973; Halliday and Matthiessen2004). Distinctions such as between the ideational and theinterpersonal function could help model which kind offunction is in the foreground and should be guiding theinteraction, which types of interactions are expected etc.

3.2

a.

4

297–299). Having at this point looked at only two types ofpragmatic frames, especially with regard to the variety of
meanings that tag questions can express, we expect that
central aspects of pragmatic frames may relate to, but are
not limited to,●●●●

The practical question arising from this is when to
annotate which frame and what these frames should be.
The examples discussed here may leave little room forinterpretation, but in cases e. g. in which two variants ofan expression exist which reflect different levels offormality, the choice may not be straightforward.
Annotation will probably depend on the question whether,and to what extent, the lexical-semantic or pragmatic
meaning is in the focus of the current research interest. As
for the questions of what types of frames we may need,previous work by Bunt and colleagues (2010) on an ISO
standard for annotating interaction types in dialogues may
be a good starting point. In their proposed taxonomy offunctions,
they
first
distinguishbetweeninformation-transfer functions and action-discussionfunctions, then specifying various types of requests,
suggestions, denials etc. In combination withsystemic-functional aspects, the taxonomy could be
extended to include further interactional aspects and bereflected in the pragmatic sub-group of frames inframenets in different languages accordingly.

circumstances such as time, in-/formal type ofthe occasion licensing the use of the target
expression,situational presuppositions, including artefacts,
such as materials and objects (e.g. water forbaptizing), recipient/audience addressed,
text- and sociolinguistic affordances specific to
the type of communication, including, forexample, choice of register and text genre,
further sociolinguistically relevant factors
including diatopic and diastratic variation, roles
and statuses of the parties involved and how they
may evolve during communication, relations toparties outside of the communication situation athand, face-saving actions of parties involved.

Importantly, pragmatic information of this kind needs to
be integrated in a frame description, regardless of the factthat they are not part of the target valence frame since
they do not necessarily instantiate semantic roles (FEs).
Thus, pragmatic frames differ substantially from semantic
frames in that they relate to conditions of appropriate use
in specific situational settings. While we expect that theconfiguration of pragmatic frames vastly differs between
7
cultures , or even between communities within a culture,
pragmatic frames should, due to their common core
structure, be more comparable between each other evenacross languages than many object-, state-, attribute,
relation- or event-related frames.

5.

In this paper, we made the case for considering pragmatic
frames as important components of any frame-based
repository such as FrameNet. One reason for this is that
pragmatic aspects - just like semantic roles - may wellbelong to the conventionalized content of linguistic signs.
This forces us to extend the description of frames in sucha way that it includes not only a well-defined
configuration of semantic roles (FEs) but also conditions
for using the frame-evoking elements adequately. It still
remains an open issue, however, in which way pragmatic
information can, and should, be built into a frame
definition. One option is to introduce a new category“pragmatic roles”; yet, it is anything but clear how to
consistently define such roles in parallel to thewell-established notion of semantic roles. Another option
is to specify usage requirements in the prose part of aframe definition. Yet another option is to enrich‘traditional’ frame-semantic descriptions by pragmatic
templates as introduced by Liedtke (2013, 2018).

We also assume that in one way or another, pragmatic
frames are always at work during communication, though
not necessarily evoked by linguistic items. Most
prominently, Goffman (1974) argued that humaninteraction is in general framed by its embedding
situational setting. Such frames, however, seem to befundamentally different from current FrameNet frames. In
terms of frame semantic theory, this is not problematic inprinciple, as in the early version, Fillmore already pointed
out that frames (or, in the old terminology, ‘scenes’) arelinked and co-activate each other e. g. “by virtue of [...]
their contexts of occurrence” (Fillmore 1975, p. 124); i. e.
unlike often practiced in annotation, there is no reason tobelieve that a linguistic expression or any otherframe-evoking material or circumstance necessarily evoke
one and only one frame. As pointed out above in 3.1,
Systemic-Functional Linguistic could provide aframework to model how semantic and pragmatic aspects
of interactions are composed and thus interact with eachother. Currently, however, we miss an integrative
approach, combining FrameNet frames with a moregeneral situation-bound and context-sensitive frame
theory that addresses semantic, pragmatic andinteractional properties of communication on a par.

Not surprisingly, very similar issues arise with reference
to construction entries (Cappelle 2017; Finkbeiner 2019).
Beyond the pragmatic frames addressed here, there isassumingly a huge variety of other types of both
pragmatic frames and pragmatic constructions peculiar toa language. Even though these units challenge standard
frame-semantic and constructionist approaches in several
ways, we have no general reservation about the
integration of these units into the type of constructiconthat we have in mind (Ziem and Flick 2019). Without
doubt, however, it is an empirically challenging task toidentify and describe pragmatic frames and constructions
in a comprehensive way.
As a first guess, we consider the following examples,
among others, as good candidates for LU or MWU
evoking pragmatic frames or construction: Greeting andleave-taking expressions (​Good morning​, ​Dear X,​
Goodbye,​ ​Kind regards​); performative verbs and7

We use the term with all reservations as to what an ‘exact’
definition of ‘culture’ could be.

Conclusions: pragmatic frames andconstructions in the constructicon5expressions (​I baptize you​); deictic and multimodal
constructions (​so+​ ADJ+gesture; Ziem 2017); expressionsof preference (​von wegen ‘No way!’ ​); implicatures; text
genre-specific constructions (e.g., pro drop in recipes);
information structure (​it was the girl who...​). Taking
phenomena like this as a benchmark (also for forthcoming
efforts in the Global FrameNet Shared Annotation Task),
we consider it worthwhile to gradually develop a robust
and sophisticated concept of “pragmatic frame” on an
empirical basis.

6.

(pp. 55–88). North Holland Publishing.
Fillmore, C. J. (1982). Frame Semantics. In The
Linguistic Society of Korea (Ed.), ​Linguistics in
the Morning Calm. Selected Papers of
SICOL-1981​ (pp. 111–137). Seoul: Hanshin.
Fillmore, C. J. (1985). Frames and the semantics ofunderstanding. ​Quaderni di Semantica​, ​6​(2),
222–254.
Fillmore, C. J., & Baker, C. F. (2010). A Frames
Approach to Semantic Analysis. In B. Heine &H. Narrog (Eds.), ​The Oxford Handbook of
Linguistic Analysis (pp. 313–339). Oxford:
Oxford University Press.
Fillmore, C. J., Kay, P., & O’Connor, M. C. (1988).
Regularity and Idiomaticity in Grammatical
Constructions: The Case of “Let Alone.”
Language,​ ​64(​ 3), 501–538.
Finkbeiner, R. (2019). Reflections on the role ofpragmatics
inConstruction
Grammar.
Constructions and Frames,​ ​11(​ 2), 171–192.
https://doi.org/10.1075/cf.00027.fin
Goffman, E. (1961). ​Encounters: Two studies in thesociologyofinteraction​.
Indianapolis:
Bobbys-Merrils.
Goffman, E. (1967). ​Interaction rituals: Essays on face toface behavior​. New York: Anchor.
Goffman, E. (1974). ​Frame analysis: An essay on theorganization of experience.​ Cambridge, MA,
USA: Harvard University Press.
Halliday, M. A. K. (1973). ​Explorations in the functionsof language.​ London: Arnold.
Halliday, M. A. K., & Matthiessen, C. M. I. M. (2004). ​An
introduction to functional grammar (3 (revised
by C.M.I.M. Matthiessen).). London: Arnold.
König, K. (2017). Question tags als Diskursmarker?
Ansätze zu einer systematischen Beschreibung
von “ne” im gesprochenen Deutsch. In H.
Blühdorn, A. Deppermann, H. Helmer, & T.
Spranz-Fogasy (Eds.), ​Diskursmarker im
Deutschen: Reflexionen und Analysen (pp.
233–258).
Göttingen:
VerlagfürGesprächsforschung.
Liedtke, F. (2013). Pragmatic templates and freeenrichment. In F. Liedtke & C. Schulze (Eds.),
Beyond words: content, context, and inference(pp. 183–205). Berlin ; Boston: De Gruyter
Mouton, Walter De Gruyter GmbH.
Liedtke, F. (2018). Frames, Skripte und pragmatische
Templates. In M. Wengeler & A. Ziem (Eds.),
Diskurs,
Wissen, Sprache: Linguistische
Annäherungenankulturwissenschaftliche
Fragen (pp. 115–135). Berlin ; Boston: De
Gruyter.
Lyngfelt, Benjamin, Borin, L., Ohara, K., & Torrent, T. T.
(Eds.).
(2018).
​Constructicography:
Constructicon development across languages​.
John Benjamins.
Ohara, K. (2018). The relations between frames andconstructions: A proposal from the Japanese

Bibliographic References

Blyth, C., & Koike, D. (2014). Interactive frames andgrammatical constructions. In S. K. Bourns & L.
L. Myers (Eds.), ​Perspectives on Linguistic
Structure and Context: studies in honor of Knud
Lambrecht(pp.
87–108).
Amsterdam ;Philadelphia: John Benjamins Publishing
Company.
Bunt, H., Alexandersson, J., Carletta, J., Choe, J.-W.,
Chengyu Fang, A., Hasida, K., et al. (2010).
Towards an ISO standard for dialogue act
annotation.
In​Seventhconference on
International
Language
Resources
andEvaluation (LREC’10).​ La Valette, Malta.
https://hal.inria.fr/inria-00544997
Cappelle, B. (2017). What’s pragmatics doing outside
constructions? In I. Depraetere & R. Salkie(Eds.), ​Semantics and Pragmatics: Drawing a
Line
(pp.
115–151).
Cham:
SpringerInternational
Publishing.
https://doi.org/10.1007/978-3-319-32247-6_8
Clausner, T. C., & Croft, W. (1997). Productivity and
Schematicity in Metaphors. ​Cognitive Science​,
21​(3), 247–282.
Columbus, G. (2010). “Ah lovely stuff, eh?” - invariant
tag meanings and usage across three varieties ofEnglish. In S. Th. Gries, S. Wulff, & M. Davies(Eds.), ​Corpus-linguistic applications: current
studies, new directions (Vol. 71, pp. 82–105).
Brill|
Rodopi.
https://doi.org/10.1163/9789042028012_007
Czulo, O., Torrent, T. T., Matos, E., Diniz da Costa, A., &Kar, D. (2019). Designing a Frame-Semantic
Machine Translation Evaluation Metric. In
Proceedings of The Second Workshop on
Human-Informed Translation and InterpretingTechnology (HiT-IT 2019)​. Varna, Bulgaria.
Czulo, O., Torrent, T., Willich, A., Ziem, A., & Laviola,
A. (submitted). A multilingual approach to theinteraction between frames and constructions:
towards a joint framework and methodology.
Fillmore, C. J. (1975). An alternative to checklist theoriesof meaning. In ​Annual Meeting of the Berkeley
Linguistics Society​ (Vol. 1, pp. 123–131).
Fillmore, C. J. (1977). Scenes-and-frames semantics,
Linguistic Structures Processing. In A. Zampolli(Ed.), ​Fundamental studies in computer science6

FrameNet constructicon. In Benja Lyngfelt, L.
Borin, K. Ohara, & T. T. Torrent (Eds.),
Constructicography: Constructicon development
across languages (pp. 141–163). Amsterdam;
Philadelphia: Benjamins.
Rohlfing, K. J., Wrede, B., Vollmer, A.-L., & Oudeyer,
P.-Y. (2016). An alternative to mapping a word
onto a concept in language acquisition:
Pragmatic frames. ​Frontiers in Psychology​, ​7​,
1–18. https://doi.org/10.3389/fpsyg.2016.00470
Searle, J. R. (1969). ​Speech acts : an essay in thephilosophy of language​. Cambridge: Univ. Pr.
Torrent, T. T., Lage, L. M., Sampaio, T. F., da Silva
Tavares, T., & da Silva Matos, E. E. (2014).
Revisiting border conflicts between framenet andconstruction grammar: Annotation policies forthebrazilianportuguese
constructicon.
Constructions and Frames,​ ​6(​ 1), 34–51.
https://doi.org/10.1075/cf.6.1.03tor
Ziem, A. (2014a). ​Frames of understanding in text anddiscourse:
Theoreticalfoundations
anddescriptive applications.​ (C. Schwerin, Trans.).
Amsterdam ; Philadelphia: John Benjamins
Publishing Company.
Ziem, A. (2014b). Frames and constructions enhance text
coherence: The case of DNI resolutions in
spoken discourse. ​Yearbook of the German
Cognitive Linguistics Association,​ ​2(​ 1), 3–20.
Ziem, A. (2017). Do we really need a Multimodal
Construction Grammar? ​Linguistics Vanguard:
Special issue “Towards a multimodal
construction
grammar,”3​(1),
1–9.
https://doi.org/10.1515/lingvan-2016-0095

7.

Ohara, K., Fujii, S., Ohori, T., Suzuki, R., Saito, H., &Ishizaki, S. (2004). The Japanese FrameNet
project: an introduction. In ​Proceedings of thesatellite workshop “Building lexical resourcesfrom semantically annotated corpora” (pp.
9–11). Presented at the Fourth international
conference on Language Resources and
Evaluation, European Language Resources
Association.
Robinson, K. (2006). ​Do schools kill creativity? Presented
at the TED - ideas worth spreading.
https://www.ted.com/talks/sir_ken_robinson_do_
schools_kill_creativity. Accessed 20 February
2020
Ruppenhofer, J., Ellsworth, M., Petruck, M. R. L.,
Johnson, C. R., Baker, C. F., & Scheffczyk, J.
(2016). FrameNet II: Extended Theory andPractice.
https://framenet2.icsi.berkeley.edu/docs/r1.7/book.pdf
Subirats Rüggeberg, C., & Petruck, M. (2003). Surprise:
Spanish FrameNet! In E. Hajičová, A.
Kotéšovcová, & J. Mirovský (Eds.), ​Proceedingsof the workshop on Frame semantics​. Presented
at the XVII International Congress of Linguists,
Prague: Matfyzpress.
Torrent, T. T., & Ellsworth, M. (2013). Behind theLabels: Criteria for Defining Analytical
Categories in FrameNet Brasil. ​Veredas​, ​17,​
44–65.
Torrent, T. T., Ellsworth, M., Baker, C. F., & Matos, E.
(2018). The Multilingual FrameNet Shared
Annotation Task: a Preliminary Report. In T. T.
Torrent, L. Borin, & C. F. Baker (Eds.),
Proceedings of the International FrameNet
Workshop 2018: Multilingual Framenets andConstructicons.​
Paris, France: European
Language Resources Association (ELRA).
Ziem, A., & Flick, J. (2019). Constructicography at work:
implementation and application of the German
Constructicon. In ​Yearbook of the German
Cognitive Linguistics Association.​

Language Resource References

Baker, C. F., & Sato, H. (2003). The framenet data andsoftware. In ​Proceedings of the 41st Annual
Meeting on Association for Computational
Linguistics (Vol. 2, pp. 161–164). Association
for Computational Linguistics.
Borin, L., Dannélls, D., Forsberg, M., Gronostaj, M. T., &Kokkinakis, D. (2010). The past meets thepresent in Swedish FrameNet++. In ​14th
EURALEX international congress​ (pp. 269–281).
Burchardt, A., Erk, K., Frank, A., Kowalski, A., Pado, S.,& Pinkal, M. (2006). The SALSA corpus: a
German corpus resource for lexical semantics. In
Proceedings of LREC 2006 (pp. 969–974).
Genoa, Italy.
Fillmore, C. J., Lee-Goldman, R. R., & Rhomieux, R.
(2012). The FrameNet Constructicon. In H. C.
Boas & I. A. Sag (Eds.), ​Sign-Based
Construction Grammar. (pp. 309–372). Stanford:
CSLI Publications.
Gilardi, L., & Baker, C. F. (2018). Learning to alignacross languages: Toward multilingual framenet.
In ​Proceedings of the International FrameNet
Workshop​ (pp. 13–22).

Acknowledgements
Research presented in this paper is funded by
CAPES/PROBRAL and DAAD PPP Programs, under the
grant numbers 88887.144043/2017-00 and 57390800,
respectively.

7

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 8–12
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Finding Corresponding Constructions in English and Japanese
in a TED Talk Parallel Corpus using Frames-and-Constructions Analysis
Kyoko Ohara

Keio University/RIKEN
4-1-1 Hiyoshi, Kohoku-ku, Yokohama City 223-8521, Japanohara@hc.st.keio.ac.jp
Abstract

This paper reports on an effort to search for corresponding constructions in English and Japanese in a TED Talk parallel corpus, using
frames-and-constructions analysis (Ohara, 2019; Ohara and Okubo, 2020; cf. Czulo, 2013, 2017). The purpose of the paper is twofold: (1) to demonstrate the validity of frames-and-constructions analysis to search for corresponding constructions in typologicallyunrelated languages; and (2) to assess whether the “Do schools kill creativity?” TED Talk parallel corpus, annotated in various
languages for Multilingual FrameNet, is a good starting place for building a multilingual constructicon. The analysis showed thatsimilar to our previous findings involving texts in a Japanese to English bilingual children’s book, the TED Talk bilingual transcripts
include pairs of constructions that share similar pragmatic functions. While the TED Talk parallel corpus constitutes a good resource
for frame semantic annotation in multiple languages, it may not be the ideal place to start aligning constructions among typologicallyunrelated languages. Finally, this work shows that the proposed method, which focuses on heads of sentences, seems valid forsearching for corresponding constructions in transcripts of spoken data, as well as in written data of typologically-unrelated languages.

Keywords: Japanese FrameNet, pragmatic function, multilingual constsructicon

1.

constructions in transcripts of spoken data, as well as
in written data of typologically-unrelated languages.

Introduction

The organization of the rest of the paper is as follows.
Section 2 discusses background to the study. Section 3
presents the hypothesis, method, and the results of theanalysis. Section 4 addresses the functional mismatches in
the parallel corpus, the validity of the method, and theappropriateness of the corpus as a starting point foraligning constructions in a multilingual constructicon.
Finally, Section 5 provides a conclusion and prospects for
future work.

This paper reports on an effort to find corresponding
Japanese and English grammatical constructions in a TED
Talk parallel corpus, using the frames-and-constructions
analysis method proposed in Ohara (2019) and Ohara andOkubo (2020). The method focuses on heads of sentencesin language, where a head is defined as “the mostcontentful word that most closely denotes the samefunction as the phrase (or clause) as a whole (cf. Croft, In
Preparation: 417).” The purpose of the paper is two-fold:(1) to demonstrate the validity of frames-and-construction
analysis as a methodology to search for corresponding
constructions in a pair of typologically-unrelated
languages such as English and Japanese; and (2) to assess
whether the “Do schools kill creativity?” TED Talk
parallel corpus, whose sentences have been annotated inframe-semantic terms in various languages, including
English, Brazilian Portuguese, French, German, andJapanese for Multilingual FrameNet, is a good starting
place to align constructions for building a
multilingual/contrastive constructicon.

2.

Related Work

The frames-and-constructions analysis method describes
meanings and structures of sentences, focusing on thesemantic frames evoked by various linguistic expressions
in the sentences. It is grounded in the theories of Frame
Semantics and Construction Grammar (Fillmore andBaker, 2010; Fillmore, 2013). Czulo (2013, 2017,elsewhere) proposed this method as a translation model,
based on analyses of German and English parallel data1.
Those works hypothesized that ideally the semantic frameof the translation matches that of the original (the primacyof the frame hypothesis). However, often cases of frame
mismatches exist between pairs of source and target
sentences and Czulo (2013, 2017) argued that structural
divergence can be a cause for frame mismatch, in addition
to cultural, typological, and perspectival differences.
Czulo also observed that even when a frame mismatchexists because of structural divergence between source
and target sentences, the two sentences may share thesame pragmatic function. This observation led to thesuggestion that the function of a construction may takeprecedence over exact frame match.

Our analysis revealed the following:
・ There are indeed pairs of sentences that constituteinstances of corresponding constructions in English
and Japanese that share similar pragmatic functions in
the TED Talk bilingual transcripts, similar to ourprevious findings for texts in a Japanese–English
bilingual children’s book;・ While the TED Talk parallel corpus constitutes agood resource for frame semantic annotation, it may
not be the ideal place to start aligning constructions
across typologically-unrelated languages, likely as aresult of characteristics of the genre of subtitles;
・ The proposed frames-and-constructions analysis
method, an approach that focuses on heads ofsentences, seems valid to search for corresponding

1

Czulo uses the term “constructions-and-frames analysis” inCzulo (2013) but since frame comparison is a crucial step in thismethod (cf. Section 3.2), I will use the term “frames-andconstructions analysis” in this paper.

8Building on Czulo’s (2013) work, Ohara (2019) andOhara & Okubo (2020) examined whether frames-andconstructions analysis is a valid methodology to search forand align comparable constructions between Japanese andEnglish, a pair of typologically unrelated languages. That
work analyzed 674 pairs of Japanese and English
sentences in a bilingual children’s book. They identified
the semantic frames evoked by the heads of source andtarget sentences and found 483 pairs of frame mismatches.
Among them, 106 pairs exhibited structural divergences.
Among the 106 pairs of structurally divergent sentences,
55 pairs exhibited the same pragmatic functions (cf. Table
1, Section 3.3). In other words, the study foundcorresponding constructions in Japanese and English
based on pragmatic functions, even in cases of structural
divergence and frame mismatches. Those results
suggested the usefulness of frames-and-constructions
analysis for finding comparable constructions acrosstypologically unrelated languages such as Japanese andEnglish, where structural divergence is well-documented.

2.

3.

However, the study that Ohara (2019) and Ohara andOkubo (2020) reported is preliminary; and no study existsthat explored the validity of the method in analyzing
translation from English to Japanese, spoken language,
genres other than narratives, and anything other thanchildren’s language. Thus, this paper applies the method
to analyze English and Japanese sentences that appear in
the “Do schools kill creativity?” TED Talk parallel corpus.
Analysts already have annotated this corpus with semantic
frames and FEs in various languages for Multilingual
FrameNet.

3.

4.

3.3

Frames-and-Constructions Analysis of
TED Talk Parallel Transcripts

Table 1 summarizes the results of our analysis using thesteps described in Section 3.2. The table shows thenumbers of sentence pairs that exhibit frame
match/mismatch, structural divergence, and functional
match in the TED Talk parallel corpus, in comparison
with those in a bilingual children’s book Anpanman I (cf.
Section 2).

Hypotheses

Prior to the present analysis, we formed three hypotheses
about characteristics of the English and Japanese
sentences in the TED Talk parallel transcripts. First, theTED Talk “Do schools kill creativity?” is a presentation
aimed at persuading its audience. Thus, one hypothesis isthat the English original transcript would include many
constructions that exhibit pragmatic functions. Second,
the Japanese version is a translated version of the English
original transcript. Consequently, another hypothesis isthat the Japanese translation would contain constructions
that exhibit similar pragmatic functions as those in the
English original. Finally, a third hypothesis is thelikelihood of finding corresponding English and Japaneseconstructions sharing the same or similar pragmaticfunctions, in spite of also showing frame mismatch andstructural divergence.

3.2

1) Sentence Pairs
2) Frame Match in 1)2’) Frame Mismatch in 1)
3) Structural Divergence in 2’)
4) Functional Match in 3)

TED(E to J)
122
75
42
22
9

Anpanman(J to E)
674
191
483
106
55

Table 1: The numbers of frame match/mismatch,structural divergence, and functional match in TED
There was one sentence pair that ultimately evoke thesame set of semantic frames through frame integration. In
addition, there were four sentence pairs in which the twoframes evoked by the English and Japanese heads are
related via a FrameNet frame-to-frame relation (cf. Step 3
above). These are the reasons why the sum of the numberof frame match and that of frame mismatch does not equal
the total number of sentence pairs in the TED Talk
parallel corpus.

MethodThe actual adopted steps of the frames-and-constructions
analysis in this study appear below. The analysis
concentrated on sentence-level grammatical constructions.
1.

Results

We examined 242 English original sentences from the
TED Talk. Sometimes one English sentence wastranslated into Japanese with more than one sentence; atother times, multiple English sentences were translated
into one Japanese sentence. We concentrated on analyzing
sentence pairs in which the English original sentence ismore or less straightforwardly translated into Japanesewith one sentence. There were 122 such sentence pairs.

This section is divided into three parts that describe thefollowing: (1) hypotheses formed prior to the present
analysis; (2) details about the proposed method; and (3)results of the analysis on the TED Talk bilingualtranscripts.

3.1

Identify the head of each of the English and Japanese
sentence pairs.
Frame Comparison:
Determine the semantic frames evoked by the headsof the English and Japanese sentence pairs; check forframe mismatch; exclude two kinds of cases fromframe mismatch. One kind has to do with cases inwhich a pair of English and Japanese sentences
ultimately evokes the same set of semantic frames
through frame integration (integration of frames
evoked by words and phrases in a sentence that
ultimately leads to an understanding of the wholesentence) within each sentence. The other kind
involves cases in which the two frames evoked by the
English and Japanese heads are related via any
FrameNet frame-to-frame relations (Ruppenhofer etal., 2016).
Structural Comparison:
Identify the structure of each of the English andJapanese sentences; check for English and Japanese
structural divergence.
Functional Comparison:
Identify the functions of the English and Japaneseconstructions.

Head Identification:

9Let us describe the results in relation to the threehypotheses in Section 3.1. Our first hypothesis was that
the English original transcript would contain many
constructions that exhibit pragmatic functions. Indeed, theEnglish version of the talk includes sentence structuresthat focus either the whole or parts of a sentence, such as
pseudo-cleft sentences (1), repetition (2), emphasis (3),and cataphora (4).

creativy TOP literacy ability COM same degreekyôiku ni‘Creativity is as necessary to education as literacy.’(4’)
a. (=(2’b))Pikaso wa katsute kô
Picaso TOP once

(1) Pseudo-cleft:a.

children TOP all

if

“kô

GEN artist COP

alien NOM education site

kyôikutte

nan noLOC come COND

tame niaru no?”public education CONJ what NOM purpose DAT exist Q

What we do know is, if you're not prepared to be
wrong, you'll never come up with anything original -if you're not prepared to be wrong. (#77)
Picasso once said this, he said that all children areborn artists.(#84)

b.

born

‘Picasso once said like this, “children are all born
artists.”’b. moshi eirian ga kyôiku genba ni yatteki tara(2) Repetition:a.

îmashita

like.this said

“kodomo wa mina umarenagara no âtisuto da”

Actually, what I find is everybody has an interest in
education. (#13)
What we do know is, if you're not prepared to be
wrong, you'll never come up with anything original -if you're not prepared to be wrong. (#77)

b.

hitsuyô desu

education DAT necessity COP

tohushigini omou deshô

QUOTE mysteriously think would‘If an alien comes to (an) education site, (s/he) would
wonder, “for what purpose does public education
exist?”’(3) Emphasis:

The third hypothesis concerned finding English andJapanese constructions that exhibit a structural divergenceand frame mismatch, yet have the same pragmaticfunction. The analysis indeed found instances of suchcases. (5) is an example. The heads of the English andJapanese sentences in (5) are stop and surunja arimasen‘don’t!’ respectively (Step 1, Section 3.1). The English
and Japanese structures are of the Imperative
construction (cxn) and of V-surunjanai cxn
respectively (Step 2). The head stop in the English
sentence evokes the Activity_stop frame, whilesurunja arimasen in the Japanese sentence evokes thePreventing_or_letting frame (Step 3). Finally,both sentences function to order the addressee to stop an
activity (Step 4).

My contention is that creativity now is as important ineducation as literacy, and we should treat it with thesame status. (#43)(4) Cataphora:
a. Picasso once said this, he said that all children areborn artists. (#84)
b. If you were to visit education, as an alien, and say"What's it for, public education?" (#141)
Second, we expected to find in the Japanese translation
constructions with similar pragmatic functions as those of
the English original. The results of the analysis refutedthat expectation. Except for the translation of (4a), listed
below as (4’a), which uses cataphora to emphasize a quotefrom Picasso, none of the Japanese translations of theaforementioned English sentences (1-4) has structures that
focus the whole or a part of the sentence or emphasize thespeaker’s claims. This situation contrasts with that of the
English original sentences.

(5) Structural divergence, frame mismatch, and same
pragmatic function:
E: And stopActivity_stop speaking like that. (#105)
J: sonna hanashi kata surunja arimasenPreventing_or_lettingthat.way speech way don’t‘Don’t speak like that.’(1’)
a. jissai daremo ga

E: Imperative construction (cxn)
J: V-surunjanai cxn

kyôiku ni kanshin ga arundesu

actually everybody NOM education DAT interest NOM exist

literal translation2. ‘Actually, everybody has an interestin education.’b. (=(2’a))... machigaeru koto oosoreteitara kesshite
make.mistake thing ACC be.afraid

neverdokusôteki na mono nado omoitsuk

anaioriginal

E&J: Prohibiting function

4.

thing etc. come.up.with NEG‘... if (you are) afraid of making mistakes, (you) will
never come up with anything original.’(3’)
sôzôsei wa shikiji nôryoku to

Discussion

This section discusses functional mismatches in English
and Japanese in the parallel transcripts, the validity of theframes-and-constructions analysis method, and theappropriateness of using the TED Talk transcripts foraligning constructions for building a multilingual
constructicon.

4.1

onaji kurai

Functional mismatches in the TED paralleltranscripts

This subsection discusses the results with respect to thesecond hypothesis in Section 3.1. The second hypothesis
in Section 3.1 was that the Japanese translation would

2

All the translations of the Japanese sentences into English inthis paper are literal translations.

10

contain constructions that exhibit similar pragmaticfunctions as those in the English original. It turned outthat English sentence structures that focus certain of theirelements were often NOT translated into Japanese using
constructions with similar pragmatic functions.

corresponding. Pairs of constructions exist in the TED
Talk parallel transcripts that share the same pragmaticfunction while exhibiting frame mismatch and structural
divergence. It may thus be possible to hypothesize that thefunction of a construction takes precedence over exact
frame match as Czulo suggests. However, it is beyond thescope of this present paper to test this hypothesis.

It may be a consequence of properties of the genre,
specifically, of the Japanese transcript. While the English
version is an actual transcript of the oral presentation, theJapanese version is primarily a set of subtitles, that is,
captions displayed at the bottom of a screen that translate
the English transcript. In fact, the sentences in theJapanese version tend to be short and telegraphic,presumably because of the limited space allocated for
subtitles and the requirement to be displayed in synchwith the audio-visual information in the video clip. Thus,
what makes sense is to think of the Japanese transcript asa set of subtitles, something that should be seen and read
together with the video clip as part of multimodal
information, NOT as a translation. This study has yet to
conduct a thorough analysis of the video clip. Some sortof substitute for the pragmatic function to focus a sentenceelement missing in many of the Japanese sentence
structures may be found in the audio-visual information(including speech and gestural information) in the video
clip.

4.2

4.3

Toward a multilingual constructicon

While the TED Talk parallel corpus is a good resource forframe-semantic annotation in individual languages, it may
not be the ideal resource as a starting point to alignconstructions for building a bilingual constructicon
between English and another language, because of thecharacteristics of the genre of subtitles discussed in
Section 4.1. We may indeed be able to find better
functional alignment between two translated subtitle
transcripts, as opposed to comparing one translation to theoriginal3. Applying the frames-and-constructions method
to translated subtitles of two or more languages may
therefore be a better strategy to build a multilingual
constructicon from the parallel corpus.

5.

Summary and Future Work

This section summarizes the findings of the work
presented here:

Validity of the Frames-and-Constructions
Analysis

Since we were able to found pairs of corresponding
constructions in English and Japanese in the TED
transcripts, the four steps of the frames-and-constructions
analysis proposed in Section 3.2 seem useful in analyzing
transcripts of spoken data, in addition to written data. This
assessment is legitimate since the concepts embodied in
the four steps (i.e., head, sentence structure, semantic
frame, and function) are also found in transcripts ofspoken data. The proposed four steps particularly
emphasize the notion of head. Since the concept is
considered universal and since heads can be found insentences in transcripts of spoken data as well, identifying
sentential heads first facilitates accurate linguistic analysisof sentence structures (cf. Croft, In Preparation; Croft et al.
2017).

•In this respect, note Lyngfelt et al.’s (2018) proposal
concerning alignment of constructions across languages.
Based on the analyses of English, Swedish, and Brazilian
Portuguese constructions, that work proposed a four-step
comparison of constructions (Lyngfelt et al. 2018: 267).
The first step is to ask the question “is there acorresponding construction, or set of constructions, in thetarget language?”. While finding corresponding
constructions among typologically related languages suchas the three languages above may be easy, at least in thecase of Japanese and English, identifying corresponding
structures is quite difficult. Analyzing a parallel corpus
using the frames-and-constructions analysis method,which primarily relies on the concept of head, seems to bea more straightforward way of conducting the analysis.

••

Pairs of constructions in English and Japanese thatshare similar pragmatic functions exist in the TED
Talk bilingual transcripts. This is similar to ourfindings involving texts in a Japanese to English
bilingual children’s book. Therefore, the proposed
frames-and-constructions analysis method seems
valid not only for written language but also fortranscripts of spoken data.
While the TED Talk parallel corpus is a good
resource for frame semantic annotation in individual
languages, it may not be the ideal place to startaligning
constructions
acrosstypologicallyunrelated languages, because of the characteristicsof the genre of subtitles.
The frames-and-constructions analysis method
proposed here, namely, the one that focuses on thehead of a sentence in each language, seems valid tosearch for corresponding constructions intypologically-unrelated languages.

As Section 4.2 indicates, the four steps of the present
frames-and-constructions analysis predicts that even in thecase of frame mismatch, and even when structural
divergence exists, if the functions of two constructions in
the two different languages are the same, then the twoconstructions are comparable. Croft (In Preparation) andCroft et al. (2017) argue that syntax is primarily motivatedby information packaging, and secondarily by semantics.
Therefore how the proposed frames-and-constructions
analysis method relates to Croft’s claim is worth

The proposed four steps of frames-and-constructions
analysis predicts that even when frame mismatch andstructural divergence are present, if functions are the same,then the two constructions can be considered3

I would like to thank one of the reviewers for pointing this out
to me.

11

Takashi Y. (1991). Anpanman 1. Translated by Yuriko
Tamaki. Tokyo: Froebel Kan.

investigating in detail. Of particular interest is how what
we have called “pragmatic functions” interacts with
Croft’s “information packing.”

Bibliographical References
Croft, W. (In Preparation). Morphosyntax: Constructionsof the World’s Languages.
Croft, W., Nordquist, D., Looney, K., and Regan, M.
(2017). Linguistic typology meets universal
dependencies. In Markus Dickinson, et al. editors,
Proceedings of the 15th International Workshop on
Treebanks and Linguistic Theories (TLT15), pages 6375. CEUR Workshop Proceedings.
Czulo, O. (2013). Constructions-and-frames analysis oftranslations: the interplay of syntax and semantics intranslations
betweenEnglishandGerman.
Constructions and Frames, 5(2): 143-167.
Czulo, O. (2017). Aspects of a primacy of frame model oftranslation. In S. Hansen-Schirra, O. Czulo and S.
Hofmann (Eds.), Empirical modelling of translationand interpreting. Berlin: Language Science Press, pp.
465-490.
Fillmore, C. J. (2013). Berkeley Construction Grammar.
In: T. Hoffmann, & G. Trousdale (Eds.), The Oxford
Handbook of Construction Grammar. Oxford: Oxford
University Press, pp. 111–132.
Fillmore, C. J. & Baker, C. F. (2010). A frames approachto semantic analysis. In: B. Heine and H. Narrog (Eds.),
The Oxford Handbook of Linguistic Analysis. Oxford:
Oxford University Press, pp. 313–339.
Lyngfelt, B., Torrent, T.T., Laviola, A., Bäckström, L.,
Hannesdóttir, A.H., and Matos, E.E.S. (2018). Aligning
constructions across languages: a trilingual comparison
between English, Swedish, and Brazilian Portuguese. In
B. Lyngfelt, L. Borin, K. Ohara, and T.T. Torrent (Eds.),
Constructicography:
Constructicon
Development
across Languages. Amsterdam: John Benjamins
Publishing, pp. 255-302.
Ohara, K. (2019). Frames-and-constructions analyses ofJapanese and English Bilingual Children’s Book. Paper
presented at the Theme Session “Cross-theoretical
Perspectiveson
Frame-based
LexicalandConstructional Analyses: Bridging Qualitative andQuantitative Studies” the 15th International Cognitive
Linguistics Conference (ICLC 15), Nishinomiya, Japan,
August 9.
Ohara, K. and Okubo, Y. (2020). Finding corresponding
constructions in a Japanese-English Contrastive
children’s book using the frames-and-constructions
analysis. (In Japanese). Proceedings of NLP2020, pages
921-924, Ibaraki, Japan, March. The Association for
Natural Language Processing.
Ruppenhofer, J., Ellsworth, M., Petruck, M., Johnson, C.,& Scheffczyk, J. (2016). FrameNet II: Extended theory
and practice. Technical report. Berkeley: ICSI.

Language Resource References
Japanese FrameNet.
http://jfn.st.hc.keio.ac.jp
FrameNet.
https://framenet.icsi.berkeley.edu“Do schools kill creativity?” TED2006, February 2006.
https://www.ted.com/talks/sir_ken_robinson_do_schools_kill_creativity?language=en

12Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 13–22
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Towards Reference-Aware FrameNet Annotation
Levi Remijnsea and Gosse MinnemabaVrije Universiteit Amsterdam
De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands
l.remijnse@vu.nlbRijksuniversiteit Groningen
Oude Kijk in ’t Jatstraat 26, 9712 EK Groningen, The Netherlands
g.f.minnema@rug.nl

Abstract
In this paper, we introduce the task of using FrameNet to link structured information about real-world events to the conceptual frames
used in texts describing these events. We show that frames made relevant by the knowledge of the real-world event can be captured bycomplementing standard lexicon-driven FrameNet annotations with frame annotations derived through pragmatic inference. We propose
a two-layered annotation scheme with a ‘strict’ FrameNet-compatible lexical layer and a ‘loose’ layer capturing frames that are inferredfrom referential data.

1. Introduction

In this paper, we will show how FrameNet annotations canbe used as a resource for showing how structured knowledge about real-world events is conceptualized in text. Figure 1 shows an example of how FrameNet can be used toanalyze how a single event can be described from different
perspectives. While both texts mention the basic fact that akilling took place, the lower text stays close to the facts andprovides details about the shooting event itself, whereas theupper text is less detailed and takes a more interpretative
perspective by telling us that the event came to be seen as acrime (murder). This is reflected in the frame annotations:both texts evoke K ILLING, but only the upper text evokes
O FFENSES, whereas the lower one evokes U SE FIREARM.

Written narratives can describe a single real-world eventin different ways. In particular, an event of great cultural
importance often generates a growing portion of writtenreferential texts over time, all displaying various linguistic forms when referring to that same event or componentsof the event (Vossen et al., 2018a). These linguistic forms
activate conceptual representations displaying perspectives,
goals and motivations. In order to systematically investigate how the components of a single event are conceptually represented across texts, large-scale resources areneeded that, on the one hand, link knowledge about realworld events to event mentions in text, and on the other
hand link these mentions to conceptual information of thatevent. FrameNet can be a useful resource for linking event
mentions to conceptual information, given that it provides
a rich database of conceptual knowledge about event andsituation types, which are linked both to each other and to
lexical expressions evoking this conceptual knowledge.

In this fictitious example, the frames that are expressed by
the lexical items in the texts fit well with the conceptual
information needed to understand the perspective taken bythese texts. However, this is not always true in natural texts,
as in many cases, event descriptions are implicit. For example, “John was shot and died” does not contain any particular lexical item expressing a killing event, and would not
be annotated with K ILLING following FrameNet annotation
standards. Yet, the sentence clearly refers to such an event.

KILLING“An elderly lady from London killed
her husband. She was arrested andcharged with murder.”OFFENSES
OffensePerpetrator
Agent
Firearm

Event type: murder
Perpetrator: Jane Doe
Victim: John Doe
Weapon: gun

In this paper, we will analyze such challenges, and propose a way to more comprehensively annotate the relationship between frames and referential data. In short, we will
introduce an inferred frame layer of annotation on top ofa ‘regular’ FrameNet annotation layer. In this way, we
can annotate event mentions that standard frame annotation
would not be able to capture, while preserving a standard
FrameNet layer, thus contributing to the global FrameNet
effort. We will illustrate the challenges and proposals we
discuss with examples in English and Dutch, but we expect
them to be relevant cross-linguistically.

KillerVictimMeans

USE_FIREARM“John heard someone ﬁre a gun. Soon
after, he was on the ground, killedby his own wife.” KILLING

Contributions

The main contributions of our work are:

• We identify challenges for performing FrameNet annotation guided by referential data (Section 4.);

Figure 1: Fictitious example of the same event described indifferent ways with different frames• We propose a solution in the form of an extra anno-

13

Frames vs. events
Event type: category of real-world events
Example: murder, election

tation layer for pragmatically inferred frames (Section 5.);
• We show the implications of our approach for pragmatics and frame semantics (Section 6.);

Event instance: individual event in the real world
Example: the murder of JFK• We implement the inferred frame layer in an annotation tool as part of the Dutch FrameNet project1 – formore details, see Postma et al. (this workshop).

Frame type: frame entry in the FrameNet database,formally a tuple hT, E, Ri (T : set of target LUs, E: setof frame elements, R: set of frame-frame relations.
Example: K ILLING = h{kill.v, . . .}, {Killer, . . .},{hInherited by, E XECUTIONi, . . .}i.

2. Terminology
In order to avoid confusion between concepts from the ‘reference world’ and the ‘frame world’, some key terminologythat we will rely on throughout this paper is given in Box 1.
While these definitions might seem obvious, when linking
frame annotations to information about real-world events, it
is important to make an explicit distinction between eventsand frames on one hand and types, instances, and mentions
on the other hand. Not doing so could easily cause confusion in an example like (1):(1)

Frame mention: expression of a frame type in text,
formally a tuple hf, t, ei (f : frame type, t: target LU, e:
set of (frame element name, frame element span) pairs.
Example: given “He killed JFK”, annotate:
hK ILLING, killed, {hKiller, hei, hVictim, JFKi}i.
Box 1: Key terminology for our annotation taska. He killed the murderer of JFK, who was assassinated two days earlier.
b. He shot the murderer of JFK, who had died twodays earlier.
c. He murdered someone yesterday, and did it
again today.

(2)

In (1a), “killed”, “murderer”, and “assassinated”, all describe the same event type (murder) but refer to two different instances of this event type (the “murderer” and “assassinated” refer to the murder of JFK, “killed” refers to
the murder of JFK’s killer). They also all evoke the same
frame type (K ILLING), while each of them is a separate
mention of this frame. On the other hand, in (1b), “shot”,“murderer” and “died”, again refer to two event instancesof the type murder,2 but evoke three different frame types(H IT TARGET, K ILLING, D EATH), introducing a single
mention of each of these. Finally, in (1c), “murdered” and“did it again” describe two instances of the murder event
type, but only “murdered” is a mention of the K ILLING
frame type.

3.
3.1.

a. C OMMERCE SELL[Time Yesterday], [Seller John] sold [Buyer Mary][Goods a book].
b. C OMMERCE BUY[Buyer A woman] bought [Goods a novel][Place in the shop].

In (2a), “sold” is a lexical unit that evokes C OM MERCE SELL . This frame comes with an inventory offrame elements, some of which are necessary for thereader to process the frame (core elements). For C OM MERCE SELL , these are the Buyer, “Mary”, the Seller,“John”, and the Goods, “a book”. Similarly, in (2b),“bought” evokes C OMMERCE BUY, which has the same
frame elements: a Buyer, expressed by ‘a woman’, Goods,
expressed by “a novel”, and a Seller, which is unexpressedin this sentence.
The overlap of semantic roles between these two frame
types indicates that both C OMMERCE SELL and C OM MERCE BUY have a Perspective on relation with the abstract (‘non-lexical’) frame type C OMMERCE GOODS TRANSFER . This relation encodes the fact that both frame
types describe the same abstract concept, but from different perspectives: C OMMERCE SELL takes the point of viewof the Seller, whereas C OMMERCE BUY takes that of thebuyer. In this way, FrameNet provides us with rich information about variation in framing on a conceptual level.

Background

FrameNet and conceptual information

FrameNet (Baker et al., 2003; Ruppenhofer et al., 2010a)
provides a useful paradigm to analyze how conceptual information is encoded in language. Within this paradigm,
lexical units (word forms with a specific sense) can evoke
frame types, which are schematic representations of situations involving participants and other conceptual roles.
These semantic roles (frame elements, or FEs) are expressed by constituents. Frame mentions are analyzed
within clause boundaries. Two typical examples are givenin (2):

3.2.

Reference-driven annotation

Besides representing conceptual knowledge, a point of interest is to capture variations in the way that texts framecomponents of the real-world event that they refer to. We
want to know, for instance, whether the sentences in (2) describe the same event instance, and hence, whether “Mary”
in (2a) and “a woman” in (2b) both refer to the same participant of this event instance. In order to annotate texts
with this type of information, we need a resource providing structured data about events in the real world and texts
describing these events.

1

www.dutchframenet.nl
Note that knowledge of the real-world event is necessary torecognize that “shot” and “died” both describe a murder event:the lexical content of these words does not imply murder (one canbe shot without dying, and one can die without having been murdered), but in this context they do refer to (subevents of) murders.
2

14

We make use of the data-to-text method (Vossen et al.,2018b; Vossen et al., in press) in order to establish such a resource. This method inverts the usual process of annotating
data: instead of starting from (unstructured) text and thenannotating it with referential information, we start fromstructured information about real-world event instances andthen match these to texts describing these instances. Moreconcretely, we query Wikidata (Vrandečić and Krötzsch,2014) for a set of event instances belonging to a particular
event type. The Wikidata API then returns records of such
instances, accompanied by structured data (minimally: theevent type, data, location and participants). Wikidata also
provides the Wikipedia text pages in various languages,
which in turn provide hyperlinks that point to other texts
referring to the same event. We aggregate the Wikipedia
texts themselves with the texts they point to, to build a corpus of reference texts linked to event instances.

found, tells us that the two sentences refer to the same event
instance in the real world. This allows us to make the linkto the referential level by annotating “buy” and “sell” as referring to the event instance, and “Mary” and “a woman”
as referring to the same participant in that event instance.
In integrating these annotations, we find that the two sentences show conceptual variation in framing of the same
event instance in the real world.
The typical frames generate expectations about the frame
types to be found in the reference texts. Often, these frame
types are also conceptually necessary for recognizing thefor instance, a text can only be interpreted as describing amurder event if the conceptual content of K ILLING is somehow expressed in the text. Hence, in addition to guiding
expectations of the most probable frame types to be foundin the texts, the typical frames function as a ‘checklist’ forthe annotator to explore to what extent the typical frame
types are encoded in the text. Annotating whether or noteach typical frame is indeed expressed provides much information about the perspective of a text; for instance, inthe example texts discussed in the introduction (Figure 1),
O FFENSES is a typical frame for describing murder events,
but the fact that only one of the texts expresses this frame
type tells us something about the different perspectives of
the two texts.

Next, we prepare the corpus for manual FrameNet annotation. FrameNet contains a large number of different frame
types (1224 in Berkeley FrameNet for English).3 In order
to efficiently annotate large corpora, we restrict the scopeof our FrameNet annotations to only include frame types
that are known to be relevant for the event types in ourdataset. To achieve this, we first automatically annotate
the acquired corpus using Open-SESAME, a state-of-theart frame semantic role labeler (Swayamdipta et al., 2017).
Then, by analyzing the frequency distribution of the frame
types found in the automatic annotations, we define a listof typical frames containing the frame types that are mostdominant in texts referring to a particular type of event.4
To summarize, utilizing the data-to-text method results inthe following data:

The next subsection elaborates on the integration of frame
annotations and referential annotations.

As we will show in Section 4., in some cases, typical frames
are expressed in the text, but do not have a target word corresponding to a lexical unit in FrameNet, nor can they bederived through frame-to-frame relations. In such cases,
we run into an inherent limitation of FrameNet: FrameNet
is, at heart, a lexicographical project; conceptual information is always ‘activated’ through a direct correspondence
between a lexical unit and a frame. This limitation has
been noted even from within the field of frame semantics:
Fillmore himself has allowed for the possibility that frame
types, in some cases, are not evoked by lexical units, but byother linguistic features (Andor, 2010, p. 158). If we wantto account for the way in which frame types related to thereferential level are activated in corpora, we need to complement the lexical semantic approach of FrameNet with abroader view that takes into account compositional semantics and pragmatics. In Section 4., we motivate this view.

3.3.

3.4.

• Records of a set of event instances belonging to one
event type (e.g. ‘murder’);• A corpus of reference texts for each event instance;• Structured data for each event instance;• A list of typical frames belonging to the event type.

Integrating FrameNet in Referential
Annotations

FrameNet and Inference

The notion of ‘inference’ is crucial for the annotation approach proposed in this paper: we aim to annotate frame
mentions that are not directly evoked by a lexical unit but
whose relevance can be inferred from the textual and referential context of an event. Inference in the context offrame semantics has been studied in the literature, but thenotion we use in this paper is subtly different. Here, we
provide a brief overview of notions of inference found in
the FrameNet literature and how our notion differs from it.

The product of the data-to-text method enables the annotator to annotate frame mentions representing the conceptual content of each text, and then link these mentions to
structured data about the corresponding event instance. Returning to the examples in (2), we see that on the conceptual(frame) level, “Mary” is the Buyer of C OMMERCE S ELL in(2a) and that “a woman” is the Buyer of C OMMERCE B UYin (2b). Next, let us assume that the structured data weFrame-to-frame relations In the FrameNet literature,
inference is often connected to frame-to-frame relations.
For example, Chang et al. (2002) propose a scheme formodeling shared inferential structure between frame types.
An example of frame types with shared inferential structure3

See
https://framenet.icsi.berkeley.edu/
fndrupal/current_status, consulted on 2020-02-20.
4This is done by applying TFIDF weighting to frame type frequencies; see Vossen et al. (in press) for a detailed description ofthis method.

15

4.

are C OMMERCE BUY and C OMMERCE SELL: both refer tothe same type of event in the real world; hence, when oneof these frame types is used, it can be inferred that the otherframe type is also conceptually ‘active’. Different frame-toframe relations give rise to different kinds of inferences; forexample, Sikos and Padó (2018), investigate the Using relation as a source for paraphrases. This allows, for example,
for the inference of L ABELING (“he called him a hero”)
from J UDGEMENT COMMUNICATION (“he praised him
for being a hero”).

In this section, we detail and motivate the main challenges that we see for structured-data-driven frame annotation that cannot be solved within the standard frameworkof FrameNet. We first motivate the general problem, andthen discuss a number of concrete problems that we would
like to address. An overview of these problems is shown in
Box 2.
Annotation Challenges

For the purposes of this paper, we focus on a different kindof inference: we are interested in frame types whose conceptual content is ‘activated’ by a text, but cannot be annotated as being evoked by a lexical unit. While, in a subset ofsuch cases, there might be a frame-to-frame relationship between the frame type of interest and other frame types thatare evoked in the text, this is not always the case. Moreover, even if such a frame-to-frame relationship is present,
this might not be sufficient for licensing the inference. In
the example “John was shot and died” (discussed in the introduction), “die” evokes D EATH, which has a Causative
relation with K ILLING, but this relation alone is not enoughto make the inference: the fact that someone died does not
imply that this person was also killed. Instead, we can inferthat a killing did take place from the textual context (“Johnwas shot”).

Problem: how to link n LUs to m frame types
Many-to-One
Compositionality: ≥ 2 LUs, ≥ 1 frame type
Complex Verbs: verb components, ≥ 1 frame type(s)One-to-Many
Frame Overlap: 1 LU, ≥ 2 frame types
Lexical Gaps: out-of-vocab LU, ≥ 1 frame type(s)Box 2: Overview of the annotation challenges

4.1.

The Coverage Problem

A general issue of FrameNet that has been noted in the literature is that it covers many frame types while only a limited
number of number of annotations are available per frame
type and per lexical unit (Palmer and Sporleder, 2010;
Vossen et al., 2018b). As a logical consequence, when annotating texts with a limited set of frame types, as in our
approach, the number of annotations per text would be expected to be small. Indeed, results from the CALOR projectfor French (Marzinotto et al., 2018), in which a small subset(53 frame types) of all possible FrameNet frame types wasannotated, show that the number of sentences with at least
one frame mention varied between 21%–34%, dependingon the topic of the annotated texts.

Cognitive frames The idea of frames that are present but
not evoked by a lexical unit is also known from the literature about cognitive frames,5 as is evident in the following
famous example from Minksy:(3)

Challenges for Reference-Aware
Annotation

Mary was invited to Jack’s party. She wondered if
he would like a kite.
(Minsky, 1974)

Here, the lexical unit “party” evokes S OCIAL EVENT. The
second sentence, “she wondered if he would like a kite”
gives us reason to think that the party described is of a specific kind: most likely a birthday party. This would suggest
the relevance of a frame type such as B IRTHDAY PARTY(not currently existent): from our cultural knowledge, weknow that parties at which gifts are given are typically birthdays or some other type of commemorative event.

One of the texts that we annotated in preliminary annotation experiments, describing the killing of visitors of a
Christmas market in Berlin during a terror attack in 2016,is shown in Table 1. Our aim is to show whether each of thereferential attributes of the event is expressed in the text,
and if so, how it is conceptualized with frame mentions.
For this particular text, reasoning from structured data, one
would expect at least K ILLING to be activated, and possibly also O FFENSES, U SE FIREARM, and/or W EAPONS(depending on whether the event is seen as an offense andwhether the authors choose to mention the weapon). Surprisingly, it turns out that none of these frame types is
evoked in the text in relation to the event mention of interest; even though “he was killed in a shootout . . . ” contains
a K ILLING frame mention, this is in relation to a secondary
event mentioned in the text (i.e. the killing of the perpetrator of the main murder event described in the text). Moreover, none of the frame types evoked by the lexical units inthe text can be linked to the typical frames through a frameto-frame relation; if this had been the case, we might havebeen able to indirectly annotate the frame types of interest,

However, this notion of inference goes beyond what we areaiming for in this paper. In the above example, it could
be guessed what kind of party is at play, but the inference
does not follow directly from the text: it could be someother party where, for whatever reason, gifts are given.
This means that annotators would have to rely on their cultural knowledge. By contrast, within our framework, world
knowledge can play a role in deriving inferred frame types,
but their conceptual content should always be fully specified by the linguistic cues in the text. However, unlike instandard FrameNet annotation, these cues are not limited tosingle lexical items, but can comprise larger constructions.
5
We assume the distinction between cognitive and linguistic
frames proposed by Fillmore (2008).

16

[Wikidata] Q28036573
Event type: murder
Time: 2019-12-19
Location: Berlin
Participant: Annis Amri
Number deaths: 12
Weapon: truck[Text] “2016 Berlin truck attack”
On 19 December 2016, a truck was deliberately driven into theChristmas market next to the Kaiser Wilhelm Memorial Church
at Breitscheidplatz in Berlin, leaving 12 people dead and 56 others injured. [. . . ] The perpetrator was Anis Amri, a Tunisianfailed asylum seeker. Four days after the attack, he was killed ina shootout with police near Milan in Italy. [. . . ][Typical Frames]{ K ILLING,U SE FIREARM,O FFENSES,W EAPON,
C OMMIT CRIME }Table 1: Example output of the data-to-text pipeline.
as discussed in Section 3.4..

in (5) (already briefly discussed above) is a clear exampleof this:However, from a close examination of the text, we find thateach of the referential attributes from the structured data is
in fact mentioned, but without using any lexical units belonging to one of the typical frames. We argue that the conceptual content of these frame types is still relevant for describing how the event instance is expressed in the text, andthat this should be reflected in the annotations. For example, in FrameNet, the definition of K ILLING is given as “A
Killer or Cause causes the death of the Victim”. A ‘killing’
event is very clearly expressed in the text by “a truck wasdeliberately driven into the Christmas market . . . leaving 12
people dead”. However, it is difficult to specify which lexical unit(s), if any, evokes this particular frame mention in
the standard FrameNet sense.

(5)

The sentence describes an action (“drive”) with the consequence (“leaving”) of people dying (“dead”); while noneof these is a ‘killing word’ per se, the sum of these components imply (or even entail) that a killing event took place.
We would like to capture in our annotations that (the conceptual content of) K ILLING is relevant for this sentence,but standard FrameNet annotation does not allow us to annotate this, since there is no lexical target for K ILLING, nor
can K ILLING be derived through other frame types that are
evoked in the text.6

Work on what has become known as the implicit semantic role labeling task (Ruppenhofer et al., 2010b) addresses
a related problem: semantic roles are sometimes ‘missing’in the sentence of their associated predicate, but are conceptually ‘activated’ by this predicate and expressed elsewhere in the discourse. In example (4), the Charges roleof “cleared” is not explicitly expressed, but can be inferredbecause “murder” is still active from the previous sentence:(4)

Complex Verbs A special case of the Compositionality
Problem is the Complex Verbs Problem, in which the targets that jointly activate a frame type are all part of a complex (prepositional) verb:(6)

In a lengthy court case the defendant was tried[Charges for murder]. In the end, he was cleared.
(Ruppenhofer et al., 2010b, p. 107)

a. O PERATE VEHICLE. . . [Vehicle a truck] was deliberately driven[Goal into the Christmas market] . . .
b. I MPACT. . . [Impactor a truck] was deliberately ? driven? into [Impactee the Christmas market] . . .

Since FrameNet lists “drive”, but not “drive into”, as a lexical unit, the canonical analysis of (6) should be (6a). However, in this sentence, “into” does not simply add a destination to “drive”, but modifies the meaning of “drive” so thatit expresses not just a driving event, but also a hitting event.
Hence, one would like to annotate a mention of I MPACT aswell as of O PERATE VEHICLE.

The challenges we address in this paper are also related to
implicit semantic roles, but in a more abstract way: in ourcase it is not the fillers of semantic roles, but the frame types
defining these semantic roles that are unexpressed and haveto be inferred. In the remainder of this section, we will
discuss these challenges in more detail. In Section 5., wewill propose a solution to these challenges.

4.2.

K ILLING[Cause a truck] was deliberately ? driven . . .
? leaving [Victim 12 people] ? dead . . .

The Complex Verb Problem is particularly relevant in
Dutch, which has many complex verbs that are often discontinuous:7‘Many-to-One’ Problems

In the first class of challenges we encountered, at least one
frame type is relevant for describing how an event instanceis conceptualized, but there is no lexical unit in the text
that, under standard FrameNet assumptions, would evoke
this frame type. Instead, several items in the text together
allow the reader to infer that the frame type is relevant, andgive rise to annotating a mention of this frame.

(7)

6

Toen reed een vrachtwagen op het publiek inthen drove a truckon the crowd into

For example, “dead” evokes D EAD OR ALIVE, which is (distantly) related to K ILLING, but does not imply its relevance (thefact that someone dies does not imply that someone was killed).
7
From the Dutch version of the Wikipedia article about the
Berlin Christmas market attack (https://nl.wikipedia.
org/wiki/Aanslag_op_kerstmarkt_in_Berlijn_
op_19_december_2016).

Compositionality The Compositionality Problem occurs
when multiple lexical items, through the composition of
their meanings, ‘activate’ a single frame type. The sentence

17‘Then, a truck (deliberately) drove into the crowd’“Murder” is a lexical unit in both O FFENSES and K ILLING,
and has an almost identical meaning in both of them.
Which of the two frame types should be annotated depends on the context: O FFENSES.murder is activated only
when there is a governing verb such as ‘convict’ or ‘accuse’; in other contexts K ILLING.murder is activated. In(9), we clearly have an O FFENSE context rather than a
K ILLING context, but this does not mean that the meaningof K ILLING is not also active: while the sentence, through amention of O FFENSES, tells us that someone was convictedof a crime (further specified as the ‘murder of JFK’), it also
tells us that the murder happened in the first place, which
we would like to capture using a mention of K ILLING.

a. O PERATE VEHICLE[Time toen] ? reed [Vehicle een vrachtwagen][Goal op het publiek in]b. I MPACT[Time toen] ? reed [Impactor een vrachtwagen][Impactee ? op het publiek] ? in
Here, the verb inrijden (op) “(deliberately) drive into” expresses the same two meanings (i.e., driving and hitting) as“drive into” in (6). However, “in” in “inrijden” is arguably‘more part of the verb’ than “into” in “drive into”; thus, it islikely that “inrijden” would be a separate lexical unit in the(still to be developed) Dutch FrameNet. Hence, in (6), thecorrect analysis under standard FrameNet annotation would
be to use O PERATE VEHICLE (because “drive”, not “driveinto” exists in FrameNet). By contrast, in Dutch FrameNet,“inrijden” would most likely be a lexical unit of I MPACT.
Hence, (6) and (7) have an almost identical semantic content but would get very different analyses, where one of therelevant frame types is lost. Ideally, in our annotations wewould like to capture both of the two relevant frame types.

4.3.

Lexical Gaps An extreme case of the Frame Overlap
Problem occurs when a particular lexical unit does not exist
in FrameNet, but would be a potential target for some frame
type. We call this the Lexical Gaps Problem: a single lexical unit is associated with zero frame types in FrameNet,
but at least one frame type is relevant for annotation. For
example, in (10), “perpetrator” is not listed as a lexical unit
for C OMMIT CRIME, but is a very likely target for it, especially because the verb “perpetrate” is listed under thatframe type.8‘One-to-Many’

The second class of challenges that we identify applies in
the inverse situation of the ‘many-to-one’ challenge: these
consist of cases with a certain number of relevant frame
types, but not enough lexical units to evoke all of theseframe types.

(10)

It is well-known that the FrameNet lexicon is incomplete,especially when annotating out-of-domain corpora (Hartmann et al., 2017). In this sense, the Lexical Gaps problem
seems more superficial than the other problems discussed
in this section. Yet, the lexical gaps detected by using our
method of structured-data-driven annotation require somekind of inference on the part of the annotator. Namely,
the list of typical frames guides the annotator in inferring
frame types from potential lexical units currently missing
in FrameNet.

Frame Overlap Under the Frame Overlap Problem, asingle lexical unit is relevant for more than one frame type.
An example is given in (8):(8)

H OSTILE ENCOUNTER[Side 1 he] was killed in a shootout [Side 2 with police]5.

In FrameNet, “shootout” is listed as a lexical unit of H OS TILE ENCOUNTER . However, the lexical semantics of“shootout” clearly involves the use of a firearm, whichmakes U SE FIREARM conceptually relevant as well. Since
U SE FIREARM is part of the typical frames for murder
events, we would like our annotations to reflect the factthat the text indeed expresses a U SE FIREARM event. A
naive solution would be to add a lexical unit “shootout” to
U SE FIREARM so that we could annotate that frame type.
This would not work well, since U SE FIREARM, thoughconceptually relevant, does not fit well with the structure of
the sentence: a typical context of U SE FIREARM are sentences like “[Agent she] fired [Firearm her gun]”, with thefirearm and the shooter, rather than the participants in a
conflict, as core roles.

Towards a Workable Solution for
Annotating Inferred Frames

In this section, we aim to address the challenges previously
explained by proposing an extra annotation layer (next to,
not instead of, traditional FrameNet annotation) for capturing inferred frames whose conceptual content is expressedin the text without explicitly using one of the frame type’s
lexical units, but through inference. This layer would allow annotators to use any combination of words in the text
as a ‘trigger’ for any number of frame mentions. While
this idea is conceptually simple, some challenges need to
be overcome for implementing it in practice: how do we
make sure we get enough data? How do we apply the annotations in a consistent way?

5.1.

An even more subtle version of the Frame Overlap Problem
arises from the hypothetical example in (9):(9)

C OMMIT CRIME
The ? perpetrator was [Perpetrator Anis Amri] . . .

Introducing Inferred Frame Annotation

The overall annotation pipeline that we propose is shownin Figure 2. The process starts with choosing event types

O FFENSES[Perpetrator He] was convicted for the[Offense murder] of [Victim JFK].

8
For comparison: in K ILLING, both “murder” and “murderer”
are listed as targets.

18

1

2

Select event
types

Data-to-text
Find structured
event data, texts,
typical frames

3‘Strict’ FN
annotation

4
Annotate frame types
in typical frames
Link frame tokens toevent tokens

Inferredframesannotation

Adapt strict
annotation method

OR
Crowd-source
annotations

Figure 2: Overall annotation pipelineof interest and running the data-to-text pipeline (see Section 3.2.) to obtain linked event data and texts (Steps 1
and 2). Then, ‘strict’ FrameNet annotation is applied (Step
3): this annotation step will be done following standard
FrameNet guidelines, except that (i) only frame types inthe typical frames, selected by the data-to-text algorithm,
will be taken into consideration and that (ii) frame mentions will be linked to event instances and their attributes
in the structured data, much like in the initial example we
gave in Figure 1. This step will be done by annotators, who
need to be trained in applying FrameNet annotation guidelines. Finally (Step 4), we will annotate the inferred frame
layer that we have motivated in this paper.

[SCREEN 1/3]

The text at the bottom might describe one ormore of the following mini-stories:

according to the text, someone kills
someone else (KILLING)
according to the text, someone shoots witha gun or similar weapon (USE_FIREARM)
according to the text, someone committed acrime or is accused of it (OFFENSES)“He was killed in a shootout with police”

Annotation on this layer is much ‘looser’ than the annotation done in step 3. Annotators do not need to respect the
FrameNet rule of ‘one lexical unit, one frame mention’, but
are free to annotate any number of frame mentions basedon any combination of lexical items in the text. An inherent risk of this type of ‘free-style’ annotation is that interannotator agreement is likely to be lower, simply because
the number of possible annotation decisions is much larger
and less constrained than under standard FrameNet annotation.

(a) Screen 1: explanation of frame types[SCREEN 2/3]

Do you think the KILLING mini-story is expressedin the text?

YES / NO

5.2.

If yes, please click on the words in the text thatmade you think the story is expressed.

Annotation Strategies

Currently, we see two possible (not necessarily exclusive)
paths to mitigating this risk. The first option involves aqualitative approach that aims to make the procedure thatannotators follow as consistent as possible. Alternatively, aquantitative approach would use crowd-sourcing for gathering as much data for every text as possible, and then comparing and aggregating the annotations from different annotators.

“He was killed in a shootout with police”(b) Screen 2: selecting target words[SCREEN 3/3]

For each of the participants in the KILLING
mini-story, click any words in the text that
describe them.

Under the qualitative option, we would integrate annotation of Step 3 and Step 4: the annotators would annotate
both layers in the same way, using the same tools. The advantage would be that the annotators are trained in doing
FrameNet annotation, which improves the consistency of
the annotations. However, due to the looseness of the task,
we still expect considerable disagreements between different annotators. Moreover, training and deploying expert
annotators is costly and time-consuming.

Killer: the person who killed someone else

“He was killed in a shootout with police”
Victim: the person who was killed“He was killed in a shootout with police”

On the other hand, the quantitative option would ‘embrace’the unconstrained nature of the inferred frame layer, anduse crowd sourcing to gather as much data as possible. This
would mean moving further away from standard FrameNet
annotation, given that the annotators would be unfamiliar
with FrameNet and its philosophy. Annotations are also(c) Screen 3: selecting frame elements

Figure 3: Mockup of a crowd-sourcing interface (possibleuser input marked in bold)19

likely to be less consistent: different annotators might havedifferent standards for what words are relevant for eachframe mention.

be pragmatic instead of lexical in nature. In the following
subsections, we will elaborate on the potential benefits of
categorizing the output in this way.

However, previous studies have shown that annotation tasks
similar to FrameNet annotation, such as PropBank-style semantic role labeling, can be successfully addressed using(partial) crowd sourcing (Wang et al., 2017). Moreover,
the task of annotating the inferred frame layer is potentiallymore suitable for crowd sourcing than standard FrameNet
annotation is: since there are no strict guidelines that the
annotations need to adhere to, it is not clear how consistent the annotations need to be with one another in order
to be acceptable. In fact, provided that enough data points
are collected, it might be interesting to get a wide rangeof possible annotations from different annotators applying
slightly different strategies, and then to look for patterns
that apply across annotators. After the annotation process,a ‘canonical’ representation of the annotations could be obtained by filtering out infrequent annotations.9

6.1.

Certain annotations can be categorized as conventional.
These annotations could not be performed in traditional
FrameNet, but nevertheless seem to show a consistent mapping to the same targets across texts, and therefore mightshow a lexical coverage problem. These conventional inferences can provide useful insights for enriching or adapting the FrameNet database. The most typical examples ofannotations that reveal coverage problems, are the ones related to the Lexical Gaps Problem (see Section 4.3.). Whena word that is not yet listed in FrameNet is consistently
annotated as activating a particular frame type, this wordmight be a lexical unit that is still missing in the frame
type’s inventory and could be added to it. However, because of the ‘looseness’ of the inferred frame layer, it isalso possible that a word is very often annotated with a particular frame type, but does not qualify for being a lexical
unit in the standard FrameNet sense.

A possible way to present the task to crowd annotators
would be as shown in the example in Figure 3. In the firstscreen, the sentence to be annotated would be shown together with simple explanations of the frame types in the
typical frames (which could be called ‘mini-stories’ forpeople unfamiliar with FrameNet). Next, for every frame
type, the annotators would be asked to indicate if they think
the text expresses it, and if so, which words in the sentence contribute to it. Finally, if the frame type is indeed
expressed, the same question is asked for all of the (core)frame element.

For instance, “perpetrator” is currently not listed in
FrameNet, but is conceptually relevant for O FFENSES, so it
is conceivable that many annotators would annotate it as activating this frame type, even though it does not fit well withthe structure of O FFENSES (which exclusively lists kindsof offenses such as “murder.n”, “robbery.n”). However, thefact that the word is frequently annotated still suggests itshould be added to FrameNet. A potential strategy to deal
with this is to look for a better fit in frame types directly
related to the one that is annotated. In this case, a good fit
could be C OMMITTING CRIME (as we argued previously),
which is connected to O FFENSES through the Is used byrelation.

For implementing the crowd-sourcing task, we propose
making use of the Wordrobe gamification platform (Venhuizen et al., 2013). In Wordrobe, annotators get scoresbased on how consistent they are with other annotators, andare encouraged (e.g. through ‘leader boards’) to aim forhigher scores. This encourages consistency and makes theannotation task more interesting for participants.

6.

Conventional Inferences and FrameNet
Coverage

Another type of conventional annotation that provides cuesfor enriching FrameNet is related to the Frame Overlap
Problem (see Section 4.3.): if annotators consistently annotate particular frame types on the inferred frame layer asactivated by the same targets, this could be a strong indicator that there exists a relation between these frame types.
For instance, if O FFENSES is often annotated for the same
lexical items as K ILLING, then these frame types are likely
to be related.

Discussion

The output of the inferred frame layer forms a scheme displaying a group of n words for each frame mention that, according to the annotator, activates the corresponding frame
type. In this section, we will argue that the inferences that
led to each of these annotations can be categorized as either ‘conventional’ (i.e., always apply) or ‘situational’ (i.e.,
only apply in a specific context). We expect that most conventional inferences indicate coverage gaps in FrameNet.
Once identified, these could be used to enrich the database.
On the other hand, we expect the situational inferences toFinally, the Frame Overlap problem can also provide cuesthat some lexical units are conceptually related to more than
one frame type. Even when one of these frame types clearlyfits best (e.g., H OSTILE ENCOUNTER for “shootout”, see
example (8)), the conceptual content of another frame typemay still be relevant to such a degree that it can be viewed
as part of the lexical meaning of the target word. This could
be suggested by a large number of annotations of this frame
type on the inferred frame layer (e.g., U SE FIREARM for“shootout”). A possible way for encoding this in the lexicon would be to introduce frame-lexical unit relations in
FrameNet. Currently, lexical units can only be related toframe types through the ‘evoke’ relationship, which means

9This should be done on different levels. For example, in Figure 3b, there should be a mention of K ILLING in the final representation only if a majority of annotators answers “yes” (framelevel); “shootout” should be kept as a target word for this mention
only if a majority of annotators included it in their target span (target level); and “police” should be kept as a mention for the Killer
frame element only if a majority of annotators included it (frame
element level).

20

that every lexical unit can be related to only one frame type.
However, as we have shown, lexical units can make theconceptual content of more than one frame type relevant
without, strictly speaking, evoking all of these frame types.
Allowing for secondary frame-lexical unit relations would
allow us to model one-to-many mappings without weakening the existing ‘evoke’ relationship.

6.2.

stance in the real world can be conceptualized in text in different ways using frames. We showed that, in some cases(e.g. in the example in Figure 1), this can be done within
the standard FrameNet annotation framework. However,in many cases the annotation scheme needs to be extended
in order to allow for annotating frame mentions without
an explicit lexical target. As a general solution, we proposed adding an inferred frame layer that allows arbitrary
text spans to serve as a ‘trigger’ for any number of frame
mentions, and suggested two possible ways to annotate thelayer: either using a traditional FrameNet annotation process with annotators trained specifically for the task, or using crowd-sourcing. Finally, we show that the output of
the inferred frame layer could be used as a basis for pragmatic analysis, and for extending the lexical coverage ofFrameNet.

Situational Inferences and Pragmatics

The remainder of the annotations in the inferred frame layer
can be categorized as situational. For instance, from theconceptually related linguistic components in sentence (5),K ILLING is inferred with the aid of situational knowledge
about the incident. This inference differs from the inference
leading to B IRTHDAY PARTY in (3), which is derived fromboth cultural knowledge and cues that are not conceptuallyrelated but frequently co-occur in the context of this frame
type.

8.

The research reported in this article was funded by the
Dutch National Science organisation (NWO) through the
project Framing situations in the Dutch language.

In the field of Gricean pragmatics, annotations like the onein (5) can be analyzed with respect to the means of inference (entailment, implicature, etc., see Levinson (1983) andGrice (1975)) by which frame mentions are pragmaticallyderived. Also, one could investigate external factors, suchas historical distance and cultural background, underlying
these inferences.

9.

Bibliographical References

Andor, J. (2010). Discussing frame semantics: The state of
the art: An interview with Charles J. Fillmore. Review of
Cognitive Linguistics, 8(1):157–176.
Baker, C. F., Fillmore, C. J., and Cronin, B. (2003). The
structure of the FrameNet database. International Journal of Lexicography, 16(3):281–296.
Chang, N., Petruck, M. R. L., and Narayanan, S. (2002).
From frames to inference. In In Proceedings of the FirstInternational Workshop on Scalable Natural Language
Understanding.
Fillmore, C. J. (2008). The merging of “frames”. In
Favretti R. Rossini, editor, Frames, corpora, and knowledge representation, pages 2–12. Bologna: Bononia University Press.
Grice, H. P. (1975). Logic and conversation. In Peter Coleet al., editors, Speech acts, pages 41–58. Brill, Leiden.
Hartmann, S., Kuznetsov, I., Martin, T., and Gurevych, I.
(2017). Out-of-domain FrameNet semantic role labeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 471–482.
Levinson, S. C. (1983). Pragmatics. Cambridge Textbooks in Linguistics. Cambridge University Press.
Marzinotto, G., Auguste, J., Bechet, F., Damnati, G., andNasr, A. (2018). Semantic frame parsing for information extraction: the CALOR corpus. In Proceedings ofthe Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Minsky, M. (1974). A framework for representing knowledge. MIT-AI Laboratory Memo, 306.
Palmer, A. and Sporleder, C. (2010). Evaluating
FrameNet-style semantic parsing: the role of coverage
gaps in FrameNet. In COLING 2010: Posters, pages
928–936.
Postma, M., Remijnse, L., Ilievski, F., Fokkens, A., Titarsolej, S., and Vossen, P. (this workshop). Combining

Another way in which this type of situational inference isrelevant for pragmatics is by exposing discourse relations.
This crucially depends on the observation that event instances, after being introduced in the beginning of a text,
may be implicated in the remainder of the text. By annotating the referential relationship between the initial event
mention and implicated event mentions, we implicitly capture this discourse relation and use it to combine the conceptual content from the frame types they evoke. For example, in the text in Table 1, once the murder event instance has been introduced (by “a truck was deliberately
driven into the Christmas market . . . leaving 12 peopledead”, which under our approach could be annotated withK ILLING), it will be implicitly active in the remainder of
the text. This leads words like “perpetrator” (which evokes
C OMMITTING CRIME) to be interpreted against the background of this event. Marking the two event mentions as
referentially related then allows us to connect their associated frame mentions as well. Given that K ILLING is still‘active’ in the discourse, we can infer that “perpetrator”
refers to a murder, and not to some other crime.
The incidental nature of these inferences makes it hard forresearchers to model them in such a way that they can be
added to FrameNet. One could wonder if researchers want
incidental relations between frame types to be implemented
in such a lexicographical project at all. Rather, situationallyinferred frames show that even a fully developed version of
FrameNet would not allow us to annotate all frame mentions referring to an event instance.

7.

Acknowledgements

Summary

In this paper, we introduced a new use case of FrameNet:using frame annotations for showing how a single event in-21

conceptual and referential annotation to study variationin framing.
Ruppenhofer, J., Ellsworth, M., Petruck, M. R. L., Johnson,
C. R., and Schefczyk, J. (2010a). FrameNet II: Extended
theory and practice.
Ruppenhofer, J., Sporleder, C., Morante, R., Baker, C., andPalmer, M. (2010b). SemEval-2010 Task 10: Linking
Events and Their Participants in Discourse. In Proceedings of the 5th International Workshop on Semantic Evaluation.
Sikos, J. and Padó, S. (2018). Framenet’s using relation
as a source of concept-based paraphrases. Constructions
and Frames, 10(1):38–60.
Swayamdipta, S., Thomson, S., Dyer, C., and Smith, N. A.
(2017). Frame-semantic parsing with softmax-margin
segmental rnns and a syntactic scaffold. arXiv preprint
arXiv:1706.09528.
Venhuizen, N. J., Basile, V., Evang, K., and Bos, J. (2013).
Gamification for word sense labeling. In Proceedings ofthe 10th International Conference on Computational Semantics (IWCS 2013) – Short Papers, pages 397–403.
Vossen, P., Caselli, T., and Cybulska, A. (2018a). How
concrete do we get telling stories? Topics in cognitive
science, 10(3):621–640.
Vossen, P., Ilievski, F., Postma, M., and Segers, R. (2018b).
Do not annotate, but validate: a data-to-text method for
capturing event data. In Proceedings of the EleventhInternational Conference on Language Resources and
Evaluation (LREC-2018).
Vossen, P., Ilievski, F., Postma, M., Fokkens, A., G., M.,
and L., R. (in press). Large-scale cross-lingual language
resources for referencing and framing. Paper to be presented at LREC 2020.
Vrandečić, D. and Krötzsch, M. (2014). Wikidata: a freecollaborative knowledge base.
Wang, C., Akbik, A., Chiticariu, L., Li, Y., Xia, F., and
Xu, A. (2017). CROWD-IN-THE-LOOP: A hybrid approach for annotating semantic roles. In Proceedings ofthe 2017 Conference on Empirical Methods in Natural
Language Processing, Copenhagen, Denmark.

10.

Language Resource References

Collin F. Baker. (2015). FrameNet. International Computer Science Institute, Berkeley, version 1.7.
Vrandečić, Denny and Krötzsch, Markus. (2014). Wikidata: a free collaborative knowledge base.

22

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 23–30
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Frame-Based Annotation of Multimodal Corpora: Tracking (A)Synchronies in
Meaning Construction
Frederico Belcavello, Marcelo Viridiano, Alexandre Diniz da Costa, Ely Edison da Silva Matos,
Tiago Timponi Torrent
FrameNet Brasil – Federal University of Juiz de Fora
Juiz de Fora, Brazil{frederico.belcavello, alexandre.costa, ely.matos, tiago.torrent}@ufjf.edu.br
marcelo.viridiano@gmail.com
Abstract

Multimodal aspects of human communication are key in several applications of Natural Language Processing, such as MachineTranslation and Natural Language Generation. Despite recent advances in integrating multimodality into Computational Linguistics, themerge between NLP and Computer Vision techniques is still timid, especially when it comes to providing fine-grained accounts formeaning construction. This paper reports on research aiming to determine appropriate methodology and develop a computational tool toannotate multimodal corpora according to a principled structured semantic representation of events, relations and entities: FrameNet.
Taking a Brazilian television travel show as corpus, a pilot study was conducted to annotate the frames that are evoked by the audio andthe ones that are evoked by visual elements. We also implemented a Multimodal Annotation tool which allows annotators to choose
frames and locate frame elements both in the text and in the images, while keeping track of the time span in which those elements areactive in each modality. Results suggest that adding a multimodal domain to the linguistic layer of annotation and analysis contributes
both to enrich the kind of information that can be tagged in a corpus, and to enhance FrameNet as a model of linguistic cognition.
Keywords: Frame Semantics, Multimodal Annotation, FrameNet

1. Introduction

chosen also configures a novel experimental setting for
research on integrated image and text comprehension,
since, in this corpus, text is not a direct description of theimage sequence, but correlates to it indirectly in a myriadof ways.

The FrameNet Brasil Lab has been engaged in developing
resources and applications for Tourism (Torrent el al.,2014; Diniz da Costa et al., 2018) using Frames – in theway they were defined by Fillmore (1982) – as structured
representations of interrelated concepts. Frames are, then,the pivot structures for Frame Semantics, in which words
are understood relative to the broader conceptual scenes
they evoke (Fillmore, 1977). As the computational
implementation of Frame Semantics, FrameNet has beendeveloped as a lexicographic database that describes thewords in a language against a computational representationof linguistic cognition based on frames, their frame
elements (FEs) and the relations between them. The
analysis is attested by the annotation of sentences
representing how lexical units (LUs) instantiate the frames
they evoke. FrameNet projects have been started producing
databases in many languages, such as Brazilian
Portuguese.1

The methodology defined was to:
1.

2.

3.

To accomplish the steps (ii) and (iii) we developed aMultimodal Annotation Module for the FrameNet Brasil
Webtool.

In order to make FrameNet Brasil able to conductmultimodal analysis, we outlined the hypothesis thatsimilarly to the way in which words in a sentence evoke
frames and organize their elements in the syntactic locality
accompanying them, visual elements in video may, then,(i) evoke frames and organize their elements on the screenor (ii) work complementarily with the frame evocation
patterns of the sentences narrated simultaneously to theirappearance on screen, providing different profiling andperspective options for meaning construction.

The results achieved so far suggest that, at least for this TV
format but maybe also for others, a fine grained semantic
annotation tackling the (a)synchronous correlations thattake place in a multimodal setting may provide data that is
key to the development of research in Computational
Linguistics and Machine Learning whose focus lies on the
integration of computer vision and natural language
processing and generation. Moreover, multimodal
annotation may also enrich the development of FrameNets,
to the extent that correlations found between modalities canattest the modeling choices made by those building framebased resources.

To test the hypothesis, we designed a pilot experiment for
which we selected a Brazilian television travel show
critically acclaimed as an excellent example of goodpractices in audiovisual composition. The TV format
1annotate the audio transcript using the FrameNet
Brasil Annotation WebTool (Matos and Torrent,2018), that allows for the creation of frames andrelations between them, as well as for the annotationof sentences and full texts;
considering audio as the controlling modality in this
corpus, annotate the frames evoked by visual objects
or entities that are grounded on or related to theauditory guidance;
analyze synchronies and asynchronies between the
annotations.

See https://www.globalframenet.org/partners.

23

2. Computational Processing of Multimodal
Communication

the relation expressed by audio and video in the selected
corpus. This aspect will be discussed next.

Multimodal analyses have been growing in importancewithin several approaches to both Cognitive Linguistics
and Natural Language Understanding, changing the
scenario depicted by McKevitt (2003), according to whom
little progress had been made in integrating the areas of
Natural Language Processing (NLP) and Vision
Processing (VP), although there had been much success indeveloping theories, models and systems in each of theseareas separately.

3. Multimodal Grammars
Based on Jackendoff’s (2002) parallel architecture oflanguage, Cohn (2016) focuses on how grammar andmeaning coalesce in multimodal interactions, extending
beyond the semantic taxonomies typically discussed within
the domain of text–image relations. He thus classifies therelations between text and image in visual narratives,
evaluating the presence or absence of grammar structuring
each of the modalities and also the presence or absence ofsemantic dominance by one of the modalities.

Aksoy et al. (2017) present a review of the state of art on
linking natural language and vision, highlighting that therelated literature mostly focuses on generating descriptionsof static scenes or object concepts. They, then, offer anunsupervised framework which is able to link continuous
visual features to textual descriptions of videos of longmanipulation activities. The results show interesting
capacity of semantic scene understanding, although the
linguistic material is limited to automatically generated
text descriptions.

The first step of this method for analyzing multimodal
interactions would be to determine if one of the modalities
controls the other in terms of meaning, that is, if there is asemantic dominance according to which one of themodalities plays a preponderant role in determining themeaning expressed by the media. If the answer is yes, there
will be a relation of assertiveness or dominance. If theanswer is no, the relation will be of co-assertiveness or codominance.

Sun et al. (2019), on the other hand, report the developmentof a joint model for video and language representation
learning, VideoBERT, in which the text processed is
captured from the original audio of the videos that integrate
the corpus. Therefore, this model is capable of learnbidirectional joint distributions over sequences of visualand linguistic inputs. Although it is shown that the model
learns high-level semantic features, it should be pointed outthat the genre of videos selected – cooking instructions orrecipe demonstrations – offers a very straightforward
correlation between visual and auditory content, whencompared with many other TV, audiovisual orcinematography genres.

Cohn’s model considers that there is assertiveness (or coassertiveness) when both modalities have grammar - in thecase of text modality, the grammar is expressed in terms ofsyntax; in the case of image, what counts as grammar is thenarrative. The dominance (or co-dominance) will occur
when one of the modalities has grammar and the otherdoesn't.
In our study we consider that, throughout the TV show,
audio plays a controlling role in establishing meaning,although there are significant visual sequences in the formof video clips that express a linear narrative.

Turner (2018) explains that multimodality is traditionallyexpressed in three different forms of communication andmeaning construction: auditory, visual and text. Steen et al.
(2018) highlight that multimodal corpora have beenannotated for correlations involving mainly gesturecommunication and text data, and that computational
infrastructure for dealing with large multimodal corpora
has been under development. Both Turner and Steen leadan effort on this direction through the collaborative worksof The International Distributed Little Red Hen Lab™2, interms of establishing tools and methodology for analyzinglarge multimodal corpora, mostly exploring correlations
between spoken and gesture communication.

Although Cohn's (2016) model offers a coherent
framework to approach multimodal data, the author does
not incorporate any sort of fine-grained semantics into hismodel. Nonetheless, he recognizes the importance of using
one for adequately tackling the interrelations andinteractions between modalities and its components.
Given the lack of research incorporating fine-grainedmodels of semantic cognition into multimodal analyses,
the research presented in this paper aims to tackle the issueof meaning construction in multimodal settings,
specifically on what concerns the interaction between
audio (verbal expression transcribed into text) and video(not necessarily gesture communication), based on aprincipled structured model of human semantic cognition:
FrameNet. Such a model is presented next.

FrameNet Brasil, then, aims to establish an approachcomplementary to these works, since it is based on theestablishment of fine-grained frame-based relations
between the auditory and visual modalities, which is not
restricted to human gestures. Moreover, it builds on Cohn’s(2016) systematization of the semantic investigation in
multimodal data, according to the grammaticality of themodalities involved. It was used as a reference to evaluate
2

4. FrameNet and Frame-Based Semantic
Representation
Frames have a long history in both AI (Minsky, 1975) andlinguistics (Fillmore, 1982) as structured representations ofinterrelated concepts. In Frame Semantics, words are

See http://www.redhenlab.org

24understood relative to the broader conceptual scenes theyevoke (Fillmore, 1977). Hence, the expression child-safebeach, for example, is understood only in the context of ascene in which an Asset (the child) is exposed to somepotentially Harmful_event (a strong sea current, forexample).

LU via a subtype of quale elaborated on by a frame. Thoserelations will be discussed next.

4.2 Frame Mediated Ternary Qualia Relations
Although frame-to-frame and frame element-to-frame
relations already provide a fine-grained semantic
representation, they are unable to capture differences in thesemantics of a group of lexical units within one same
frame. Such differences are relevant for the semantic
representation of (multimodal) texts, as the pilot analysis
in this paper will demonstrate.3

This theoretical insight is the basis for lexicographic
resources such as Berkeley FrameNet and its sister projects
in other languages. Currently, there are FrameNet projects
for several languages besides English, including Chinese,
French, German, Italian, Japanese, Korean, Spanish,
Swedish and Brazilian Portuguese. These frame-based
resources have been applied to different Natural Language
Understanding problems, such as conversational Artificial
Inteligence (Vanzo et al. 2019) and paraphrase generation(Callison-Burch and Van Durme 2018).

The Generative Lexicon Theory (GLT) (Pustejovsky,
1995) arises as an approach to lexical semantics focusing
on the combinatorial and denotational properties of words,
as well as on peculiar aspects of the lexicon such aspolysemy and type coercion. The advance of the theory is
due to a dissatisfaction of many theoretical andcomputational linguists with the characterization of thelexicon as a closed and static set of syntactic,
morphological and semantic traits.

4.1 Frame-to-Frame and Frame Element-toFrame Relations
All framenets are composed of frames and their associatedroles in a network of typed relations such as inheritance,
perspective and subframe. The Risk_scenario frame
alluded to above, for example, is an umbrella frame forseveral more specific perspectivized frames such as
Being_at_risk (in which the Asset is exposed to arisky situation) and Run_risk (in which a Protagonist
puts an Asset at risk voluntarily). Each perspective may be
evoked by different words or by one same lexeme withdifferent syntactic instantiation patterns.

Qualia roles emerged as characteristics or different
possible context predication modes of a lexical item.
Pustejovsky and Jezek (2016) argue that qualia roles“indicate a single aspect of a word’s meaning, defined on
the basis of the relation between the concept expressed by
the word and another concept that the word evokes”. There
are four main qualia roles:
1.

Being_at_risk, for example, is evoked by adjectives
such as unsafe.a and nouns such as risk.n in constructions
like X is at risk. On the other hand, Run_risk is evokedby verbs such as risk.v and also by risk.n, but in a different
construction: Y has put X at risk (Fillmore and Atkins
1992). The database structure also features annotatedsentences, which attest the use of a given word in the target
frame.

2.
3.

On top of the frame-to-frame relations traditionally used inmost – if not every – FrameNet, FrameNet Brasil also
developed other types of relations aimed at enriching thedatabase structure. One of these relations links FEs to theframes licensing the lexical items that typically instantiate
those elements. Hence, the FE Tourist, in the Touring
frame, for instance, is linked via and FE-to-frame relation
to the People_by_leisure_activity frame.
Another relation connects core FEs to non-core FEs in thesame frame when the latter can act as metonymic
substitutes for the first (see Gamonal, 2017).

4.

The Formal quale is the relation that distinguishes anentity within a larger domain. Like a taxonomic
categorization, it includes characteristics likeorientation, shape, dimensions, color, position, size
etc.
The Constitutive quale is established between an
object and its constituents and the material involved inits production.
The Telic quale is associated with the purpose orfunction of the entity. We can expand this role to apersistent and prototypical property (function, purposeor action) of the entity (object, place or person).
The Agentive quale refers to the factors that are
involved in the origin or "coming into existence" of anentity. Characteristics included in this relation are thecreator, the artifact, the natural type and a causal chain.

Figure 1 exemplifies these qualia roles for the word
pizza.n.

Another group of relations developed by FrameNet Brasil
holds between LUs and is inspired by qualia roles, basedon Pustejovsky’s (1995) categorization. From the four
original qualia types – agentive, constitutive, formal andtelic – FrameNet Brasil has developed frame-mediatedternary relations in which a given LU is linked to anotherFig. 1. Qualia roles for pizza.n3
One could argue that the creation of even more fine-grainedframes would solve the problem mentioned here. Nonetheless,such a solution would be more time consuming and, at the sametime, split unnecessarily into different frames, plus sharing thesame background semantics and the same valence properties.

25

In Figure 1, we see that food.n is represented as formal_of
pizza.n, being a more general category to which pizza
belongs. The word eat.v is telic_of pizza.n since the latter
is made to be eaten. Because it is an ingredient used in it,
flour.n is constitutive_of pizza.n, while cook.n and pizza
restaurant.n are agentive_of pizza.n, because they
represent the person who causes the pizza to come into
existence, and the place that prototypically sells it,
respectively. Through qualia roles, a semantic relation isestablished between two words, providing a specific wordwith semantic features.

Fig. 2. Ternary qualia relations for pizza.n in the FrameNet
Brasil database

Figure 2 provides an example of this implementation. In
the FrameNet Brasil database the LU pizza.n has relation
with five other LUs via qualia. The LU pizza.n has an
Agentive relation (created_by) with pizza restaurant.n andcook.n. This relation is mediated by the
Cooking_creation frame, which relates pizza.n to the
FE Produced_food and pizza restaurant.n and cook.n to the
FE Cook.4 The LU pizza.n has also a Constitutive relation(is_made_of) with the LU flour.n, which is mediated by the
Ingredients frame, pizza.n being related to the FE Productand flour.n to the FE Material. The Formal relation(instance_of) is established via the Exemplar frame,
pizza.n being related to the FE Instance and food.n to the
FE Type. Finally, the Telic relation (meant_to) establishes
that pizza.n is related to the FE Tool, i.e. the object orprocess that has been designed specifically to achieve a
purpose, in the Tool_purpose frame. As for eat.v, it isrelated to the FE Purpose in the same frame.

One recurrent problem of working with qualia is that thefour relations just presented above are too generic. This has
led to the proposal of long lists of subtypes for each relation(Lenci et al. 2000). However, instead of incorporating
another list of relations to the FN-Br database, we use
frames in this same database as mediators of ternary qualia
relations to address both the lack of direct links between
LUs in the framenet model and the poor specificity ofqualia relations. In this innovative type of ternary relation,
two LUs, 1 and 2, are linked to each other via a given quale
using the background structure of frames as a way to makethe quale role denser in terms of semantic information. For
each quale, a set of frames was chosen from the FN-Br
database based on the aspects of such quale they specify.
LU1 would be related to an FE of the background frame,
whereas LU2 would be related to another FE of the same
frame. The frame would specify the semantics of therelation. The relations are represented in a directional
fashion, that is, they are to be interpreted as unidirectional,
although it is possible to create inverse relations.

Figure 3 presents a diagram which details the ternary
relations described for pizza.n .

Fig. 3. Diagram of the ternary qualia relations for pizza.n4
Because we also implement metonymy relations between FEs,
the peripheral FE Place can stand for the core FE Cook in the
Cooking_creation frame.

26

As a general policy, only core – and core unexpressed –FEs can be recruited as ternary qualia mediators. The
reason behind this policy relates to the very distinction
between core and non-core FEs in FrameNet methodology:
only core FEs are absolutely frame-specific, hence, they
are the only ones that actually differentiate one frame fromanother.

providing different profiling and perspective options formeaning construction, while also exploring alternative
connections between concepts in the FrameNet Brasilmodel. To test the validity of this hypothesis and, therefore,
the potential relevance of the project, an exploratory corpus
study was conducted and is described in the next section.

5. Exploratory corpus study and annotation
tool

The other policy refers to the degree of generality of frames
recruited as mediators for the ternary qualia relations.
Frames should be as general as possible, provided that they
do not conflict with or overgeneralize the quale. For
example, there are two more general frames in theinheritance chain leading to the Tool_purpose frame in
the FrameNet Brasil database: Inherent_purpose andRelation. The Relation frame overgeneralizes the
Telic quale, since it states that two Entities are related via
a Relation_type. Because no constraints are posited for theRelation_type, it could actually refer to any type of qualia.

FrameNet Brasil has been building a fine-grained semantic
infrastructure and developing resources and applications
for the Tourism domain (Torrent et al., 2014; Diniz da
Costa et al. 2018). Therefore, this exploratory study
reported in this paper refers to the same such domain.

5.1 The Corpus
The corpus is composed by the first season of the Brazilian
television travel show "Pedro pelo Mundo" (Pedro aroundthe world). There are 10 episodes, of 23 minutes each. In
each episode we see the host exploring a city, region or
country, highlighting its cultural and socioeconomic
aspects. The TV format combines voice-over sequences,
short interviews and video clip sequences in a wellintegrated script that offers rich composition of audio andvideo. For each episode, the audio transcription generates
approximately 200 sentences, which means 2000 sentences
for the entire season. Following the FrameNet Brasil fulltext annotation average of 6.1 annotation sets per sentence,
the annotation of the whole textual part of the corpus
should yield, when complete, about 12,200 lexical
annotation sets.

On the other hand, Inherent_purpose and
Tool_purpose differ in terms of the nature of the LU1.
In the former, it is a natural entity or phenomenon, while,in the latter, it is created by a living being. Such a
difference relates to Pustejovsky’s (2001) discussion on the
difference between natural and functional types, and,
therefore, the Tool_purpose frame should be used as
the mediator for the Telic relation between some manmade
item and its intended purpose, while theInherent_purpose frame should be used for the Telicrelation between a natural entity and the purpose that may
be imposed to it in some context.

5.2 Annotation Method
In the first step for the analysis conducted in the study, one
annotator manually annotated the audio transcript of one
random episode of the first season, using the FrameNet
Brasil Web Annotation Tool (Matos and Torrent, 2018) –an open source database management and annotation toolthat allows for the creation of frames and relations between
them – and following FrameNet’s guidelines for full-text
annotation. An example of the sort of annotation carriedout in this project is shown in Figure 4.

Given the possibilities enabled by the language model justdescribed, as pointed out before, the hypothesis being
investigated in this work is that, similarly to the way inwhich words in a sentence evoke frames and organize theirelements in the syntactic locality accompanying them,video scenes may also either (i) evoke frames and organize
their elements on the screen, or (ii) complement the frame
evocation patterns of the sentences they are attached to,Fig. 4. Example of a sentence annotated for frames in the FN-Br WebTool

27

LEXICAL UNIT
AUDIO FRAME
quando (when)Temp_collocation
pensa (think)
Cogitation
primeira (first)
Ordinal_numbers
coisa (thing)
Entityvem à mente (come to mind) Cogitation
homem (man)Peoplesaia (skirt)
Clothing
whiskyFood
escocês (Scottish)
Origingaita de fole (bagpipe)
Noise_makers

AUDIO TIME VIDEO FRAME
32.03 to 32.08 32.18 to 32.29 33.20 to 34.02 34.03 to 34.11 34.14 to 35.02 35.03 to 35.14 People_by_origin
35.17 to 35.29 Clothing
36.00 to 36.10 Food
36.11 to 36.23 36.24 to 37.23 Noise_makers

VIDEO TIME
36.12 to 37.12
36.12 to 37.12
35.02 to 36.12
36.12 to 37.12

SYNC
async
async
async
async
async
async
async
sync
async
sync

Table 1. Audio (text) and video annotation comparison.

After the annotation of the audio transcript has been carriedout, the same annotator annotated the video superimposedin the episodes for the same categories. Next, we contrasted
the annotations, searching for matching frames while also
considering the synchronicity or asynchronicity of theframes instantiated in both. The time stamps associated tothe audio transcripts and the video were taken as thecorrelational unit between the two modalities.

Although the latter inherits the first, this seems to be only
one of the correspondences between them.
The LU evoking the People_by_origin frame ishomem.n ‘man’. This LU does not bring any informationon the origin of the person, therefore, the frame evoked isthe most general of the People family of frames in
FrameNet Brasil. Nonetheless, in the video annotation, theannotator chose the People_by_origin frame, whichis evoked by the Object 7, as shown in Figure 5. The reason
behind this choice is the fact that the man depicted in thevideo right after the audio mentions homem de saia ‘manin skirt’ is wearing a kilt and playing a bagpipe, which area typical clothing and musical instrument of Scotland,
respectively. This combination of factors makes it very
likely to infer that what we see is a Scottish person.
Therefore, it makes possible to the annotator to choose thePeople_by_origin frame instead of the People
frame.

5.3 Sample Annotation Discussion
In the remainder of this section, we present and discuss thedata obtained from the multimodal annotation of one
sentence in the corpus, transcribed in (1).

(1) Quando a gente pensa na Escócia, a primeira coisa quevem à mente é homem de saia, whisky escocês e gaita
de fole.
‘When we think of Scotland, the first thing that comesto mind is man in skirt, Scottish whisky and bagpipe’.
The full annotation of (1) yielded ten lexical annotation
sets, while the annotation of the video it is superimposed togenerated four visual annotation sets. Table 1 presents
these data and how they synchronize – or not.

The first question that arises from this sample annotation is
how such a reasoning could be captured by some nonhuman tagger. Moreover, one could wonder whether thiskind of annotation is supported by the FrameNet Brasil
language model. Ternary qualia relations provide theanswer to both of them (see Figure 6).

The six lines in white present frames found only in theaudio. Because the annotation is audio-oriented, we did notannotate the frames that were present only in the video forthis pilot study, although we plan to include them in thenear future. The four lines highlighted in grey show thematches between frames annotated for both text/audio andvideo, although there is asynchrony in two of them and an
indirect match in one of those two. The asynchrony is dueto the fact that although evoked by both text/audio andvideo, the occurrences do not coincide in terms of time. In
both cases the text/audio evocation occurs before theelements appear visually on screen.

First, a subtype of the formal quale, mediated by the Typeframe connects the LUs kilt.n and saia.n ‘skirt’ inFrameNet Brasil. Second, a subtype of the constitutive
quale mediated by the Idiosyncrasy frame connects
the LU kilt.n, instantiating the FE Idiosyncrasy to the LU
escocês.n ‘Scot’, instantiating the FE Entity in this frame.
Finally,the
LU
escocês.nevokesthePeople_by_origin frame, which is precisely the one
evoked by the Object 7 in Figure 5.
Figure 6 presents a summary of the connections between
the multimodal elements annotated for (1), which can befound in FrameNet Brasil enriched language model.

The indirect correspondence between the frames People,
annotated for text/audio, and People_by_origin,
annotated for the video is more interesting though.

28

Fig. 5. Screenshot of the FN-Br Webtool Multimodal Annotation Module

Fig. 6. Summary of connections

6. Conclusions

with the original FN-Br WebTool, which could annotate
only text. The combination of both modules is crucial tomultimodal annotation, since timing has demonstrated to
be a key issue in measuring frame correlations acrossdifferent media. Thus, the Multimodal Module allows
annotators to choose frames and locate frame elements bothin the text and in the images, while keeping track of thetime span in which those elements are active in the video
and in the audio.

In this paper, we presented a tool and annotation scheme
for fine-grained annotation of multimodal corpora. Such atool controls for the synchronicity between different media
types and allows for cross-modality annotation, yielding, as
an annotation product, material that can shed light on therole of multimodality in language comprehension. This
new annotation module was projected to run combined29

There are several text annotation tools and several video
and/or image annotation tools. However, they do notcontrol for the synchronicity between different media types
nor allow for cross-modality annotation. Also, none ofthem are frame-based and, therefore, none of them yield, as
an annotation product, material that can shed light on therole of multimodality in language comprehension.

Matos, E. E. S., Torrent, T. T. (2018) FN-Br WebTool:
FrameNet Brasil Web Annotation Tool. INPI
Registration Number BR512018051603-3
McKevitt, P. (2003, January). MultiModal semantic
representation. In First Working Meeting of the SIGSEM
Working Group on the Representation of MultiModal
Semantic Information (pp. 1-16).
Minsky, M. (1975). A Framework for Representing
Knowledge: In Winston, PH (eds.), The Psychology ofComputer Vision, 211–277.
Pustejovsky, J. (1995). The Generative Lexicon.
Cambridge, USA, MIT Press.
Pustejovsky, J. (2001). Type construction and the logic ofconcepts. In: P. Bouillon and F. Busa. The language ofword meaning (pp. 91–123). Cambridge University
Press, New York, USA.
Pustejovsky, J. & Ježek, E. (2016). Qualia Structure. In:_____. Integrating Generative Lexicon and Lexical
Semantic Resources (pp. 11–55). Tutorial at The
Language Resources and Evaluation Conference (LREC
2016), Protoroz, Slovenia.
Steen, F. F., Hougaard, A., Joo, J., Olza, I., Cánovas, C. P.,
Pleshakova, A., ... & Turner, M. (2018). Toward an
infrastructurefordata-driven
multimodal
communication research. Linguistics Vanguard, 4(1).
Sun, C., Myers, A., Vondrick, C., Murphy, K., & Schmid,
C. (2019). Videobert: A joint model for video andlanguage representation learning. arXiv preprint
arXiv:1904.01766.
Torrent, T., Salomão, M. M., Campos, F., Braga, R., Matos,E., Gamonal, M., ... & Peron, S. (2014, August). Copa
2014 FrameNet Brasil: a frame-based trilingual
electronic dictionary for the Football World Cup. In
Proceedings of COLING 2014, the 25th International
Conference on Computational Linguistics: System
Demonstrations (pp. 10-14).
Turner, M. (2018). The Role of Creativity in Multimodal
Construction Grammar. Zeitschrift für Anglistik undAmerikanistik, 66(3), 357-370.
Vanzo, A., Bastianelli, E., & Lemon, O. (2019).
Hierarchical multi-task natural language understandingfor cross-domain conversational ai: HERMIT NLU.
arXiv preprint arXiv:1910.00912.

Future work includes the creation of a gold standard
multimodal annotated corpus that may be used in Machine
Learning applications such as Automatic Visual Semantic
Role Labeling and video indexing.

7. Acknowledgements
The FrameNet Brasil Computational Linguistics Lab isfunded by the CAPES PROBRAL (grant #88887.144043/2017-00) and the CAPES STINT (grant #99999.009910/2014-00) programs. Mr. Belcavello’s PhD
research is funded by the CAPES PDSE program (grant #88881.362052/2019-01). Mr. Diniz da Costa’s PhD
research was funded by the CAPES PROBRAL program(grant # 88887.185051/2018-00).

8. Bibliographical References
Aksoy, E. E., Ovchinnikova, E., Orhan, A., Yang, Y., &Asfour, T. (2017). Unsupervised linking of visualfeatures to textual descriptions in long manipulation
activities. IEEE Robotics and Automation Letters, 2(3),1397-1404.
Callison-Burch, C., & Van Durme, B. (2018). Large-Scale
Paraphrasing for Natural Language Understanding.
Johns Hopkins University Baltimore United States.
Cohn, N. (2016). A multimodal parallel architecture: A
cognitive framework for multimodal interactions.
Cognition, 146, 304-323.
Diniz da Costa, A., Gamonal, M. A., Paiva, V. M. R. L.,Marção, N. D., Peron-Corrêa, S. R., de Almeida, V. G.,... & Torrent, T. T. (2018). FrameNet-Based Modeling of
the Domains of Tourism and Sports for the Developmentof a Personal Travel Assistant Application. In
Proceedings of the Eleventh International Conference on
Language Resources and Evaluation (pp. 6-12).
Fillmore, C. J. (1977). The case for case reopened. Syntax
and semantics, 8(1977), 59-82.
Fillmore, C. J. (1982). Frame semantics. IN: Linguistic
society of Korea (org). Linguistics in the morning calm.
(1982) pp. 111-137. Hanshin Publishing Co., Seoul.
Fillmore, C. J., & Atkins, B. T. (1992). Toward a framebased lexicon: The semantics of RISK and its neighbors.
Frames, fields, and contrasts: New essays in semantic
and lexical organization, 103, 75-102.
Gamonal, M. A. (2017). Modelagem LinguísticoComputacional de Metonímias na Base deConhecimento Multilíngue (m.knob) da FrameNet
Brasil. Ph.D. Dissertation in Linguistics. Universidade
Federal de Juiz de Fora. Juiz de Fora.
Jackendoff, R., & Jackendoff, R. S. (2002). Foundations oflanguage: Brain, meaning, grammar, evolution. Oxford
University Press, USA.
Lenci, A., Busa, F., Ruimy, N., Gola, E., Monachini, M.,
Calzolari, N. and Zampolli, A. (2000). Simple Linguistic
Specifications, Deliverable D2, 1. Istituto di Linguistica
Computazionale Antonio Zampolli, Pisa, Italy.

30

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 31–40
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Combining Conceptual and Referential Annotation to Study Variation in
Framing
Marten Postma, Levi Remijnse, Filip Ilievski, Antske Fokkens, Sam Titarsolej, Piek Vossen
Vrije Universiteit Amsterdam
De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands{m.c.postma,l.remijnse,antske.fokkens,piek.vossen}@vu.nl
ilievski@isi.edu, s.titarsolej@gmail.com
Abstract
We introduce an annotation tool whose purpose is to gain insights into variation of framing by combining FrameNet annotation withreferential annotation. English FrameNet enables researchers to study variation in framing at the conceptual level as well through its
packaging in language. We enrich FrameNet annotations in two ways. First, we introduce the referential aspect. Secondly, we annotate
on complete texts to encode connections between mentions. As a result, we can analyze the variation of framing for one particular
event across multiple mentions and (cross-lingual) documents. We can examine how an event is framed over time and how core frame
elements are expressed throughout a complete text. The data model starts with a representation of an event type. Each event type has
many incidents linked to it, and each incident has several reference texts describing it as well as structured data about the incident.
The user can apply two types of annotations: 1) mappings from expressions to frames and frame elements, 2) reference relations frommentions to events and participants of the structured data.
Keywords: FrameNet, reference, annotation tool

1. Introduction

FrameNet-based Semantic Role Labelers.
The majority of the described efforts have mainly investigated frame annotations at the sentence level, as alreadyobserved by Fillmore, evidenced by the following quote:“since FrameNet has been working mainly on single sentences and has done nothing (yet) on connections withinwhole texts, the FrameNet database has nothing direct tooffer.” (Andor, 2010, p.168)
We aim at combining FrameNet annotations with referential annotations in order to analyze framing variationin texts describing an event. For this we need to extend
FrameNet annotations to the discourse level. Following
the data-to-text method described in Vossen et al. (2018),
we make use of the data acquisition platform described in
Vossen et al. (2020) to enable this type of research, forwhich we require:
1. a referential representation of an incident, i.e., an event
instance such as the 2012 Slovenian presidential election,with structured information about the location, time, andparticipants of the incident. 2. each incident to be taggedwith one or more event types, e.g., election. This makes
it possible to generalize over incidents of the same typeto learn which frames are typical. 3. different texts thatmake reference to the same incident, possibly written in
multiple languages with varying document creation times
and from different sources, which provides us with insights
into cross-lingual differences, source perspectives and theimpact of historical distance to the incident time (Cybulska
and Vossen, 2011). 4. an environment for efficient and consistent FrameNet and reference annotation to (given) structured data. This makes it possible to consider the framingof the incident throughout all texts that make reference to it
as a discourse unit.
In this paper, we introduce an annotation tool in which both
structured data about an incident and many reference texts
describing that one incident are simultaneously presented tothe user. This interface enables both FrameNet-based anno-

We construct narratives to describe events in the world
around us. The language that we use in those narratives
forms a lens that filters the actual components of those
events, e.g., their time, location, and participants, according to our perspectives. This way, narratives function not
only as structured collections of informative references tothe event, but also as collections of conceptual representations of that same event. For instance, texts describing
the attack on the World Trade Center express their references in various linguistic forms: ‘9/11’, ‘September 11 attacks’, ‘the 2001 attacks’ (all of which are timestamps withdifferent specificity), ‘a series of four coordinated terrorist
attacks’ (focus on the organizational aspect), ‘destructionof the towers in America’ (focus on the damaging aspect),etc. This set of references is a small share of all the various references in a growing portion of written texts. With
multiple texts written in different languages about a single
real-world event, one could analyze variation of framing ofan event by combining the conceptual and referential information. To perform such an analysis, we need both semantic resources to describe this conceptual information, andinformation about the components of the real-world event.
English FrameNet (Ruppenhofer et al., 2006) brought conceptual framing research to a computational setting. The
English lexicon made it possible to gain insight into the relationship between lexical items and the semantic framesthat they evoke. English FrameNet has also motivated researchers to create FrameNets in other languages such asin Japanese (Ohara et al., 2004), German (Burchardt et al.,2009), Swedish (Heppin and Gronostaj, 2012), Brazilian
Portuguese (Laviola et al., 2017), Spanish (Subirats andSato, 2003), French (Djemaa et al., 2016), Hebrew (Hayoun and Elhadad, 2016), and Latvian (Gruzitis et al., 2018).
Multiple annotation efforts resulted in many corpora andalso served as training, development, and test data to train

31

tations as well as referential linking to the incident that thereference texts make reference to. The analysis of conceptual and referential framing enriches research into variationin framing beyond the level of sentences and across different types of reference texts and languages.
This paper is structured as follows. In Section 2., we introduce English FrameNet and the related work on frame
annotation, followed by a discussion on combining conceptual and referential annotation in Section 3. We introduce
the annotation tool in Section 4. Finally, we discuss thepossibilities of the tool and future plans in Section 5., andconclude the paper in Section 6.

2.

the P ERPETRATOR frame element and ‘the children’ expresses the V ICTIM frame element. These frame elements
are called core frame elements, i.e., they need to be overtly
specified in order for the reader to process the frame. Other
types of frame elements, like ‘yesterday’, are peripheral,
meaning that they modify the frame.
When a core frame element is not present in the predicate scope, it is annotated as a Null Instantiation, which
we paraphrase as being unexpressed. In Subsection 3.1.,
we will elaborate on the phenomenon of unexpressed coreframe elements and how we propose to treat them. Frames
are situated in semantic space through frame-to-frame relations. In these relations, one frame is the more abstract superframe, and the other is the less abstract subframe. Oneof the relations through which Kidnapping is situated has
an inheritance frame-to-frame relationship with Committing crime, which is a conceptually corresponding yet lessspecific superframe . In Subsection 3.1., we will show how
frame-to-frame relations are used in FrameNet to explore
variation in framing.

Background

This section introduces the theoretical notions and implementations of FrameNet. Subsection 2.1. describes the relevant terminology and basic principles of frame semantics.
In Subsection 2.2., we provide a brief overview of currently
available frame annotation tools.

2.1.

FrameNet

2.2.

Frame semantics is a theory of linguistic meaning that assumes that the meaning of words is (partially) activatedthrough the frames that they evoke (Ruppenhofer et al.,2006). A frame is a schematic representation of a concept,
which is triggered by a lexical unit. This lexical unit is thesense of an expression in spoken or written discourse. For
the purpose of this paper, we model these conceptual relationships using RDF, as displayed in Figure 1. In this figure, the expression ‘kidnapped’ is disambiguated to a lexical unit via the ontolex:sense relationship (McCrae et al.,2017). The lexical unit evokes the frame Kidnapping via
the ontolex:evokes relationship.1

To the best of our knowledge, there are four publicly accessible and popular FrameNet annotation tools.
Annotation for English FrameNet (Ruppenhofer et al.,2006) is based on four layers. The target is the target
word that will be tagged with a frame label. Each constituent of the target is a candidate for frame element annotation. Each constituent is labeled with a grammatical
function and a phrase type. Grammatical functions aresyntactic relations that a constituent fulfills with respect tothe target word, e.g., object in the case of a verbal target
word. Phrase types indicate the syntactic category of theconstituent, e.g., noun phrase in the case of the constituent‘the man’. English FrameNet annotates one sentence at atime, in which one target word is labeled with a frame andits frame elements. An annotator first labels a target wordwith a frame label. Consequently, the grammatical function and the phrase type of each of the constituents of thetarget word are shown, which can be corrected manually.
The annotation guidelines are built upon the values of thegrammatical function and the phrase type of a constituent,i.e., these notions guide the annotator in deciding whichframe element to apply. In the online demo of the annotation tool, the grammatical functions and the phrase types
are not shown to avoid visual clutter.
The Global FrameNet Project (Torrent et al., 2018) builds
upon the annotation setup of English FrameNet. The corenovelty lies in moving to a multilingual setting. The aim
is to gain insight into how different languages frame translations of the same texts. This is accomplished by enriching the annotation by allowing annotators to specify why
a certain annotation was not possible based on the existing
frames, e.g., too specific or too general.
WebAnno (Eckart de Castilho et al., 2016) is a generic webbased annotation tool for semantic and syntactic structures,of which FrameNet annotation is one of the options. The
main emphasis of the tool is on the relation between syntactic and semantic structures, which drives the annotation
effort. The tool offers the possibility of introducing con-

Frame
Kidnapping

ontolex:evokes
LexicalUnit

expression
ontolex:sense"kidnapped"

Figure 1: RDF modeling for conceptual relationships. The
expression kidnapped expresses one of its senses as a lexical unit. This lexical unit evokes the Kidnapping frame.
Each frame is further associated with a characteristic set offrame elements that apply to the syntactic realization of thephrases dominated by the frame. We refer to Example (1).
(1)

FrameNet annotation tools

Kidnapping[P ERPETRATOR Two men] kidnapped [V ICTIM
the children] [T IME yesterday].

In this example, ‘kidnapped’ evokes Kidnapping, which
consists of several frame elements. ‘Two men’ expresses
1

We chose to not use the OntoLex (McCrae et al., 2017) relationship ontolex:reference since it might lead to confusion in
distinguishing between conceptual and referential relationships.

32straint rules in order to speed up annotation.
Salto (Burchardt et al., 2006) is a multi-level annotation
tool, which can be used to annotate FrameNet information. The annotation starts with a syntactic analysis of asentence. After determining the target word and labeling itwith a frame, the constituents can be tagged with a frame
element by means of drag and drop functionality.
All four described annotation tools provide the functionality to annotate FrameNet information. All of them startwith a syntactic analysis of the sentence and annotate
FrameNet information on top of that analysis. They differ
in what syntactic information is used and how this drives
the annotations.
The annotation tool described in this paper differs from theexisting annotation tools. The main difference stems fromthe choice of the central element in the tool. Whereas mostexisting tools use a sentence as the central element, our tool
makes use of an incident, i.e., an event, as the primary unit.
For each event, the user is shown both structured data andtexts that all make reference to the same incident. The purpose of the annotation effort is to make it possible to annotate mentions conceptually by linking to FrameNet, andreferentially towards the structured data.

3.

whether the two predicates in (2) refer to the same eventin the real world, which would entail that ‘Mary’ and ‘awoman’ refer to the same referent. The current tool aims
to implement structured data about the event, enabling theannotator to annotate on both the conceptual and the referential level. This allows us to investigate variation in a
broader sense: not just the framing of abstract concepts,
but also with respect to the referent.
In addition to variation in subframes belonging to a superframe, variation in framing can be observed when measuring the extent to which core frame elements are expressed.
According to FrameNet, core frame elements are necessarycomponents of a frame (Ruppenhofer et al., 2006). Yet,core frame elements often remain unexpressed in a sentence. FrameNet distinguishes between unexpressed coreframe elements that are left out due to syntactic constraints
or allowances (e.g., passivization, imperatives, pro-drop)and core frame elements that are left out due to anaphoric
reasons: they are already given as part of the surrounding
context of the sentence. See the examples below, takenfrom the FrameNet database (Ruppenhofer et al., 2006).
(3)a. [N EW LEADER Khan himself] was elected[ROLE a Congress party MP for Rampur].

Variation in Framing of Referenceb. Also [T IME on July 13] [S ELECTOR thecongress] elected [N EW LEADER Gorbarev][F UNCTION to head a commission [...]]In this section, we discuss variation in framing at the referential level. In Subsection 3.1., the means of variationin framing within the FrameNet paradigm is discussed, as
well as the merits of adding the referential level. In Subsection 3.2., we introduce a data model to facilitate referential
annotations as well as the main data resource used. In Subsection 3.3., we propose to add a relationship between an
expression and a frame in order to make the connection between the referent of an expression and its evoked frame
explicit.

3.1.

In both sentences in (3),the verb evokes
Change of leadership.
One of the core frame elements of this frame is S ELECTOR: the person or group‘responsible for a change in leadership’ (FrameNet lexical database; (Ruppenhofer et al., 2006)). In (3a), theS ELECTOR is unexpressed, which can be assigned to thesyntax, since passivized constructions allow speakers toleave out the agent. However, (3b) shows an active syntax,
while the S ELECTOR remains unexpressed. Moreover,
Change of leadership contains more core frame elements that are unexpressed in (3), such as, for instance,
O LD LEADER, O LD ORDER, and B ODY. These core frame
elements are regarded as part of the contextual knowledge
and not considered sufficiently relevant to express.
The current categorization of unexpressed core frame elements in FrameNet is syntax-driven, meaning that theseframe elements are analyzed within sentence boundaries.
When their absence is assumed to be bounded by sentenceexternal words or phrases, this information is not further
specified. The downside of this approach is that we do notgain insight into the way that these core frame elements are
linguistically encoded in the full discourse or if they are
encoded at all. Certain approaches address this problem
by going beyond the predicate scope in annotating unexpressed core frame elements. For instance, in SemEval2010 Task 10: Linking Events and Their Participants in
Discourse (Ruppenhofer et al., 2010), unexpressed coreframe elements were annotated outside of the scope of thepredicate in order to gain insight into the referents of these
unexpressed roles. A small number of texts from a workof Arthur Conan Doyle were annotated. There were three

Variation of framing in FrameNet

Within FrameNet, variation in framing can be observed bymeasuring the degree to which different subframes stand ina similar frame-to-frame relation to a superframe. See aclassic example below.
(2)

Change of leadershipa. Commerce sell
[T IME Yesterday,] [S ELLER John] sold[B UYER Mary] [G OODS a book].
b. Commerce buy[B UYER A woman] bought [G OODS a novel][P LACE in the shop].

In (2a), ‘sold’ evokes Commerce sell, with ‘Mary’ labeled
as the Buyer. In (2b), ‘bought’ evokes Commerce buy,with ‘a woman’ labeled as the B UYER. Both frames are related to the abstract frame Commerce goods transfer andshow a different perspective on this event. This way, variation in framing is measured on a conceptual level, comparing different variants of subframes related to one abstract
superframe.
In capturing variation in framing at a conceptual level, theannotation provides no knowledge concerning the referential level of the text. For instance, we lack insight as to33

Sentence 1, the target word ‘kidnapped’ evokes Kidnapping and also refers to the event in which Israeli soldiers
were kidnapped that is an instance of a kidnapping event.
Similarly, the noun ‘attack’ in Sentence 3 evokes Attack,and the event it refers to is an instance of the evoked frame.
On the contrary, the noun ‘kidnappers’ in Sentence 2 evokes
Kidnapping, but it refers to Hezbollah, which means thatthe instance that the expression refers to is not an instanceof the evoked frame but an instance of the concept person.
These role-designating nouns typically serve as a frame element of the verb they are governed by. Finally, the verb
‘can’ in Sentence 4 evokes Possibility, but it is unclear
what it refers to. There is no clear relationship between
the evoked frame and what the target word refers to.

participating systems. The results showed that this is a very
challenging task for Natural Language Processing systems.
Building upon the insights gained from SemEval-2010 Task
10, we consider the text as a cohesive narrative structure
and allow for annotation of core frame elements throughout the full text. We hypothesize that frames and frame
elements are evoked either directly or indirectly throughout the discourse in relation to the minimally required referential level. The tool, therefore, allows for annotationof frame and frame element relations at both the subword
level, e.g., compounds, as well as across sentences. Being
able to annotate frames and (core) frame elements throughout the text also allows us to analyze how different sources
frame the same situation differently and to explore the underlying factors of unexpression and other differences.
We discuss the implementation of this adaptation in Section 4. and its function in Subsection 5.1.

3.2.

instance"wd:Q41053

ontolex:evokes

Data model & main data resource

To facilitate the combination of conceptual and referential
annotation, we make use of a data model in which an incident is the central element.
Let R be a registry of real-world incidents, i.e., event instances. Let Ri be a real-world incident and let Ri ∈ R.
Each Ri contains structured data about the real-world incident, e.g., the period or time when the event happened, its
location, and information about which participants played arole and in which capacity. Let Et be an event type, which
is a categorization of a real-world incident. Finally, there
are reference texts, which are descriptions of real-world incidents (Ri ), e.g., a news article describing what happened.
Each Ri can have multiple reference texts.
The main data resource used in the annotation tool is Wikidata (Vrandečić and Krötzsch, 2014). We represent a Wikidata item, i.e., a description of an event instance, as an incident. Wikidata provides structured data about the incident, such as the time, location, and participants. Also, a
Wikidata item lists Wikipedia pages in multiple languages
that make reference to that specific Wikidata item, which
we represent as the reference texts. Finally, each Wikidata
item is tagged with one or multiple instance of (PropertyP31) relationships, which indicate the event type(s) of the
Wikidata item.

3.3.

Frame
Kidnapping

gaf:denotes

LexicalUnit

expression
ontolex:sense"kidnappers"

Figure 2: RDF modeling for conceptual and referential relationships without the dfn:isOfFrame relationship. The expression ‘kidnappers’ refers to the Wikidata item Q41053(Hezbollah) and evokes the Kidnapping frame. Since
Hezbollah is not an instance of kidnapping, no rdf:type relationship is assigned.
rdf:type

Frame

ontolex:evokes

dfn:isOfFrame

LexicalUnit

instance"wd:Q491346"

Kidnapping

gaf:denotes
expression

ontolex:sense"kidnapped"

Figure 3: RDF modeling for conceptual and referential relationships with the dfn:isOfFrame relationship. The expression ‘kidnapped’ refers to the Wikidata item Q491346(kidnapping of Kim Dae-jung) and evokes the Kidnapping
frame. Since kidnapping of Kim Dae-jung is an instance ofkidnapping, the rdf:type relationship is assigned and hence
also the dfn:isOfFrame relationship.

The connection between a frame and areferent of an expression

We extend the conceptual RDF relationships with referential ones, for which we use Figures 2 and 3 for clarification
purposes. For all target words in Table 1, it is the case thatthey evoke a frame. For most examples (all except Sentence 4), there is a referential link, which we model via
the gaf:denotes relationship as part of the GAF framework(Fokkens et al., 2014). In the case that the referent thatthe expressions refer to is an instance of the evoked frame,
we create an instance of relationship, for which we use therdf:type relationship, between the incident and the evoked
frame (see Figure 3). We make this relationship explicit byestablishing a http://rdf.cltl.nl/dfn/isOfFrame link between
the LexicalUnit and the Frame. In cases where the referent is not an instance of the evoked frame, the rdf:type andhttp://rdf.cltl.nl/dfn/isOfFrame relationship are absent (seeFigure 2).

English FrameNet (Ruppenhofer et al., 2006) uses theevoke relationship to relate lexical units to frames. However, the evoke relationship does not provide information
about the relationship between the event that an expression
refers to and the evoked frame. In our approach, we makethis information explicit in our annotation. We clarify thedistinction through the examples in Table 1.
Table 1 provides examples that highlight the relationshipbetween the evoked frames and the instances that theexpressions refer to. All examples originate from theEnglish Wikipedia page describing the 2006 Hezbollah
cross-border raid (Wikidata identifier Q2026122), in which
Hezbollah conducted a raid on Israeli territory in 2006.
During the raid, Hezbollah kidnapped Israeli soldiers. In

34

ID

Sentence

POS

EvokesRefers toRelation frame to incident

1

Six Western tourists were kidnappedby Al-Faran on 4 July 1995.

verb

Kidnapping

the kidnapping as part of
Wikidata item Q2026122

the referent is
an instance ofthe frame.

2

In December 1995, the kidnappers left
a note that they were no longer
holding the men hostage.

noun

Kidnapping

Hezbollah

the referent
is not an instanceof the frame

3

Top Hezbollah official Ghaleb Awali
was assassinated in a car bomb attackin the Dahiya in Beirut in July 2004

noun

Attackthe car bombing as part of
Wikidata item Q2026122

the referent is
an instance ofthe frame.

4

Israel can get to Hezbollah
anywhere in Lebanon

verb

Possibility-

there is noreferential relation

Table 1: Examples sentences taken from the English Wikipedia page describing the 2006 Hezbollah cross-border raid
(Wikidata identifier Q2026122). The first column indicates the example sentence identifier, the second shows the example
sentence, the third provides the part of speech tag of the target word, the fourth which frame the target word evokes, the fifth
column provides information about what the target word refers to, and the last column indicates the relationship between
the evoked frame and its referent.

4.

Data-to-text Annotation Tool

In this section, we introduce the annotation tool, for which
we present the workflow in Figure 4. The tool starts fromdata that is aggregated through the MWEP platform described in Vossen et al. (2020). It contains structured dataon incidents of a specific type, e.g., murders, elections,sports events, etc. paired with reference texts linked to thespecific incidents.
After the login, the annotator first selects an event type, after which a list of incidents is given. Next, the annotator
can select a specific incident from the list, after which thestructured data is shown with all the reference texts.
The user first has the option to correct the tokenization in
the texts to ensure that multi-words and compounds are correctly represented. After deciding on the markables, three
types of annotation types can be chosen: Frame, Frame
Element, or Reference. With Frame and Frame Element,
the user can annotate predicates with their corresponding
frames and frame elements. In contrast, Reference is used
to link textual mentions to the structured data of the incident. This enables coreferential mentions, i.e., linked to thesame incident, to obtain different frame annotations, whichforms the basis for analyzing variation.
The front-end of the tool makes use of Bootstrap CSS2
and jQuery3 , and the server-side operations are handled by
Node.js4 .
In Subsection 4.1., we introduce the resources used in thetool. The Subsections 4.2., 4.3., 4.4., 4.5., 4.6., 4.7., and4.8. explain the main components of the annotation tool.

Login

Incident Selection

Markable Correction

Annotation typeMarkable Selection

Frame

Frame
Element

Reference

Frame
Selection

Frame
Element
Selection

Reference
Selection

Frame
Relation
Type

Save
Frame
Element

Save
Reference
Annotation

4.1.

Resources

In this subsection, we introduce the resources used in the
annotation tool, i.e., the lexicon and the data.
lexicon We make use of the canonical version 1.7 ofFrameNet (Fillmore and Baker, 2010; Ruppenhofer et al.,2006). All annotations make use of a Resource Description
Framework (RDF) of FrameNet, for which the two mostcommon resources are Framester (Gangemi et al., 2016)

Save
Frame
Annotation2

https://getbootstrap.com/docs/3.4/css/
https://jquery.com/
4https://nodejs.org/en/
3

Figure 4: Annotation workflow

35

and PreMOn (Corcoglioniti et al., 2016). We chose to use
PreMOn since the project was more active.5
data acquisition We have developed a data architecture(Vossen et al., 2020) to obtain and represent the data according to the data model as presented in Subsection 3.2.,
for which we primarily make use of Wikidata (Vrandečić
and Krötzsch, 2014).
preprocessing spaCy6 is used for sentence splitting, tokenization, and part of speech tagging, for which models in
English, Dutch, and Italian are used. The preprocessing isstored in the NLP Annotation Format (NAF) (Fokkens etal., 2014), a stand-off, multilayered annotation schema forrepresenting linguistic annotations.7 We retrained OpenSESAME (Swayamdipta et al., 2017) to tag the reference
texts with FrameNet frames.8

4.2.

In this step of the annotation process, the user can correct
the automatic tokenization by indicating which combination of tokens serve as phrasal verbs or idioms. Also, theuser can decompose compounds into separate components.
For cases in which multiple tokens should be merged, theuser clicks on the tokens that are part of the construction
and indicates whether they belong to the category of phrasal
verbs or idioms. We follow English FrameNet in assigning
the part of speech tag V to phrasal verbs and idio for idioms.
If the user now clicks on one of the tokens of a construction, all tokens that belong to it are selected. It is no longer
possible to annotate parts of the construction as predicatesor frame elements. Also, any annotation on the level of theindividual parts of the construction is deprecated and will
no longer be used nor rendered in the tool.
Annotators are also asked to detect endocentric compounds, i.e., compounds consisting of a grammatical head
and a modifier (Lexicon of Linguistics, 2020a), which makeit possible to annotate components of the compounds withframes and frame elements. After clicking on the detected
endocentric compound, the user is asked to indicate thecomponents of the compound as well as which component
serves as the frame-evoking unit. For each component, theuser needs to indicate the lemma and the part of speech according to the Universal Dependencies version 2 (Nivre etal., 2017) part of speech tagset. After specifying a compound, the user can now click on the separate componentsof the compound and can no longer click on the compound
as a whole. Also, any previous annotation of the compound
as a whole is ignored and will no longer be rendered.

Login

A unique session identifier is created for each annotator foreach annotation session. Each annotation will then be accompanied by this session identifier and the timestamp of
the annotation, which allows analyses per annotation session and per annotator. No annotations are removed. Automatically generated annotations are represented in the sameway using identifiers and timestamps.

4.3.

Incident Selection

The user will first have to choose a specific data release,e.g., version 1.0. From this data release, an event type ischosen, e.g., murder (Wikidata identifier Q132821). From
the available incidents that belong to the chosen event type,one incident is selected.
After clicking on Load Incident, the user is presented withthe structured data about the incident, e.g., the location,
time, and participants. Also, all available reference texts
that make reference to the selected incident are shown, possibly in multiple languages.
The existing annotations for each reference text are highlighted. The user can observe the difference between manual and automatic annotations, which is designed such thatthe user can focus more on validating than on full-text annotation.

4.4.

4.5.

Annotation TypeThe next step involves deciding which type of annotation
to perform. There are three options: Frame (see Subsection
4.6.), Frame Element (see Subsection 4.7.), and Reference(see Subsection 4.8.). For each annotation, this is the firststep. Note that our tool does not assume that there is alreadya FrameNet lexicon beforehand.

4.6.

Frame

The goal of the Frame annotation type is to annotate predicates with FrameNet frames as well as to indicate the Frame
Relation Type. After selecting a markable, the user clicks
to observe a dropdown list in which all FrameNet frames
are divided into four groups: 1. Typical frames this category contains the frames that are typically expected given
the type of the selected incident, e.g., Killing and Offenses
for the event type murder. 2. candidate frames for lemmaand part of speech: the candidate frames given the lemmaand part of speech of the markable are shown here. 3. candidate frames for lemma: the candidate frames given thelemma of the markable are shown here. 4. other all other
FrameNet frames are shown here.
When a user selects a frame from the dropdown list, more
information about the frame is shown in the right panel.
Also, the user has to indicate the Frame Relation type (seeSubsection 3.3.). In the case that the incident to which theexpression refers is an instance of the evoked frame, theuser selects isOfFrame. Otherwise, the user selects evoke.

Markable Correction

Linguistic phenomena in which there is a many-to-many
relationship between a token and a concept are a crucial
problem for language technology (Sag et al., 2002). Idioms, phrasal verbs, and compounds are cases in which thisoccurs. In at least two phenomena, i.e., idioms and phrasal
verbs, multiple tokens combined refer to one concept or semantic unit (Lexicon of Linguistics, 2020b; Quirk, 2010).
In contrast, compounds consist of one token, but they canevoke multiple frames and frame element relations.
5
We downloaded the dataset from the following link:
https://knowledgestore.fbk.eu/files/premon/
dataset/latest/premon-2018a-fn17-noinf.tql.
gz.
6
https://spacy.io/7
https://github.com/newsreader/NAF
8
Our wrapper is available at: https://github.com/
cltl/run_open-sesame.

36

Figure 5: Snapshot of the data-to-text annotation tool.
V ICTIM is mentioned in the sentence before the target sentence. Only after all core frame elements have been annotated, the user is able to switch to a different Annotation
Type.
After clicking Save, the annotation is added to the corresponding NAF file of the reference text using the PreMOn
URI identifier for the frame element.

After clicking Save, the annotation is added to the corresponding NAF file of the reference text using the PreMOn
URI identifier for the frame.
Finally, we allow the user to click multiple predicates atonce and annotate a batch of predicates with the same framelabel and frame relation type.

4.7.

Frame Element

4.8.

The goal of the Frame Element Annotation Type is first to
indicate which frame elements are found in the predicate
scope. If core frame elements are not found in the scopeof the predicate, the user is asked to try to annotate them inthe context surrounding the predicate.
Given that a user has previously annotated a predicate witha frame label, the user can now also annotate frame elements for this predicate. The user clicks on the markableand selects the frame element from a dropdown list. After annotating at least one frame element for a frame, e.g.,
K ILLER for Killing, a table is shown in the right panel ofthe tool, of which an example is shown in Table 2.
Frame Element

Type

Annotated

Expressed

K ILLERV ICTIMC AUSEM EANS

Core
Core
Core
Core
Core

true
false
false
false
false

true
false
false
false
false

I NSTRUMENT

Reference

Alongside the reference texts, the user is shown structured data about the main incident, as shown in the top
right corner of Figure 5. The structured data table consists of five pieces of information. The event type of theincident is shown, which we obtain from the Wikidata instance of relationship (Property P31) as well as the Wikidata item identifier. Also, we categorize the properties of a
Wikidata item into three classes and model them using the
Simple-Event-Model (SEM (Van Hage et al., 2011)). Locations are mapped to sem:hasPlace, temporal expressions
to sem:hasTimeStamp, and participants to sem:hasActor.
The user clicks on some tokens and then indicates, by clicking on a link in the structured table, that the markable refers
to the reference, e.g., ‘The kidnapper’ refers to Hezbollah(Wikidata item Q41053) which is expressed in RDF using
a gaf:denotes relation.
The annotator can also modify the structured data table.
He or she can add and remove values to sem:hasPlace,
sem:hasTimeStamp, and sem:hasActor, provided that theuser provides Wikidata items as values.

Table 2: Frame Element Annotation

5.

Table 2 presents the information shown to the user during
the frame element annotation phase. The user can keep
track of all frame element annotations for an active frame.
The annotator is asked to attempt to find evidence for eachcore frame element in the predicate scope. If the frame element is unexpressed, the user is asked to look for evidenceof the frame element in the surrounding context, e.g., theDiscussion

In this section, we elaborate on the ways in which the current tool supports us in providing frame annotations of texts
referring to a real-world event. In Subsection 5.1., we discuss the extent to which the tool in its current state directs
the annotator. In Subsection 5.2., we present future plans
for the tool to capture inferred frames.

37

ID

Sentence

EvokesFrame Element

1

The Tunisian man who prosecutors say
perpetrated last month’s terrorist attack [...]

Committing crime

P ERPETRATOR

2

he ploughed a truck into a crowded Christmas market

ImpactAGENT

3

Amri carried out the attack

Intentionally act

AGENT

4

he hijacked a truck

PiracyP ERPETRATOR

5

he [...] shot its Polish driver

Hit target

AGENT

6

he [...] drove it into the crowded market

Operate vehicle

D RIVER

Table 3: Example sentences taken from reference texts referencing Anis Amri (Wikidata identifier Q28052669), the agentof the 2016 Berlin Attack (Wikidata identifier Q28036573). The first column indicates the identifier, the second column
shows the example sentence with the reference to Anis Amri in italics and the frame-evoking predicate in bold, the value ofthe third column is the evoked frame, and the last column shows which frame element the reference to Anis Amri expresses.

5.1.

Functionality

user first annotates Example (4a), ‘elections’ is marked asa predicate evoking Change of leadership, which contains
many core frame elements, which are ROLE, B ODY, F UNC TION , N EW L EADER , O LD L EADER , O LD O RDER , andS ELECTOR. Example (4a) contains annotations for threeframe elements, which are Role, Location, and Time. Roleis the only core frame element with an annotation in Example (4a) out of the many core frame elements. No evidence
is found for the other core frame elements. The next stepfor the user is to find mentions of these unexpressed coreframe elements. Evidence for the specific old leader of theelection, i.e., core frame element O LD L EADER, is foundin Example (4b). The annotator continues to look for evidence for the remaining core frame elements.
Finally, we provide the annotator with a list of typicalframes. The list consists of frames that are relevant for theperception of the event, e.g., Offenses and Use firearm fora murder event. Some of those typical frames are even necessarily evoked, e.g., Killing in the case of a murder event.
The main rational to use this list is to restrict the annotations to the most important mentions of the event, basedon the assumption that only a subset of frames is used todescribe an event of a specific type. Moreover, if the obligatory typical frames are not found in a text, this then leads
to questions about evocation through inference.

The current annotation tool enables the annotator to perform two parallel annotations. In choosing Frame or Frame
Element as the Annotation Type, the annotator traditionally performs frame annotations. In addition, he or she
can select the Reference Annotation Type to mark words
that refer to the structured data. These parallel annotations
can be performed on the same expression in the text, whichmeans that this expression is annotated as both contributing
to a specific frame and simultaneously referencing a component of the real-world event. The resulting annotation
scheme displays which words refer to the structured data as
well as how these words frame the data. In other words,
variation in framing across texts can be measured concerning a fixed real-world referent. For instance, all sentencesin Table 3 make reference to the 2016 Berlin attack. In
these sentences, Anis Amri is being referred to by the expressions ‘The Tunisian man’, ‘he’, and ‘Amri’. However,these expressions belong to different predicates, each evoking a different frame. Hence, each expression is labeledwith a different frame element, which provides insight into
how the referent is framed.
A second function of the tool is to annotate unexpressedcore frame elements beyond the scope of the predicate.
Given the assumption that frames construct a cohesive narrative in referring to a main event, we expect these coreframe elements to occur somewhere in the reference text
to support the reader’s understanding of the narrative. If
no evidence is found for a core frame element, this raises
questions about its retrievability: whether it is inferred fromworld knowledge or expressed in other reference texts.
Example 4 highlights this functionality.
(4)

5.2.

Desired functions

During preliminary annotation experiments with the tool,
we found that a substantial portion of the frames indicatedby the typical frames are not derived from reference texts.
For instance, Killing is not evoked by lexical units in a reference text of a murder event. We argue that the necessary
frames are still activated in the text, but that they are derived
through pragmatic inference rather than lexical evocation.
These inferences are derived from linguistic cues that are
not marked as lexical units within FrameNet. Hence, theseare different inferences than the ones discussed by Changet al. (2002), who use the notion of inference in FrameNet
for a frame that is inevitably processed during the evocationof another frame, e.g. Commerce buy is always activatedwith the evocation of Commerce sell and vice versa. These
inferences are actually based on a frame that is evoked by alexical unit. Also, most inferences we detected could not

Change of leadershipa. [ROLE Presidential] elections were held
[L OCATION in Slovakia] [T IME in March2019].
b. [O LD LEADER Incumbent President Andrej
Kiska] did not run for a second term.

The sentences in (4) form the onset of a text referring to thepresidential election of Slovakia in 2019. Assuming that the38

be derived by any current frame-to-frame relations. See
the sentence in (5), which refers to the 2016 Berlin Attack(Wikidata identifier Q28036573).
(5)

Chang, N., Petruck, M. R. L., and Narayanan, S. (2002).
From Frames to Inference. In Proceedings of the FirstInternational Workshop on Scalable Natural Language
Understanding.
Corcoglioniti, F., Rospocher, M., Aprosio, A. P., andTonelli, S. (2016). PreMOn: a Lemon Extension for
Exposing Predicate Models as Linked Data. In Proceedings of the Tenth International Conference on Language
Resources and Evaluation (LREC’16), pages 877–884,
Portorož, Slovenia, May. European Language Resources
Association (ELRA).
Cybulska, A. and Vossen, P. (2011). Historical Event Extraction from Text. In Proceedings of the 5th ACL-HLTWorkshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities.
Eckart de Castilho, R., Mújdricza-Maydt, É., Yimam,
S. M., Hartmann, S., Gurevych, I., Frank, A., and Biemann, C. (2016). A Web-based Tool for the Integrated
Annotation of Semantic and Syntactic Structures. In
Proceedings of the Workshop on Language Technology
Resources and Tools for Digital Humanities (LT4DH).
Fillmore, C. J. and Baker, C. (2010). A Frames Approach
to Semantic Analysis. In The Oxford handbook of linguistic analysis. Oxford University Press.
Fokkens, A., Soroa, A., Beloki, Z., Ockeloen, N., Rigau,
G., Van Hage, W. R., and Vossen, P. (2014). NAF and
GAF: Linking Linguistic Annotations. In Proceedings
10th Joint ISO-ACL SIGSEM Workshop on Interoperable
Semantic Annotation.
Gangemi, A., Alam, M., Asprino, L., Presutti, V., and Recupero, D. R. (2016). Framester: A Wide Coverage Linguistic Linked Data Hub. In European Knowledge Acquisition Workshop.
Levinson, S. C. (1983). Pragmatics. Cambridge Textbooks in Linguistics. Cambridge University Press.
Lexicon of Linguistics. (2020a). Exocentric compound.
https://lexicon.hum.uu.nl/?lemma=
Endocentric+compound&lemmacode=778&
lemma=Endocentric+compound&lemmacode=
778.
Lexicon of Linguistics. (2020b). Idiom. https://
lexicon.hum.uu.nl/?lemma=Endocentric+
compound&lemmacode=778&lemma=
Endocentric+compound&lemmacode=778.
McCrae, J. P., Bosque-Gil, J., Gracia, J., Buitelaar, P., andCimiano, P. (2017). The Ontolex-Lemon model: development and applications. In Proceedings of eLex 2017conference.
Nivre, J., Agić, Ž., Ahrenberg, L., Aranzabe, M. J., Asahara, M., Atutxa, A., Ballesteros, M., Bauer, J., Bengoetxea, K., Bhat, R. A., Bick, E., Bosco, C., Bouma, G.,
Bowman, S., Candito, M., Cebiroğlu Eryiğit, G., Celano,
G. G. A., Chalub, F., Choi, J., Çöltekin, Ç., Connor, M.,
Davidson, E., de Marneffe, M.-C., de Paiva, V., Diaz deIlarraza, A., Dobrovoljc, K., Dozat, T., Droganova, K.,
Dwivedi, P., Eli, M., Erjavec, T., Farkas, R., Foster, J.,
Freitas, C., Gajdošová, K., Galbraith, D., Garcia, M.,
Ginter, F., Goenaga, I., Gojenola, K., Gökırmak, M.,
Goldberg, Y., Gómez Guinovart, X., Gonzáles Saave-

Killing[C AUSE a truck] was ? deliberately ? driven into
the Christmas market, ? leaving [V ICTIM 12 people] ? dead

Although the words in (5) separately do not evoke Killing,
the sum of the components ‘driven’, ‘leaving’, and ‘dead’
activate this frame by means of entailment. Moreover, ‘deliberately’ acts as a cue from which Offenses could bederived. The way these frames are activated can only be
traced by complementing the lexical semantic analysis offrame semantics with a pragmatic analysis in terms of inference. If the annotator is guided in pointing out the linguistic cues, the different ways in which frames are inferredfrom these cues can be schematized. The data can then be
analyzed with respect to their pragmatic type, e.g., entailment, implicature (Levinson, 1983); and the possible factors that account for the inference e.g., historical distanceof the publication or genre conventions.
One of the next steps in the development of the annotation
tool is to implement an inferred frame layer for the annotator that allows him or her to mark linguistic cues fromwhich a frame is pragmatically derived, on top of the traditional FrameNet annotation module. After targeting words
as lexical units, this inferred frame layer will ask the annotator to mark n linguistic cues in the text that might derive
any of the remaining unannotated typical frames. We refer
to Remijnse and Minnema (2020) for a detailed descriptionof this proposal.

6. Conclusion
In this paper, we introduced an annotation tool in which annotations can be made to both the conceptual and referential level. For an event type, the tool delivers a collection ofincidents, each accompanied with structured data and reference texts in different languages. The annotator can marktargets in the texts to frame-annotate and mark the same targets to annotate referential relations to the structured data.
From the output of this annotation process, patterns of variation in framing can be extracted concerning reference to asingle referent.

7.

Acknowledgements

The research reported in this article was funded by the
Dutch National Science organisation (NWO) through the
project Framing situations in the Dutch language.

8. Bibliographical References
Andor, J. (2010). Discussing Frame Semantics: The Stateof the Art: An Interview with Charles J. Fillmore. Review of Cognitive Linguistics. Published under the auspices of the Spanish Cognitive Linguistics Association,
8(1):157–176.
Burchardt, A., Erk, K., Frank, A., Kowalski, A., Pado,
S., and Pinkal, M. (2006). SALTO-A Versatile MultiLevel Annotation Tool. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006).

39

dra, B., Grioni, M., Grūzı̄tis, N., Guillaume, B., Habash,N., Hajič, J., Hà Mỹ, L., Haug, D., Hladká, B., Hohle,
P., Ion, R., Irimia, E., Johannsen, A., Jørgensen, F.,
Kaşıkara, H., Kanayama, H., Kanerva, J., Kotsyba, N.,
Krek, S., Laippala, V., Lê H`ông, P., Lenci, A., Ljubešić,N., Lyashevskaya, O., Lynn, T., Makazhanov, A., Manning, C., Mărănduc, C., Mareček, D., Martı́nez Alonso,
H., Martins, A., Mašek, J., Matsumoto, Y., McDonald,R., Missilä, A., Mititelu, V., Miyao, Y., Montemagni, S.,
More, A., Mori, S., Moskalevskyi, B., Muischnek, K.,
Mustafina, N., Müürisep, K., Nguy˜ên Thi., L., Nguy˜ênThi. Minh, H., Nikolaev, V., Nurmi, H., Ojala, S., Osenova, P., Øvrelid, L., Pascual, E., Passarotti, M., Perez,
C.-A., Perrier, G., Petrov, S., Piitulainen, J., Plank, B.,
Popel, M., Pretkalniņa, L., Prokopidis, P., Puolakainen,
T., Pyysalo, S., Rademaker, A., Ramasamy, L., Real,L., Rituma, L., Rosa, R., Saleh, S., Sanguinetti, M.,
Saulı̄te, B., Schuster, S., Seddah, D., Seeker, W., Seraji, M., Shakurova, L., Shen, M., Sichinava, D., Silveira,N., Simi, M., Simionescu, R., Simkó, K., Šimková, M.,
Simov, K., Smith, A., Suhr, A., Sulubacak, U., Szántó,Z., Taji, D., Tanaka, T., Tsarfaty, R., Tyers, F., Uematsu,
S., Uria, L., van Noord, G., Varga, V., Vincze, V., Washington, J. N., Žabokrtský, Z., Zeldes, A., Zeman, D.,
and Zhu, H. (2017). Universal Dependencies 2.0. LINDAT/CLARIN digital library at the Institute of Formaland Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
Quirk, R. (2010). A Comprehensive Grammar of the English Language. Pearson Education India.
Remijnse, L. and Minnema, G. (2020). Towards
Reference-Aware FrameNet Annotation. In The International FrameNet Workshop 2020.
Ruppenhofer, J., Ellsworth, M., Schwarzer-Petruck, M.,
Johnson, C. R., and Scheffczyk, J. (2006). FrameNet
II: Extended theory and practice. FrameNet Project.
Ruppenhofer, J., Sporleder, C., Morante, R., Baker, C., andPalmer, M. (2010). SemEval-2010 Task 10: Linking
Events and Their Participants in Discourse. In Proceedings of the 5th International Workshop on Semantic Evaluation.
Sag, I. A., Baldwin, T., Bond, F., Copestake, A., andFlickinger, D. (2002). Multiword Expressions: A Painin the Neck for NLP. In Computational Linguistics andIntelligent Text Processing.
Swayamdipta, S., Thomson, S., Dyer, C., and Smith,
N. A. (2017). Frame-Semantic Parsing with SoftmaxMargin Segmental RNNs and a Syntactic Scaffold.
arXiv preprint arXiv:1706.09528.
Torrent, T. T., Ellsworth, M., Baker, C., and Matos, E.
(2018). The Multilingual FrameNet Shared Annotation
Task: a preliminary report. In Proceedings of the 11th
International Conference on Language Resources and
Evaluation (LREC 2018).
Van Hage, W. R., Malaisé, V., Segers, R., Hollink, L., andSchreiber, G. (2011). Design and use of the Simple
Event Model (SEM). Web Semantics: Science, Services
and Agents on the World Wide Web, 9(2):128–136.
Vossen, P., Ilievski, F., Postma, M., and Roxane, S. (2018).
Don’t Annotate, but Validate: a Data-to-Text Method for

Capturing Event Data. In 11th edition of the Language
Resources and Evaluation Conference (LREC 2018).
Vossen, P., Ilievski, F., Postma, M., Fokkens, A., G., M.,
and L., R. (2020). Large-scale Cross-lingual Language
Resources for Referencing and Framing. In 12th Editionof the Language Resources and Evaluation Conference(LREC 2020).
Vrandečić, D. and Krötzsch, M. (2014). Wikidata: a Free
Collaborative Knowledge Base. Communications of theACM, 57(10):78–85.

9.

Language Resource References

Burchardt, A., Erk, K., Frank, A., Kowalski, A., Padó,S., and Pinkal, M. (2009). FrameNet for the Semantic Analysis of German: Annotation, Representation andAutomation. Multilingual FrameNets in Computational
Lexicography: methods and applications, 200:209–244.
Djemaa, M., Candito, M., Muller, P., and Vieu, L. (2016).
Corpus Annotation Within the French FrameNet: a
Domain-by-domain Methodology. In The 10th International Conference on Language Resources and Evaluation (LREC 2016).
Gruzitis, N., Nespore-Berzkalne, G., and Saulite, B.
(2018). Creation of Latvian FrameNet based on Universal Dependencies. In Proceedings of the International
FrameNet Workshop.
Hayoun, A. and Elhadad, M. (2016). The Hebrew
FrameNet Project. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016).
Heppin, K. F. and Gronostaj, M. T. (2012). The Rocky
Road towards a Swedish FrameNet-Creating SweFN. In
The 8th international conference on Language Resources
and Evaluation (LREC 2012).
Laviola, A., Lage, L., Marção, N., Tavares, T., Almeida,V., Matos, E., and Torrent, T. (2017). The Brazilian
Portuguese Constructicon: Modeling Constructional Inheritance, Frame Evocation and Constraints in FrameNet
Brasil. In 2017 AAAI Spring Symposium Series.
Ohara, K. H., Fujii, S., Ohori, T., Suzuki, R., Saito, H., andIshizaki, S. (2004). The Japanese Framenet Project: An
Introduction. In Proceedings of LREC-04 Satellite Workshop “Building Lexical Resources from Semantically Annotated Corpora”(LREC 2004).
Ruppenhofer, J., Ellsworth, M., Schwarzer-Petruck, M.,
Johnson, C. R., and Scheffczyk, J. (2006). FrameNet
II: Extended theory and practice. FrameNet Project.
Subirats, C. and Sato, H. (2003). Surprise! Spanish
FrameNet. In Proceedings of the Workshop on Frame Semantics at the XVII. International Congress of Linguists.

40

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 41–47
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

FrameNet Annotations Alignment using Attention-based Machine Translation
Gabriel Marzinotto

Orange Labs,
22300 Lannion, France
gabriel.marzinotto @orange.com
Abstract
This paper presents an approach to project FrameNet annotations into other languages using attention-based neural machine translation(NMT) models. The idea is to use a NMT encoder-decoder attention matrix to propose a word-to-word correspondence between thesource and the target languages. We combine this word alignment along with a set of simple rules to securely project the FrameNet
annotations into the target language. We successfully implemented, evaluated and analyzed this technique on the English-to-French
configuration. First, we analyze the obtained corpus quantitatively and qualitatively. Then, we use existing FrameNet corpora to assert
the quality of the translation. Finally, we trained a BERT-based FrameNet parser using the projected annotations and compared it to a
BERT baseline. Results show modest performance gains in the French language, giving evidence to support that our approach could
help to propagate FrameNet data-set on other languages. Moreover, this label projection approach can be extended to other sequencetagging tasks with minor modifications.
Keywords: FrameNet, Machine Translation, Cross-lingual Annotation Transfer, Cross-lingual FrameNet Parsing

1. Introduction
Frame Semantics (Fillmore, 1976) and the FrameNet (Fillmore, 2006) dictionary constitute a valuable resource anda very successful semantic representation scheme, widelyadopted and adapted for many NLP applications. For
many years a lot of works have studied the adaptability of FrameNet into other languages, showing that many
frames are entirely cross-lingual (Gilardi and Baker, 2018).
At the same time, FrameNet adaptations for more than
15 languages have been arising. Among these languages,
we count Chinese, Danish, Dutch, Finnish, Portuguese,
French, German, Hebrew, Hindi, Japanese, Korean, Latvian and Spanish.
Most of these projects use either human translation (e.g.
Finish (Pedersen et al., 2018) and Danish (Lindén et al.,2017)) while others use semi-automatic alignments (Hayoun and Elhadad, 2016). Also, projects focus on translating
the lexical units using bilingual alignments and they tend
to deliver a set of annotated examples that is significantly
smaller than FrameNet.
More recently, we have seen an initiative to build a multilingual FrameNet lexicon (Gilardi and Baker, 2018), or atleast to add relations between similar frames across different languages. Their approach is also based on lexical unittranslation using bilingual dictionaries. The objective is to
give some guidelines to counter the small divergences weexperience today, which are due to the separated evolutionof each project.
In this paper, we propose a slightly different exampledriven approach to bootstrap FrameNet corpora in new languages using Neural Machine Translation. The main idea
is to translate entire annotated sentences instead of lexicalunits. Then, using neural attention models one can alignand project the semantic annotations from the source language (English) into the target language. This allows building a synthetic FrameNet corpus with exemplar sentences.
A similar approach have already been studied using Hidden
Markov Models (Annesi and Basili, 2010) yielding good41

results in the English-Italian pair. The advances in NMT
allow revisiting this technique using attention models.
Using some post-processing, one can find some of the lexical units that could trigger a frame by looking into the trigger’s alignment. Even though this approach is limited to thelexical units for which we have an English sentence example, it allows us to introduce the full sentence in the translation process. This yields a context-aware translation of thelexical units, instead of a one by one word comparison using a dictionary and human experts annotation. We believeboth approaches are complementary. This NMT approach
can bootstrap a FrameNet lexicon with annotated examples,
which can be improved and completed by human experts.
In this paper, we detail the methodology to perform translation and alignment on the English-to-French setting. We
give metrics to evaluate this translated corpus and we introduce extrinsic evaluation approach that use the synthetic
data-set to train and test automatic FrameNet parsers.

2.

Translating and Aligning

Our objective is to automatically produce French translations for both FrameNet and SemEval-07 annotations andprovide a methodology that can be extended to other languages for which suitable translation models are available.
A sentence and its translation do not necessarily evoke
the same Frames. (Torrent et al., 2018) studies this phenomenon by looking at sentences from the Multi-lingual
FrameNet corpus and comparing the frames evoked in the
English sentences and the Portuguese translations. Theyshow that, in many cases, the frames evoked on each language differ due to different lexicalization and constructional strategies. Normally, this would imply that FrameNet
projections are extremely complex if not unfeasible. We argue that this frame mismatch is widely observed in humantranslation, but much rare in machine translations (MT).
We show this using an example from (Torrent et al., 2018),
comparing an English sentence with the Portuguese humantranslation (HT) and machine translation (MT):• PT-MT: Temos um grande interesse, em parte porque é a educação que nos leva a esse futuro que não podemos compreender.

In Table 1, we list the frames and the lexical units evokedby each sentence. We observe that MT usually does not
modify the constructional strategy and is closer to wordby-word translation than the HT sentences. Even though
MT sentences may be less sounding, the frames observedin the source and target language tend to be similar. This
ensures that the cross-lingual projections can be done, butshow that there may be a domain mismatch between natural
language and machine translated sentences.
Frame
Size
Stimulus focus
DegreeDegreeCausation
Education
Purpose
Performers roles
Bringing
Goal
Temporal colloc
Certainty
Capability
Grasp

EN
huge.ainterest.n—
partly.adv
because.c
education.n
mean.v—take.vinto.prep
future.n—can.v
grasp.v

PT-HT—
interessar.v
tanto.adv
em parte.adv
porque.ceducação.n—papel.nconduzir.v—
futuro.nmisterioso.a
—
—

PT-MT
grande.ainteresse.n—
em parte.adv
porque.ceducação.n——levar.v—
futuro.n—poder.v
compreender.v

Table 1: Frames and LUs from an English sentence and its
Portuguese human (HT) and machine (MT) translations

2.1.

Machine Translation Model

To translate the English FrameNet corpus into French weused a state-of-the-art Neural Machine Translation (NMT)
algorithm from (Ott et al., 2018) using publicly available
pre-trained models from Fairseq1 . This NMT model uses
sequence-to-sequence transformers with a total of 222M
parameters, it uses a Moses tokenizer, a Word-Piece representation optimized for NMT (Sennrich et al., 2016) anda beam search decoding strategy with a depth of 5. The
model achieves state-of-the-art performance on the newstest2014 test set from WMT’14 obtaining 43.2 BLEU scoreon the English-to-French pair.
The most important property of this model is the encoderdecoder attention mechanism (Luong et al., 2015), which isessential for our label alignment strategy. Encoder-decoder
attention allows the target-language decoder to look into
relevant word-piece information from the source-language
encoder. More precisely, for each output word-piece, there
is a soft-max distribution vector across the input wordpieces. This distribution shows in which parts of the input the model focus to yield the given output word-piece.
1

The model is transformer.wmt14.en-fr to be found athttps://pytorch.org/hub/pytorch_fairseq_
translation/

42

PleaseNONEtranslate
B:FRAME:Translating
this
B:FE:Content
EnglishI:FE:Content
B:FE:Source_representation
sentence
I:FE:Content
into
B:FE:Target_representation
FrenchI:FE:Target_representation

B:FRA tradu
ME:T ire
ranslating
B:FE:cetteContent
I:FEp:Chrase
ontent
B:FE: I:FEanglaise
Sourc :Cont
e_rep ent
resen
B:FE:
Targe en tation
t_rep
resen
tation
I:FE:T fra
arget nçais
_repr
esen
tation• PT-HT: Nos interessamos tanto por ela em parte porque é daeducação o papel de nos conduzir a esse futuro misterioso.

This attention matrix can be used as an indicator of a softalignment between the word pieces from the input to the
output. A simplified example of such an attention matrix isshown in Figure 1. This example is developed in detail in
the following subsection.

Veuil
NONElez• EN: We have a huge vested interest in it, partly because it’s education that’s meant to take us into this future that we can’t grasp.

Figure 1: Post Processed Attention Matrix from the NMT
When we look into the raw attention matrix from the NMT
we observe that the decoder’s attention is distributed between different parts of the input (Ghader and Monz, 2017)
and not only on the word-alignment equivalent. For thisreason, attention-based alignment is not straightforward.
Moreover, in many cases translation inserts tokens that do
not have an equivalent. e.g. the following sentence and its
translation:”United States helps Australia stop the fire””Les États-Unis aident l’Australie à arrêter les incendies”
We observe that the translation adds the definite article forboth ”United States” and ”Australia” and it also adds a
preposition ”à” to the lexical unit ”stop.v”. Such word insertions, that are due to some language specific structures,
often produce misleading attention vectors. This is the casefor the preposition ”à”, which introduces the goal argument of the verb ”arrêter”(to stop). Since predicting theword ”à” depends on both the verb ”aider” and the roleof ”stop” as the goal, the attention vector for ”à” is distributed among these two verbs, even though ”à” is not a
viable translation for any of them.

2.2.

FrameNet Label Alignment

To generate a translation for the SemEval-07 corpus wetranslate each sentence using the NMT model describedabove (2.1.). For each sentence, we recover the attention
matrix and apply the following post-processing steps:
Re-establish tokenization: Since the NMT model uses
word-piece representations, the first step is to project theattention matrix into a full-word form. To do this, we perform sub-matrix sums on the sets of rows (and columns)

Part-of-Speech (POS) weighting: To avoid alignment
mismatches as those described above (2.1.), we perform
POS and dependency parsing on both the input and theoutput sentences using spaCy2 . SpaCy has close to stateof-the-art performance on these tasks. We use the POS topost-process the attention matrix applying two simple rules.
First, we mask structural POS such as "PUNCT", "DET","INTJ", "SYM", "X", "AUX", "PART". We do this because the attention vectors for these POS are hard to aligndue to their structural nature (see 2.2.). Second, we encourage the alignment between words with the same POS. To
do this, we multiply by 10 the attention matrix entry (i, j)
if the input and output words (wi wj ) have the same POS.
Finally, we normalize the matrix by columns, e.g. we divide each entry (i, j) by the sum of the values in column j.
This allows interpreting each matrix entry as the percentage
of attention the word wj pays to the input wi .
Label Projection: To project annotations from the English FrameNet into the French translation we flattened the
FrameNet annotations over the English word tokens. Then,we paired each output word to the input word with the highest attention score and propagated the input labels into thematching output token, as shown in Figure 1. Since themodel is not equally confident in every translation, we scoreeach label projection with the value of the attention coefficient between the input and the output word.
Confidence Threshold: To decide which labels project
and which reject, we apply a threshold on the confidence
score. The choice of the threshold is not straightforward.
If it is too low, it introduces alignment mismatches, which
can be seen as frame and frame element insertions. On theother hand, a high value will only project a few annotations.
We chose the threshold that maximizes the harmonic mean
between the the number of annotations projected and theamount of duplicated projections. This step is explainedwith more detail in Section 2.3..
frame element Completing: To ensure homogeneity inthe spans of the frame elements, we used the syntactic dependency parsing to complete the spans of the frame elements applying two simple rules. First, if a determinant or
a preposition is attached to a frame element through its syntactic parent, it inherits the label of that parent. Second, forwords masked during the POS weighing process (e.g. forbeing either "PUNCT", "DET", "INTJ", "SYM", "X","AUX" or "PART"), we assign them a frame element label
if words that precede and follow are part of the same frame
element. This allows us to merge potential frame element
segments, that got split during the alignments.
This sequence of steps is language independent and fairly
easy to implement. However, this does not mean the corpora translation process is flawless. In the next sub-section
we study the quality of the generated corpus.

2.3. Generated Corpus Analysis
In this subsection, we explore the translated corpus and establish some comparison with the original corpus.
2

spaCy website: https://spacy.io/

43First, we evaluate the projection using deletion and insertion metrics. To do so, we assign an id to each annotation(frame or frame element) of the SemEval-07 corpus. Then,
we use these ids to evaluate the French translation by counting how many ids were lost (deletions) and how many ids
got duplicated (insertions). This is a rough metric, somewhat similar to standard precision/recall evaluations. However, it does not imply that annotations are aligned to theright words in the target language. We only measure if thealgorithm finds suitable candidates to project the annotations. We use this metric because more precise evaluations
require skilled human annotators/validators. This evaluation is strongly tied to the confidence threshold selectedduring the alignment (see 2.2.). In Figure 2 we observe thedeletion/insertion trade-off of the projection computed using different values of the confidence threshold. This tradeoff achieves a maximal F1 score of 85.5%. At this point,
the insertion metric is 90.1%, meaning that there are no
more than 10% of false insertions, while the deletion metric is 81.4%. meaning that we project about 81% of the annotations and delete 19%. These alignment performances
show that there is still room for improvement, either via better NMT models or through more complex post-processing
strategies. However, in this paper, we settled at this 85.5%
F1 and we evaluate how useful this simple approach can be.

95
Precisionthat correspond to the same input (and output) words.

90
85

75

80
Recall85

Figure 2: Insertion Deletion Trade-off of the label projection
After this evaluation, we look into more detail whichparts of the projection were problematic. In Figure 3, weshow the distribution of frames that introduced the highest amount of errors during the translation. Whenever the
French column is larger(smaller) that the English column itmeans that the frame got inserted(deleted) several times.
We observe that frames such as Existence, Quantity,
Attributed information,
Capability andPartitive suffer several deletions. Many times, thesedeletions are due to alignment constrains (see 2.2.) suchas not projecting labels from auxiliary verbs. e.g. The
frame Existence in the expression ”There was a time...”translates into ”Il fut un temps...”, here the auxiliary verb
”fut” was masked. Also, for the frame Quantity inthe sentence ” Brazil helped several countries...” whichtranslates into ”Le Bresil a aidé plusieurs pays ...” the word”plusieurs” is a determinant which gets masked. Some ofthese errors could be fixed by introducing more language
specific projection rules. As for the others, changes in
the constructional strategies make label projection very
difficult and error-prone.

0.15

Englishmentation on the target language. We did not explore thisoption in our paper, and we have left it for future work.

French

0.10

3.

0.05

Required_event

Partitive

Substance

Using

Cardinal_numbers

Intentionally_act

PurposeIncrement

Capability

Hostile_encounter

Time_vector

Quantity

Attributed_information

Existence

Weapon0.00

Figure 3: Distribution of the frames with the highest number of projection errors. Whenever the French column islarger(smaller) than the English column it means that the
Frame got inserted(deleted) several times.
On the other hand, frames such as Weapon,
Time vector
andHostile encounter,
Substance suffered insertions.
These insertions
were mostly due to lexical units with prepositional or
adverbial POS. As we discussed previously, the attention
vectors for these words are hard to align and in some cases,
they get projected twice.
English0.05

French

0.04
0.03
0.02
0.01

Phenomenon

Event

Agent

DirectionTime

Descriptor

Required_situation

Group

Hypothetical_event

Theme

Message

Individuals

Quantity

LocaleEntityWeapon0.00

Figure 4: Distribution of the frame elements with the highest number of projection errors
When we repeat the same analysis on the frame elementtranslation we observe that many frame elements projection errors are the proliferation of a frame error. This is
expected since we filter frame elements without a trigger.
There are a few exceptions for this rule, such as time, agent
and theme, which are quite generic frame elements. They
can get deleted due to translation errors or due to low confidence scores. For an example of a translation error, consider the sentence ”... due to the urgency, its development
was fast ...” that gets translated into ”... face à l’urgence, ledevelopment a été rapide...”. Even though it is an acceptable translation, in the English sentence, the word ”its” isan agent frame element. When we look into the translation,
there is no match for this word (it was deleted). Maybe, amore accurate translation would add a pronoun ”leur” as in”... face à l’urgence, leur development a été rapide...”. This
sort of error does not seem dangerous, as the final sentence
is still fully annotated.
Beam Search: an interesting aspect of this approach isthat we can use NMT along with beam search decoding to
extract several translation candidates for any sentence. Thisis particularly useful when building FrameNet dictionaries,
as it allows us to generate translations with different lexical
triggers taking into account the sentence’s context. On theother hand, this technique can also be used to do data aug-

44

biGRU+BERT Semantic Parser

We propose a biGRU+BERT model architecture, inspired
from state-of-the-art models in Semantic Role Labeling (He et al., 2017) and FrameNet parsing (Yang and
Mitchell, 2017; Marzinotto et al., 2018b; Marzinotto et al.,2019). Our architecture, uses 2 layers of bidirectional GRU
stacked on top of a pre-trained ’bert-base-multilingualcased’ model from Huggingface3 . A diagram of our modelarchitecture is shown in figure 5. To encode semantic labels into flat structures we use a BIO encoding scheme. To
ensure that output sequences respect these BIO constrains
we implement an A∗ decoding strategy similar to the one
proposed by (He et al., 2017).
To deal with sentences containing multiple lexical units
we have built training samples containing only one trigger.
More precisely a sentence containing N triggers provides
N training samples. The downside of this approach is thatduring prediction time, parsing a sentence with N triggers
requires N model applications. At decoding time every
pair of { sentence, trigger } is processed by the network
to output a probability distribution on the frames and FE
for each word. Then, we apply a coherence filter to theseprobabilities to make sure that the predicted frame elements
are compatible with the predicted frame by filtering the extraneous frame elements. This coherence filter chose theframe with the highest probability on the trigger and uses
it as the predicted frame (represented as the label assignedby the tagger to the trigger). Then, given that frame, thecoherence filter masks all the frame element labels that areincompatible with the selected Frame.
FrameNet Inference
BiGRU Concatenation

B-EATERI-EATER

B-EAT

B-FOOD

I-FOOD

CAT

CAT

CAT

CAT

CAT

CAT

CAT

CAT

CAT

CAT

Backward GRU
Forward GRU
BiGRU Concatenation
Backward GRU
Forward GRU
BERT-Base-Multilingual-Casedlasourismange

lefromage

Figure 5: biGRU Model Diagram

4. FrameNet Parsing Experiments
4.1.

Data

In our experiments we use 4 FrameNet corpora:
SemEval07-EN: A corpus of full-text annotations fromFrameNet project used for the shared task 19 from
SemEval-07 (Baker et al., 2007). This corpus consistsof annotated journals and it contains 720 different Frames.
3

https://huggingface.co/transformers/

Nb.
Diff.
LU

Nb.
Diff.
FE

Nb.
Diff.
Words

720

3,197

754

121

782

140

53

145

148

Lang

Data Source

Nb. Diff.
FramesSemEval-07

EN

Journals

ASFALDA

FR

Journals

CALOR

FR

Encyclopedias

Nb.
Sentences
w/LU

Nb.
Annot.
LU

Avg. Nb.
Annot.
per LU14,1504,020

24,7707.7

33,955

13,154

16,16720.7

72,12722,60331,440215.4

Table 2: Summary and statistics about the existing FrameNet corpora. Columns from left to right:(1) Language. (2) Source of the non-annotated data. (3) Number of different frames annotated at least one time.
(4) Number of different LU annotated at least one time. (5) Number of different FE annotated at least one time.
(6) Number of sentences containing at least one annotated LU. (7) Total number of LU instances annotated.
(8) Average number of annotations per LU.
SemEval07-FR: This corpus is the translation of
SemEval07-EN into French following the methodology
described in Section 2.2.
ASFALDA French FrameNet: is the first French
FrameNet project (Djemaa et al., 2016) which outlines andproduces a FrameNet equivalent for the French language.
This corpus gathers experts frame annotations on sectionsof the journal Le Monde, it contains 121 different framesthat focus on four notional domains: commercial transactions, cognitive positions, causality, and verbal communication. ASFALDA tries as much as possible to align with
the English FrameNet structure; however, it also introduces
new frames whenever the differences in both languages do
not allow conciliation.
CALOR: is a publicly available corpus(Marzinotto etal., 2018a) of French encyclopedic documents humanannotated using FrameNet semantics. This corpus was
designed from the perspective of Information Extraction
tasks. Like ASFALDA, CALOR uses a partial parsing policy, in which annotations are limited to a small subset offrames from FrameNet. The CALOR corpus contains 53
different frames selected as the most representative and frequent within the annotated documents. Despite the small
number of Frames, CALOR is the corpus with the largest
set of annotated lexical units.
Table 2 summarizes relevant statistics on each corpus.

4.2.

Evaluation setting

We run experiments using the standard Train, Validation, and Test of each corpus. For the SemEval07-FR
corpus, these subsets are equivalent to the ones from
SemEval07-EN. To evaluate our models we use 2 of themain sub-tasks of FrameNet parsing:• Frame Identification: Consists in selecting theframes evoked by each of the lexical units in the sentence. One frame per lexical unit. Here, we use thegold annotated lexical units, and we do not try to infer
them from raw text.
• Argument Identification: Consists in finding thespans of words that correspond to semantic roles andassigning them the correct frame element labels fromthe selected Frame.
Since the set of lexical units on each corpus is different, andsince this difference is due to arbitrary choices about what45

should be annotated in the partial annotations schemes
from CALOR and ASFALDA we consider that all the lexical
unit instances are known and given as input to our model.
For the same reasons, we discard lexical unit annotations
that refer to the frame OTHER, which is an artifact to handle frames out of the scope of the partial annotation. In thissetting, each lexical unit triggers a frame from the frame
dictionary of the given corpus.
We score our models using:• Frame Identification: Accuracy on the frame classification Task with gold lexical units. FrameNet official
evaluation scripts use the frame hierarchy to introduce
a matching metric that gives partial credit when predicting related frames (e.g., a more generic Frame).
Since this hierarchy is no available on each corpus, weevaluate using exact frame matching and we do not
exploit any of the frame-frame relations proposed in
FrameNet.
• Argument Identification: Precision, Recall, and F1or the frame element detection. This metric canbe computed either using the gold or the predicted
Frames. In the official evaluation scripts, the token
span of the hypothesis must be the same as in the reference for a frame element to be correct (Hard Spans or
H-Spans). We have loosened this constraint to introduce a new variant of the evaluation metrics. Insteadof demanding exact span match, we use a weighted
correctness score proportional to the overlap ratio between the gold (Sref ) and predicted (Shyp ) spans(W-Spans) computed using equation 1.

Wspan (Sref , Shyp ) =5.

|Sref ∩ Shyp ||Sref ∪ Shyp |(1)Bi-lingual Semantic Parsing Experiments

Recent works (Pires et al., 2019) have shown that BERT
language models pre-trained on multilingual corpora havea fairly good performance in a zero-shot cross-lingual transfer setting. More precisely, if we fine-tune a multi-lingual
BERT using task-specific annotations from a monolingual
corpus and then evaluate the model in a different language,
the system will be able to generalize to the new language,
up to some extent.

Test:
Frame Id.
TRAINING CORPUS
CALOR
ASFALDASemEval-07-EN
SemEval-07-FR
SemEval-07-EN+FR

ASFALDA
Arg Id.
pred frames
F1-W
52.6
19.7
19.1
20.9

ACC
73.3
42.2
73.3
74.8

Test:
Frame Id.
ACC
98.8
77.5
78.1
79.4

CALOR
Arg Id.
pred frames
F1-W
67.8
30.5
31.0
32.4

Table 3: Performance of a biGRU+BERT on the Gold French FrameNet corpora using different training data-sets
In this experiment, we harness this zero-shot generalization
property to evaluate the quality and the relevance of our
corpus translation strategy extrinsically.
Frame Id.

SemEval-07-EN
SemEval-07-FR
CALOR
ASFALDA

ACC
85.3
80.8
77.5
42.2

Argument Id.
w/pred frames
P
R
F1
49.6
40.3
44.5
29.8
15.4
20.3
37.3
25.7
30.5
24.5
16.4
19.7

Table 4: Performance of a biGRU+BERT model trained on
the SemEval-07-EN corpus and tested on other corpora
Frame Id.

SemEval-07-FR
SemEval-07-EN
CALOR
ASFALDA

ACC
80.5
82.8
78.1
73.3

Argument Id.
w/pred frames
P
R
F1
38.5
26.0
31.1
37.3
23.3
28.7
37.7
26.3
31.0
38.6
15.8
19.1

Table 5: Performance of a biGRU+BERT model trained on
the SemEval-07-FR projected corpus
First, we train a model on the SemEval-07-EN corpus and test in on all the available corpora to establish
a zero-shot baseline performance. We train the modelfor 40 epochs and we used two validation sets, one from
SemEval-07-EN and the other form SemEval-07-FR.
This way, we retained the best performing model for eachvalidation set and language. We observed that the validation error stops decreasing earlier on the French corpus
than on the English one. This could be expected due to themonolingual training configuration. However, we did notobserve significant over-fitting or catastrophic forgetting on
the French language when doing supplementary iterations.
After training, we used the best English model to produce inferences on the SemEval-07-EN test and thebest French model to produce inferences on the 3 French
corpora. We evaluate using precision, recall and F-score
on Weighted Spans (see Section 4.2.). The results forthis experiment are shown in Table 4. We observe thattransfer learning and the language similarities between English and French can bootstrap a low-performance baseline
for French. The system is surprisingly good at detecting

46

Frames. However, it has very low performance on the full
FrameNet parsing Task, showing a particularly low recall.
In the following step, we train a model using the projected
SemEval-07-FR corpus. The performance of this model
is given in Table 5. We observe that the French modeltrained on the synthetic data-set achieves slightly better
performance than the English model baseline. It is better at the Frame Identification step, and it is slightly better in terms of Argument Identification as well. Moreover, the French model is capable of generalizing back to
the English language, showing close performances between
SemEval-07-EN and SemEval-07-FR. The fact that it
is easier to generalize toward English can be interpreted inseveral ways. One possibility is that the French FrameNet
parsing task is more difficult than its English equivalent since FrameNet was designed for English or because
French is a more verbose language. Another possibility isthat due to the alignment errors, the SemEval-07-FR test
data-set flaws would be penalizing good predictions.
In Table 3 we present the scores for the hand-annotatedFrench data-set for different configurations varying thetraining corpus. We observe that even though the Frenchmodel trained on the synthetic data-set achieves slightly
better performances than the English baseline, we are still
far from the state-of-the-art performances on each corpus obtained through training a FrameNet parser on handannotated French data-sets. However, part of this performance gap is due to the differences in the number
of Frames. A SemEval-07 model handles 720 different frames and 3197 different lexical units, therefore
it is much more prone to choosing a wrong frame thana CALOR model, which handles only 53 Frames. Another reason for this performance gap may be the domainchanges, CALOR and ASFALDA contain natural language
sentences instead of translated sentences. Moreover, previous experiments on the CALOR corpus from (Marzinotto
et al., 2019) have shown that even within the same data
source, domain changes yield around 10% F1 performancedrop on the Argument Identification Task. Finally, we
trained a model combining both SemEval-07-EN and
SemEval-07-FR and obtained small gains on the French
corpora, showing that the translated data adds some supplementary information to the model.

6.

Conclusion

This paper presented a simple method to project FrameNet
annotations into other languages using attention-based neu-

ral machine translation (NMT) models. We tested our approach on the English-to-French configuration showing that
90% of the labels can be easily projected without introducing much noise. We performed an in-depth analysis of theFrench corpus obtained through translation and we showed
the most common projection errors. Then, we use existing
French FrameNet corpora to assert the quality of the translation. We trained a BERT-based FrameNet parser using the
projected annotations and compared it to a BERT baseline
showing modest gains on French. All these results support
that our approach could help to propagate FrameNet dataset on other languages where sufficiently developed NMT
models exist. Moreover, this label projection approach canbe extended to other sequence tagging tasks with minor
modifications.

7.

Bibliographical References

Annesi, P. and Basili, R. (2010). Cross-lingual alignmentof framenet annotations through hidden markov models.
In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 12–25, Berlin,
Heidelberg. Springer Berlin Heidelberg.
Baker, C., Ellsworth, M., and Erk, K. (2007). Semeval’07
task 19: Frame semantic structure extraction. In Proceedings of the 4th International Workshop on Semantic
Evaluations, SemEval ’07, pages 99–104, Stroudsburg,
PA, USA. Association for Computational Linguistics.
Djemaa, M., Candito, M., Muller, P., and Vieu, L. (2016).
Corpus annotation within the french framenet: methodology and results. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), X:3794–3801.
Fillmore, C. J. e. a. (1976). Frame semantics and the nature of language. In Annals of the New York Academy of
Sciences: Conference on the origin and development oflanguage and speech, pages 20–32.
Fillmore, C. J. e. a. (2006). Frame semantics. Cognitive
linguistics: Basic readings, 34:373–400.
Ghader, H. and Monz, C. (2017). What does attention
in neural machine translation pay attention to? CoRR,
abs/1710.03348.
Gilardi, L. and Baker, C. (2018). Learning to align acrosslanguages: Toward multilingual framenet. In Proceedings of the International FrameNet Workshop, pages 13–
22.
Hayoun, A. and Elhadad, M. (2016). The hebrew framenet
project. In LREC.
He, L., Lee, K., Lewis, M., and Zettlemoyer, L. (2017).
Deep semantic role labeling: What works and what’s
next. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Lindén, K., Haltia, H., Luukkonen, J., Laine, A. O.,
Roivainen, H., and Väisänen, N. (2017). Finnfn 1.0:
The finnish frame semantic database. Nordic Journal of
Linguistics, 40(3):287–311.
Luong, M.-T., Pham, H., and Manning, C. D. (2015).
Effective approaches to attention-based neural machinetranslation. arXiv preprint arXiv:1508.04025.
Marzinotto, G., Auguste, J., Bechet, F., Damnati, G., andNasr, A. (2018a). Semantic Frame Parsing for Infor-

47

mation Extraction : the CALOR corpus. In LREC2018,
Miyazaki, Japan, May.
Marzinotto, G., Béchet, F., Damnati, G., and Nasr, A.
(2018b). Sources of Complexity in Semantic Frame
Parsing for Information Extraction. In International
FrameNet Workshop 2018, Miyazaki, Japan, May.
Marzinotto, G., Damnati, G., Béchet, F., and Favre, B.
(2019). Robust semantic parsing with adversarial learning for domain generalization. In Proc. of NAACL.
Ott, M., Edunov, S., Grangier, D., and Auli, M.
(2018). Scaling neural machine translation. CoRR,
abs/1806.00187.
Pedersen, B. S., Nimb, S., Søgaard, A., Hartmann, M., andOlsen, S. (2018). A danish framenet lexicon and an annotated corpus used for training and evaluating a semantic frame classifier. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Pires, T., Schlinger, E., and Garrette, D. (2019). How multilingual is multilingual bert? CoRR, abs/1906.01502.
Sennrich, R., Haddow, B., and Birch, A. (2016). Neuralmachine translation of rare words with subword units. In
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Linguistics.
Torrent, T. T., Ellsworth, M., Baker, C., and Matos,
E. (2018). The multilingual framenet shared annotation task: a preliminary report. In Proceedings ofthe Eleventh International Conference on Language Resources and Evaluation, pages 62–68.
Yang, B. and Mitchell, T. (2017). A joint sequential and relational model for frame-semantic parsing. In Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing, pages 1247–1256. Association for Computational Linguistics.

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 48–55
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Greek within the Global FrameNet Initiative: Challenges and Conclusions so far
Voula Giouli1,2, Vera Pilitsidou1, Hephaestion Christopoulos1
1

National and Kapodistrian University of Athens, 2Institute for Language & Speech Processing, ATHENA RC
Address 1, Dragatsaniou 4, 105 59 Athens, Greece
Address 2, Artemidos 6 & Epidavrou, Maroussi, Greecevoula@athenarc.gr, verapilitsidou@gmail.com, hchristo@turkmas.uoa.gr
Abstract

Large coverage lexical resources that bear deep linguistic information have always been considered useful for many natural language
processing (NLP) applications including Machine Translation (MT). In this respect, Frame-based resources have been developed formany languages following Frame Semantics and the Berkeley FrameNet project. However, to a great extent, all those efforts have been
kept fragmented. Consequentially, the Global FrameNet initiative has been conceived of as a joint effort to bring together FrameNets
in different languages. The proposed paper is aimed at describing ongoing work towards developing the Greek (EL) counterpart of the
Global FrameNet and our efforts to contribute to the Shared Annotation Task. In the paper, we will elaborate on the annotation
methodology employed, the current status and progress made so far, as well as the problems raised during annotation.

Keywords: frame semantics, FrameNet, corpus annotation, lexical resource, Greek

1.

aforementioned theory. Over the years, a number offrame-based language resources have been developed forvarious languages (FrameNet Brazil (Salomão, 2009),
Spanish FrameNet (Subirats, 2009) and Japanese
FrameNet (Ohara, 2009), and the Swedish FrameNet++(Ahlberg et al., 2014), inter alia). In this context, theGlobal FrameNet project (Torrent et al., 2018) has
evolved, in order to examine, for example, to what extent
the semantic frames developed for English are appropriate
for other languages, whether some frames are universal
and whether there are certain semantic domains in whichframes tend to vary more across languages, or whether
there are regular patterns of differences based on language
families, regional groupings, etc.

Introduction

Over the last decades, a number of frame-based lexical
resources have been developed based on the Berkeley
FrameNet project (Baker et al., 1998) for languages otherthan English. In this context, the challenge has always
been the alignment of frames across languages. In thispaper, we describe on-going work carried out in theframework of preparing the Greek component of the
Global FrameNet (FrameNet-EL). The purpose of ourwork is two-fold: (a) participation in the Multi-lingual
FrameNet shared task, and (b) the development oflanguage resources, i.e., a general-purpose lexical
resource and an annotated corpus that will be applicablefor a number of applications. The paper is organized asfollows: In section (2), we provide the background andobjectives of our work; our corpus data is described in
section (3). The methodology adopted towards developing
the language resources is presented in section (4),whereas, some preliminary results as well as issues andproblematic cases that we faced throughout the various
stages of our work so far are presented in section (5).
Finally, our conclusions and prospects for future research
are outlined in section (6).

2.

As far as Modern Greek (MG) is concerned, there has
been previous work in language for specific purposes andin language for general purposes – yet these studies
remain fragmented and limited in scope. In fact, an initialattempt to build a frame semantics lexical resource for
MG is reported in Gotsoulia et al. (2007). However, thisinitial work was conceived of as the preliminary phase ofa pilot project for the development of the basic
infrastructure and design of the actual resource. Later,Dalpanagioti (2012) followed a frame-driven approach tothe bilingual lexicographic process for creating a bilingual
lexical database of motion verbs for EL and EN. Anotherattempt was made by Pilitsidou (2018), who used the
FrameNet and Frame Semantics approach to create adomain-specific bilingual terminological database in EL
and EN for the financial domain based on corpus
evidence; the outcome of this work is a bilingual lexical
resource in electronic format consisting of financial terms(LUs) of EL and EN, which are described and defined
through the semantic frames that they evoke and thesemantic relations, as well as a fully annotated corpus invarious levels.

Background and Objectives

According to Charles J. Fillmore’s Frame Semantics(Fillmore, 1977, 1982, 1985), there is continuity between
language and experience (Petruck, 1996). In this context,
words gain their meaning in a semantic frame which canbe an event or a relation. The term “semantic frame” or“frame” refers to any system of meanings which areconnected in a way that, to understand any one of these
meanings, we must be able to understand the whole
structure to which it belongs (Fillmore, 1982: 111).
Fillmore calls the elements of such a structure “Frame
Elements” (FEs) and the words that evoke the semantic
frames “Lexical Units” of the frame (LUs).

This paper reports on our ongoing participation in the
Shared Annotation Task and the contribution to theoverall objectives collaborating with teams from otherlanguages towards developing a database of alignments offrames and FEs across languages. Therefore, the paper isFrameNet, the lexical database for the English language
for general purposes (Baker et al., 1998), was developed
at the University of Berkeley in California based on the48

aimed at describing the progress made so far as well as thevarious issues and challenges we faced while working on
the EL component of the Global FrameNet project. Effort
has also been made to detect and categorize thedifferences spotted between the MG and English
language. In the long run, our objective is to create aframe-based lexical resource for the MG language and tointegrate it into existing semantic lexica.

on the basis of their semantics. As expected, this agnostic
procedure yielded different LUs for polysemous lemmas.
In these cases, word sense discrimination was aided byexisting reference works (monolingual and bilingual
dictionaries) and corpus evidence in order to decide about
the number (and types) of senses. At the next stage,
selection of the appropriate frame each LU evokes waschallenging. Following the global guidelines provided by
the shared task organizers (Torrent el al, 2018), we
adopted the frames as defined in the 1.7 release of theBerkeley FrameNet data (BFN 1.7). Since we were notallowed to make any changes, we tried to identify theframe each LU evokes through extensive search in the
BFN 1.7 in order to locate the most appropriate one taking
also into account its best translational equivalent(s) in
English.

From another perspective, one of our objectives is to
examine whether the alignment of a Greek FrameNet withsimilar resources is feasible and whether the aligned
lexica can be utilized for the translation process. Throughthe efforts for the creation of FN in other languages, thefact that frames are to an extent universal can be proven.
As the lexical resource FrameNet can be machinereadable, it has the potential to be a very useful approach
for assisting translation. In fact, FN’s semanticorganization makes it cross-lingual, as different societies
are highly likely to recognize the same types of events(Tantos et. al., 2015: 168). A database like FN offers avery useful tool for distinguishing subtle, yet crucial,differentiations in meaning, in a way that differs fromother lexical resources, thus rendering it a very promising
tool for the translation process.

3.

In case no translational equivalent of a Greek LU in
English has been created yet in the BFN 1.7, the selectionof the appropriate frame was performed by annotators
using the following decision tree:•

Corpus Description

According to the global guidelines, annotation at thisstage was performed on the transcribed TED Talk “Doschools kill creativity?” (Robinson 2006) and the subtitles
provided for a number of languages; we report here on the
annotation of the Greek counterpart of the TED talk. The
EL corpus comprises 251 sentences and 3012 tokens. We
pre-processed the raw text at various levels of linguistic
analysis (Part-of-Speech tagging and lemmatization,
syntactic annotation) using UDPipe annotation platform(Straka & Starková, 2017).

4.

•

Annotation Methodology

The task of annotation was viewed as a two-stage
procedure: (a) creation of the LUs (or lexical annotation),and (b) annotation of the corpus using the LUs alreadycreated and extending or modifying them where needed.
Both tasks, that is, LU creation and corpus annotation,
were performed by two annotators via the dedicated
MLFN WebTool (Matos & Torrent, 2016). Blindannotation of FEs, GFs and PTs was performed separately
by each annotator. At planned intervals, comparisons of
the annotated data revealed discrepancies which wereextensively discussed and resolved so as to reach a sharedunderstanding of the task at hand and produce an initial
version of the annotated text that has been thoroughly
checked for mistakes or inconsistencies. At this stage,difficult or ambiguous cases were identified andaccounted for. A step-by-step description of the procedure
followed is provided below.

4.1.

•

•

LU Creation

The LUs to be annotated in the corpus were initiallycreated in order to make the annotation process easier; inthis respect, the approach we adopted was purely
lexicographic in the sense that we first extracted all thelemmas from the EL text and then assigned them a frame•

49

Firstly, option (A) was to search in the BFN 1.7
data for a morphologically related LU thatbelonged to a different grammatical category(Part-of-Speech) – yet, it retained the meaning of
the word to be annotated. In case an LU was
spotted, we adopted the frame assigned to it. For
example, the LU αλήθεια.n (truth) was createdunder the frame Correctness, based on theadjective αληθής.a and its translational
equivalent true.a.
If search (A) failed, we proceeded to option (B),that is, we checked if we could locate asynonymous word. In case a synonym was foundin the BFN 1.7 data, we adopted its frame, as in
the case of the adjective ταλαντούχος.a(talented); since its translational equivalenttalented.a was not listed in the BNF 1.7, its
synonymous word skilled.a led us adopt theframe Expertise.
In both cases, (A) and (B), we also checked thatthe frame selected was a perfect fit, that is, it was
actually evoked by the meaning of the LU, in thatthe latter (a) matches the underlying meaning of
the frame and (b) features at least its core FEs.
If both (A) and (B) failed, then we tried option(C), which entailed searching the list of existing
frames and trying to identify a frame that would
be the nearest match. In this case, we were
expected to report the reason the frame was notconsidered as a perfect fit by selecting the mostappropriate one from a list provided in the
annotation tool: (a) different perspective, (b)different causative alternation, (c) different
inchoative alternation, (d) different stative
alternation, (e) too specific, (f) too generic, (g)different entailment, (h) different coreness status,(i) missing FE and (j) other.
Finally, in case we were unable to locate an
appropriate frame, we left the LU underconsideration aside making a note for futurereference. This is the case of the LU βασίζομαι.v

(to be based on), for which the frames of themorphologically related lemmas could not be
adopted and no synonymous word could befound.

Types (PTs) chosen from UD tags were used to tag therealizations of FEs in the corpus. In the next section wewill elaborate further on the results obtained, focusing on
the creation of LUs and their annotation in the corpus.

It should be pointed out, however, that during theannotation process we found out that certain LUs could be
assigned to more than one frames, as for example the LUδημιουργικότητα.n (creativity); using option (A) we
located the LU creation.n that evokes the frame
Intentionally_create. At the same time, an
extensive search in the frames showed that the frame
Mental_property is also applicable.

In Figure 1, two examples of annotated sentences from thecorpus are presented.
Είχαμε ΓΕΜΙΣΕΙ [GOAL το μέρος] [THEME με ατζέντηδες
που φορούσαν μπλουζάκια] (Filling)(Implied
AGENT: We)Είχαμε γεμίσει το μέρος με ατζέντηδες [WEARER που]ΦΟΡΟΥΣΑΝ [CLOTHING μπλουζάκια] (Wearing)

Finally, following common lexicographic practices, foreach LU, we provided sense description, in the form of ashort lexicographic gloss in English.

4.2.

Ihame gemisi
topou forousan blouzakia

Corpus Annotation

Had1.pl filled
the.acc
wore3.pl T-shirts.acc

After the LUs had been created, annotation proper was
performed. At this stage, each sentence in the corpus wasannotated at the following layers: (a) Frame and Frame
Element (FE) layer, (b) Grammatical Function (GF) layer,and (c) Phrase Type (PT) annotation1.

me agentides

place.acc with agents.acc who“We had filled the place with agents wearing T-shirts”[ADDRESSEE Σας] ΟΔΗΓΗΣΑΝ, πιθανότατα, [CONTENTμακριά από κάποια πράγματα στο σχολείο] όταν ήσαστανπαιδιά, πράγματα που σας άρεσαν, [MEANS με τηδικαιολογία ότι δεν θα βρίσκατε ποτέ δουλειά κάνοντας
αυτά], σωστά; (Talking_into) (Implied SPEAKER:They)

The major challenge was the identification of the correct
LU already created (see section 4.1). Again, we also had
to tackle polysemous lemmas by selecting the mostappropriate LU to annotate. As a matter of fact, sense
discrimination was often a challenge and fine distinctions
between closely related frames made it difficult to spot thedifference between them. In these cases, context wasalways helpful, especially in cases where the distinctions
might be extremely fine. For example, the polysemous
lemma πηγαίνω.v (to go) evokes more than one frames.
However, as shown in example (1), given the context, theParticipation frame has been selected based on thecontext – instead of the Motion one:Σας οδήγησαν, πιθανότατα, μακριά από κάποια πράγματαστο σχολείο όταν ήσασταν ΠΑΙΔΙΑ, πράγματα που σαςάρεσαν, με τη δικαιολογία ότι δεν θα βρίσκατε ποτέδουλειά κάνοντας αυτά, σωστά; (People_by_age)Σας οδήγησαν, πιθανότατα, μακριά από κάποια πράγματαστο σχολείο όταν ήσασταν παιδιά, [CONTENT πράγματαπου] [EXPERIENCER σας] ΑΡΕΣΑΝ, με τη δικαιολογία ότιδεν θα βρίσκατε ποτέ δουλειά κάνοντας αυτά, σωστά;(Experiencer_focused_emotion)(1) “Δεν πάτε συχνά σε πάρτι”
Den pate sichna se parti
Not go2.pr often to parties“You don’t go to parties often”

Σας οδήγησαν, πιθανότατα, μακριά από κάποια πράγματαστο σχολείο όταν ήσασταν παιδιά, πράγματα που σαςάρεσαν, με τη ΔΙΚΑΙΟΛΟΓΙΑ [EXPLANATION ότι δεν θαβρίσκατε ποτέ δουλειά κάνοντας αυτά], σωστά;(Justification)

LUs were then populated with information regarding theFEs found in the corpus and their realizations. At the nextlevel, each FE was assigned the grammatical function it
assumes in the sentence. The following grammatical
relations are foreseen: Noun Subject (Nsubj), Object(Obj), Indirect Object (Iobj), Clause Subject (Csubj),
Clause Complement (Ccomp), Xcomp, Head, Dep, Nmod,
Appositive, Ext. These relations were adopted from
Universal Dependencies (UD)2. Subsequently, PhraseΣας οδήγησαν, πιθανότατα, μακριά από κάποια πράγματαστο σχολείο όταν ήσασταν παιδιά, πράγματα που σαςάρεσαν, με τη δικαιολογία ότι δεν θα ΒΡΙΣΚΑΤΕ [TIMEποτέ] [THEME δουλειά] κάνοντας αυτά, σωστά;(Getting) (Implied RECIPIENT: You)Σας οδήγησαν, πιθανότατα, μακριά από κάποια πράγματαστο σχολείο όταν ήσασταν παιδιά, πράγματα που σαςάρεσαν, με τη δικαιολογία ότι δεν θα βρίσκατε ποτέδουλειά
ΚΑΝΟΝΤΑΣ[ACTαυτά],
σωστά;(Intentionally_act)1

Two more layers are also foreseen by the Shared Task
organizers, namely, Other and Sentence. The layer Other
involves annotation of relative pronouns and their antecedents,
whereas, the Sentence layer features tags applicable to the wholesentence, and may include notes such as the existence of a
metaphor, or how prototypical the sentence is. For the time
being, we did not perform any annotations at these layers.
2

meros

Sas
odigisan,pragmata sto
pragmatapou

https://universaldependencies.org/u/dep/index.html

50

pithanotata,makria apo
kapia
scholiootan isastan
pedia,
sas
aresan,metidikaiologia oti dendouliakanontas afta,

tha
sosta?

vriskate

No of existing LUs
No of LUs created
Perfect fits
Non-perfect fits
No frame assigned

pote

You.plwere led
most likely
away
from
some.acc things.acc
at the.acc school.acc
when
were2.plchildren, things.nom that you.gen were
liked, withthe.accexcuse.acc that not wouldfind2.pl never job.acc doing
them,correct?

Table 1: Quantitative results of frame assignment
A recurring issue that led us to sometimes taking unusual
initiatives or resulted in the assignment of non-optimalframes are the cases of systematic polysemy, which is aphenomenon encountered across languages and should beconsidered by the FN team. Quite often, the FN catalogue
seems to take into account only a certain shade of theoccasional word’s meaning, and does not assign it to otherframes that cover its different uses. This also led us to
speculate that there might be some important frames
missing from the FN catalogue. FN in general sometimes
does not seem to distinguish between subtle
differentiations in meaning, and there are words that incertain contexts could easily be assigned to a frame, butthere are occasions where a perfect fit is impossible tofind. For example, LUs such as πανεπιστήμιο.n(university) and σχολείο.n (school) fall in this category.
According to the FN index, these LUs evoke the frame
Locale_by_use, but this is merely one of their
meanings. This is a classic case of systematic polysemy,
as the words do not only denote the building itself, butalso the institution and the activities that take place there.

“They most likely led you away from certain things at
school when you were children, things you liked, with theexcuse that you would never find a job doing those things,
correct?
Figure 1: Examples of annotations from within the corpus

5.

Preliminary Results

This being a work in progress, presented below are ourinitial findings. After completing a certain part of theannotation, the team got together and discussed theirfindings. As will be shown below, there were several
issues, some of which pertain to the nature and structureof the MG language as well as the translation of the text,
which, more often than not, was not optimal, while others
regard proposed additions or revisions of the FN framesand FE structure. We did encounter several cases of
mistranslation or bad wording, some of which made it
impossible to assign frames and FEs, while in other cases
we had to make some uncommon decisions. Issues suchas annotation of multiword expressions and grammaticaldifferences between EN and EL will be discussed morethoroughly below.

5.1.

626
603
549
54
23

There were also other cases where the FN frame was moreor less a good fit, but we did notice some missing FEs thatwould be useful in MG and perhaps other languages(sometimes in English, as well) or cases where thedescription of the frame contained FEs we consider
redundant or too specific. For example, the LU παίζω.v(to play) evokes the frame Competition; however, agame is not always competitive, and this is not captured in
the frame – or any other frame in BFN 1.7.

LU Creation: Results and Issues

In total, c. 603 LUs were created that evoke c. 250 frames;
regarding the verbs of the EL corpus, which are the main
focus, more than about 220 frames have been assigned to
the 167 unique verbs. In most cases, frame assignment via
the EN LUs was a laborious – yet straightforward – task
and the BFN 1.7 frames were proved a perfect fit,
whereas, in a number of cases, no frame seemed to be aperfect fit. It should be noted that the already existing
BFN 1.7 frames worked very well in almost all cases ofcommonly used phrases and words with a distinct andspecific meaning, even in cases of polysemy where theword meanings were quite discrete.

Another example is χορεύω.v (to dance). FN lists dance
under Self_motion, but the definition of the frame is:“The SELF_MOVER, a living being, moves under its own
direction along a PATH. Alternatively, or in addition to
PATH, an AREA, DIRECTION, SOURCE, or GOAL for the
movement may be mentioned.” There is clearly nonecessity for a PATH or DIRECTION when someone isdancing. Another possible choice, not mentioned in the
English FN, would be Moving_in_place, which issometimes true for this specific activity and sometimes
not. Perhaps a more suitable frame would be one referring
to pastime activities.

Table 1 provides quantitative data on the frame
assignment of the LUs we have taken into account so far.
As one can see, the percentage of perfect fits is quite high(87.8%), as opposed to the 8.6% of non-perfect fits andthe 3.7% percentage of the cases where no available frame
could be found. However, it should be noted that, in orderto achieve that satisfactory percentage, we often had todiverge from the frames the BFN assigns to certain LUs
or make our own choices in cases of LUs that are notindexed. The main causes for cases of non-perfect fitswere different perspective and different entailment,
followed by too specific or too general frame, missing FE
and different causative alternation.

A similar case is “Σε λίγο τα πτυχία δεν θα αξίζουν
τίποτα.” A not very elegant, but closer to the EL text backtranslation would be “In a while, the degrees will be worth
nothing.” The LU αξίζω.v (to be worth) could be assigned
to the Deserving frame. The frame’s definition
according to FN is “The existence of a
STATE_OF_AFFAIRS is sufficient reason for taking an
ACTION. The agent who is justified in taking the51

suggested ACTION is not part of the immediate scene,
however.” Based on the phrase above, we believe that the
ACTION is an optional element in the frame, since there is
no reference to what ACTION would be taken if the
STATE_OF_AFFAIRS (the degrees) were sufficient.

different frame as opposed to its middle voice counterpart
εμφανίζομαι.v (to appear or arrive). The middle voice ofthe verb can be assigned to frames such as Arriving or
Becoming_visible, while the active voice
εμφανίζω.v most certainly does not belong there.
Similarly, the LU ωφελούμαι.v (to benefit from), which ismiddle voice in MG, and its active voice ωφελώ.v (whichmeans benefit as in “These new courses will benefit thestudents”) need to be indexed under different frames. The
frame Cause_benefit_or_detriment certainly is
not the best fit for the middle voice, while other frames
only tangentially relate to the verb’s meaning. This could
imply that there’s a missing frame in the FN catalogue, as
we could not find one suitable to the middle voice form.

(2) “Σε λίγο τα πτυχία δεν θα αξίζουν τίποτα”
Se ligotaptihiaden
tha axizoun tipota
In a little the.pl.nom
degrees.nom
notwill be worth nothing“In a while the degrees will be worthless”
Another issue that caught our attention is that sometimes
FN does not seem to distinguish between the procedurethat leads to a certain result and the case where an LU
denotes being in that certain state from the beginning. At
least in EL, there are some LUs, mainly verbs, that could
assume both meanings, depending on context. Thisdistinction might appear too fine at first sight, it ishowever frame-defining, and we did notice the lack ofavailable frames in such cases. This is the case of the LUsενώνω.v (to connect, to join) and χωρίζομαι.v (to be
separated, to be divided). Here are two similar cases: the
LU ενώνω.v is not in this example a perfect fit for the
Attaching frame, as shown in (3), since the framerefers to the process of joining, not the state of being
joined.

Similarly, non-perfect fits showcase differences inperspective between MG concepts and English ones, as
shown with the verb συνταξιοδοτώ.v. In MG, the verb has
both active and passive morphology, whereas therespective verb in English “to retire” corresponds to thepassive voice; moreover, in MG, the verb has more
specific connotations, as it means to leave one’s job andget a pension, the pension being the core component of
the verb’s meaning.
The verbs επιτρέπεται (to be allowed) and πρόκειται (arough translation would be “will” or “be about to”) areanalogous to αρέσω.v (discussed in 5.2). Regarding thefirst case, we encounter the phrase “Τα παιδιά χορεύουνόλη την ώρα, αν αυτό τους επιτρέπεται,” which can beback translated as “Children dance all the time, if they areallowed to.” This is the middle voice of επιτρέπω.v (toallow), and apart from the problems with frame
assignment, there are some peculiarities in its use. A more
literal translation of the phrase “αν αυτό τους επιτρέπεται”
would be “if this is allowed to them” (consequently, it is
not a perfect fit for Preventing_or_letting or
Deny_or_grant_permission). A peculiarity of thisverb, however, is that the middle voice of the verb appears
only in the third person singular or plural, meaning that acertain act is allowed to some entity. Moreover, it should
be noted that we couldn’t find a suitable FE in thePreventing_or_letting frame denoting who isallowed or prevented from doing something. Furthermore,
in MG the entity that is allowed to do something isrealized as the object of the verb in the genitive case or as
a complement of the preposition σε (e.g. σε εμένα,meaning “to me”), not as the subject, as is the case with
the English verb to be allowed. The verb’s antonym
απαγορεύεται.v has the exact same properties as the ones
just discussed.

(3) “Ενώνει τα δύο μισά του εγκεφάλου”Εnonita
dyo misa
tou
egefalouConnects the.sg.acc two halves.acc the.sg.gen
brain.gen“Connects the two halves of the brain”
On the other hand, the LU χωρίζομαι.v is not a perfect fit
for Becoming_separated as shown in example (4),as the phrase refers to the state of being separated.
(4) “Δεν χωρίζεται σε διαμερίσματα”
Den horizete
se diamerismata
Not divided3.sg.pass into compartments.acc“It is not divided into compartments”
However, not all difficulties we encountered should be
attributed to shortages in the FN index. Some issues arise
from peculiarities and idiosyncrasies of the MG language,
such as the middle voice, which are to be expected, since
the BFN we take as a starting point was originallydesigned for the English language. Generally, passive (ornon-active) morphology of some EL verbs is found inreflexive, anti-causative and passive structures. However,in some cases, the passive morphology of some verbs
signifies usages/senses besides those entailed by the active
morphology (Clairis & Babiniotis, 2005). These
differences in meaning cannot be accounted for at earlystages of processing; as a result, when attempting toassign an EL verb to a frame, the annotation tool treats theactive and the middle voice of verbs as a single lemma.
As expected, this presents a problem in several cases, as
the active and middle forms of a verb might belong to adifferent frame. For example, the LU εμφανίζω.v (toreveal, to present) in active voice needs to be assigned to(5) “Aν αυτό τους επιτρέπεται”An aftotous
epitrepete
If this.sg.nom they.gen is allowed“If they are allowed to.”Regarding πρόκειται.v, here we have an even more
noteworthy case. It is a middle voice verb that is usedsolely in the third person singular, and more often than notpreceded by the negative particle δεν (not). An instancefrom our corpus is “δεν πρόκειται να γίνεις μουσικός”52Last but not least, the LU εισαγωγή.n (in this case,
university admission, but entrance or insertion in general– “είναι μια παρατεταμένη διαδικασία εισαγωγής στοπανεπιστήμιο”/“it is a prolonged university admission
procedure”) was also problematic. The word in thiscontext cannot be assigned to Arriving, since the framerefers to a literal arrival at a place, but neither to
Success_or_failure, as it is too specific. This is acase of a subtle differentiation in meaning which makes it
difficult to find a suitable frame(“you will never become a musician”). We assigned theverb to the Destiny frame, even though we are not verysatisfied with that choice, since the sense of fate is notalways implied by this verb. What is interesting is that thecore FEs (PROTAGONIST, ROLE, STATE_OF_AFFAIRS) are
not realized as complements to the verb but as a
subjunctive subordinate clause:(6) “Δεν πρόκειται να γίνεις μουσικός”
Den prokite na ginis
mousikos
Not will3.sg to become2.sg.sbjv
musician“You will never become a musician”

5.2.

Corpus Annotation: Results and Issues

In total, 222 out of the 251 sentences of the text wereannotated, whereas, the annotation effort amounts to 620
annotation sets for verbs, nouns, adjectives, adverbs andnumbers. The distribution of the annotated LUs per Partof-Speech (POS) is depicted in Table 2.

Furthermore, a number of ambiguous cases wereidentified. For example, the LU κάνω.v (to do or to make)was assigned to the frame Intentionally_act;
however, this is not always the case. In some cases, there
might be a different entailment, since doing something
does not always imply intention to do it. In this respect,
the frame is too specific.

POS
adjective
adverb
noun
num
verb
total

Despite the variety of frames and LUs the BFN 1.7 offers,
we encountered several instances where the FN-assigned
frames do not cover all cases of the EL lemmas. One
example is the EL noun ιδέα.n (idea: “να έχεις
πρωτότυπες ιδέες” – “having original ideas”). The FNassigned frame is Awareness. However, the particularmeaning of the word is not covered by this frame. A
possible candidate could be Coming_up_with, eventhough this frame is perhaps a bit too specific. This is acase were not all meanings of a word are covered.
Similarly, the LU ανταποδίδω.v (to reciprocate or to
return) in (7) was listed under the Request frame,
although in this specific instance it does not mean inviting
someone but returning a favour instead. Reciprocate is notindexed in FN and return is indexed only under different
meanings.

No
32
3
157
7
421
620

Table 2: Distributions of LUs per POS
It should be noted that MG is a pro-drop language, andconsequently several core elements, such as the AGENT or
the COGNIZER, appear to be missing from the sentences,while in reality they can be inferred from the verb form.
This is a special case of constructional null instantiationthat one can come across very frequently in languages thatfeature this syntactic characteristic, such as Greek,
Spanish or the Slavic languages. For this reason, it is notpossible to annotate all core FEs in the corpus, unless theannotation platform is modified as appropriate so as totake the peculiarities of pro-drop languages into account,since the labelling Null does not allow the annotator todefine the FE that appears to be missing.

(7) “Δεν ανταποδίδουν την πρόσκληση”
Den antapodidoun tin
prosklisi
Not return3.plthe.sg.acc invitation sg.acc“They don’t invite you back”

Moreover, the annotation process posed challenges due tothe genre of the text; as a result, some sentences were notannotated at all since they present phenomena like ellipsis
or pragmatic function. This is especially true for questions
like the one presented in (9):Another instance was χάνω.v (in our case, to miss, butalso to lose in other contexts): “Έχασα κάτι;” (“Did I misssomething?”). We found it impossible to assign a frame tothis meaning of the word. A possible candidate could be
Perception, but miss does not imply modalities likehear or taste.

(9) “Τι έγινε;”Tiegine?
What became?
“What happened?”

The LU βρίσκω.v (to find, to get) of the phrase “Δεν θαβρίσκατε ποτέ δουλειά” (“You would never find a job”)
posed a difficulty as well. The Getting frame implies
the acquisition of an object or some property and thechange of ownership. This is not the case here. On theother hand, the frame Being_employed, which wouldrefer to the whole phrase and not just the verb (see casesof multiword expressions below), is not suitable either,because it does not refer to the process of acquiring a job.

This meaning of the LU γίνομαι.v occurs only in the third
person singular, which is a distinctive quality that should
be noted.
From another perspective, the annotation at the GF andPT levels revealed further discrepancies and non-perfectfits. More precisely, we did notice some differences in therealization of the FEs in EN and EL that are worth
pointing out; these could either pertain to the frame
assignment itself or to differences between the structureand syntax of FEs within a given frame. An example of a(8) “Δεν θα βρίσκατε ποτέ δουλειά”
Den tha vriskate
pote doulia
Not will find3.pl.pret never job.acc“You would never find a job”

53

LU as compared to its translation in EN is depicted inTable 3. Following its EN counterpart, the EL LU
αρέσω.v (to like) was created under the frame
Experiencer_focus. However, this was proven to bea non-perfect fit, and was chosen only because no otherframe seemed appropriate. The main difference between
the verb to like and the EL verb αρέσω.v is that in English
the EXPERIENCER is always realized as the Subject of theverb; in EL, however, the EXPERIENCER is realized either
as the complement of the preposition or as the object
complement in genitive case. Moreover, in a more generaluse/meaning, the verb can be used without a complement
at all. Consequently, the CONTENT rather than the
EXPERIENCER seem to be the focus of the verb; theEXPERIENCER might not even be present in the sentence,as will be shown in the examples below. The following
table depicts the different realizations of the EN and EL
verbs αρέσω.v/to like.v:

such as the noun phrases καθηγητής πανεπιστημίου
(university professor) and εκπαιδευτικό σύστημα(educational system) could be either regarded as two
distinct words or as a homogenous whole, as, for example,a university professor is a distinct vocation compared to,e.g., καθηγητής αγγλικών.n (English teacher), which bothuse the equivalent of the word professor in a totallydifferent context. However, even if it would perhaps be
preferable to assign these phrases to a single frame as awhole, it is quite straightforward to frame them word byword.
But not all cases are that simple. As a matter of fact, anumber of idiomatic expressions found in the corpus canonly be treated as single predicates. For example, theverbal MWE δεν μας παίρνει (we can’t afford to) must be
assigned as a whole to the Capability frame.
However, as it is an idiomatic phrase, a word-by-wordtranslation would be “it doesn’t take us.” Clearly there isno point in assigning the LU take.v to the Taking framein this instance. This is also the case with a number orLight Verb constructions.

Experiencer_focus
Realization
like.vαρέσω.v
EXPERIENCER Ext.NP
Obj.NPCONTENT
Obj.NPNsubj

A fact that should be taken into account should the FN
annotation platform make it possible to assign frames to
MWEs is that MWEs are often discontinuous, as is often
the case in MG. For example, the expression in (12),belonging to the Attempt_suasion frame, consists offixed discontinuous elements and non-fixed ones:

Table 3: Realization of the LUs to like.v and αρέσω.v
Usages of the verb αρέσω are depicted in examples (10)and (11) below:(12) “Κάνε μου τη χάρη.”Kane mouti
hari
Do I.gen the.sg.acc
favour.acc“Do me this favour”, also meaning “indulge me”(10) “Πράγματα που σας άρεσαν”
Pragmatapousas
aresanThings.nom thatyou.pl.gen liked“Things you liked”(11) “Ο Γιάννης αρέσει”
O
Gianisaresi
The.sg.nom Gianis.nom is liked“Giannis is liked” (meaning, by people in general –note the absence of the EXPERIENCER)

The expression is “κάνω τη χάρη” (do the favour), and the
pronoun can be interposed in between, disrupting thecontinuity of the phrase.

6.

Conclusion

We have presented work in progress towards developing
the Greek section of the Global FrameNet Shared Task. In
an attempt to prove the universal nature of frames, effort
has been made to construct a frame-based lexical resourcefor MG and to annotate an EL corpus based on frames that
already exist for the English language. This task has notalways been an easy and straightforward one. In the paper
we have reported on the progress made so far, and on theissues encountered. Future work is already planned
towards enriching the EL data with new corpora andannotations and towards using the resource for aiding thetranslation process. In particular, a future prospect is to
add comparable corpora to the data, in order to extend thelexical resource and avoid any inconsistencies that emerge
from mistranslation or wrong wording of the translated
corpus. As a matter of fact, the need of adding more
frames or more FEs to FN, with which it would bepossible to include the differentiated meanings of LUs of
the MG language, has emerged, so that, in the future, theFinally, we should note again that there were a great many
instances of mistranslation or bad wording in the EL text,
which made the frame assignment very difficult or even
impossible in certain cases. One such case, maybe themost characteristic one, is the phrase, “Αυτοί οι άνθρωποι
που βγαίνουν από την κορυφή” (“These people who comeout of the top”), which makes no sense and it was
impossible to infer what the translator meant by it.
5.2.1.
Multiword Expressions
Multiword expressions (MWEs) have long been regarded
as a “pain in the neck” for NLP and translation alike, due
to their idiosyncratic behaviour (Sag et al., 2002). In fact,
they are lexical items characterized by lexical, syntactic,
semantic, pragmatic or statistical idiosyncrasies. We did
encounter such cases in the corpus which form solid
semantic unities and cannot be treated on a word-by-wordbasis. Some cases are debatable; for example, collocations

54

Petruck, M. R. L. (1996). Handbook of Pragmatics. Jef
Verschueren, Jan-Ola Östman, Jan Blommaert, andChris Bulcaen (eds.). Philadelphia: John Benjamins.
Pilitsidou, V. (2018). Terminology: Application of Frame
Semantics to the Thematic Field of Finance for theCreation of a Term Base, from a Translation
Perspective. (Master’s thesis, National and Kapodistrian
University of Athens, Athens, Greece). Retrieved fromhttps://pergamos.lib.uoa.gr/uoa/dl/object/2775104
Robinson, K. (2006). Do Schools Kill Creativity? TED
Talk.
Availableathttps://www.ted.com/talks/ken_robinson_says_schoo
ls_kill_creativity.
Sag, I., Baldwin, T., Bond, F., Copestake, A., andFlickinger, D. (2002). Multiword Expressions: A Painin the Neck for NLP. In A. F. Gelbukh, ed. 2002.
Proceedings of the Third International Conference on
Computational Linguistics and Intelligent Text
Processing (CICLing '02). Springer-Verlag, London,
UK, UK, 1-15.
Salomão, M. M. M. (2009). Framenet brasil: um trabalho
em progresso. Caleidoscópio 7(3): 171–182.
Straka, M. and Starková, J. (2017). Tokenizing, POS
Tagging, Lemmatizing and Parsing UD 2.0 with
UDPipe. Proceedings of the CoNLL 2017 Shared Task:
Multilingual Parsing from Raw Text to Universal
Dependencies.
Associationfor
Computational
Linguistics:
8899.
Vancouver,
Canada.
http://www.aclweb.org/anthology/K/K17/K17-3009.pdf
Subirats, C. (2009). Spanish FrameNet: A Frame-semantic
Analysis of the Spanish Lexicon: 135–162. Berlin/New
York: Mouton de Gruyter.
Tantos, A., Markantonatou, S., Anastasiadi-Symeonidi,A., and Kyriakopoulou, T. (2015). Computational
Linguistics (in Greek). [E-book] Athens: Hellenic
Academic
Libraries
Link.
https://repository.kallipos.gr/bitstream/11419/2205/13/n
lp2ndEdition.pdf
Torrent, T., Elsworth, M., Baker, C., da Silva Matos, E. E.
(2018). The Multilingual FrameNet Shared Annotation
Task: a Preliminary Report. In The International
FrameNet Workshop 2018: Multilingual FrameNets andConstructions.

database can be used for the MG language. From anotherperspective, further work is planned towards making
meaningful cross-lingual comparisons.

7.

Acknowledgments

The authors would like to thank the anonymous reviewers
and the editors for their valuable comments to themanuscript that contributed to improving the final versionof the paper. The research leading to the results presented
here was partially supported by the Translation andInterpreting MA programme of the Faculty of Turkish andModern Asian Studies of the National and Kapodistrian
University of Athens.

8.

Bibliographical References

Ahlberg, M, Borin, L., Dannélls, D., Forsberg, M,
Toporowska Gronostaj, M., Friberg Heppin, K.,
Johansson, R., Kokkinakis, D., Olsson, L.J., andUppström, J. (2014): Swedish FrameNet++ The
Beginning of the End and the End of the Beginning, inProceedings of the Fifth Swedish Language Technology
Conference, Uppsala, 13-14 November 2014.
Baker C. F., Fillmore C. J., and Lowe J. B. (1998). The
Berkeley FrameNet Project. Proceedings of the 17th
International Conference on Computational Linguistics– Volume 1 (COLING '98), Vol. 1. Association for
Computational Linguistics, Stroudsburg, PA, USA: 8690.
https://dl.acm.org/citation.cfm?doid=980451.980860
Clairis, Chr., and Babiniotis, G. (2005). Grammar of theModern Greek Language (In Greek). Athens: Ellinika
Grammata.
Dalpanagioti, T. (2012). A Frame-driven Approach to
Bilingual Lexicography: Lexical Meaning and Usage
Patterns in Greek and English Motion Verbs. PhD
Thesis. Εθνικό αρχείο διδακτορικών διατριβών.
https://www.didaktorika.gr/eadd/handle/10442/27162
Fillmore, C. J. (1977). Scenes-and-frames Semantics.
Book chapter in: Linguistic Structures Processing.
Fundamental Studies in Computer Science, vol. 59.
North Holland Publishing: 55-81.
Fillmore, C. J. (1982). Frame Semantics. Linguistics in
the Morning Calm, ed. The Linguistic Society of Korea.
Selected papers from SICOL-1981: 111-137.
Fillmore, C. J. (1885). Frames and the Semantics ofUnderstanding. Quaderni di semantica. Vol. 6, no 2:
222-254.
Gotsoulia, V, Desipri, E., Koutsombogera, M.,
Prokopidis, P., Papageorgiou, H., and Markopoulos, G.
2007. Towards a Frame Semantics Lexical Resource for
Greek. In Proceedings of the Sixth International
Workshop on Treebanks and Linguistic Theories.
Matos, E. and Torrent, T. (2016). A Flexible Tool for an
Enriched FrameNet. In Proceedings of the 9thInternational Conference on Construction Grammar(ICCG9), Juiz de Fora, Brazil, october. Federal
University of Juiz de Fora (UFJF).
Ohara, K. H. (2009). Frame-based Contrastive Lexical
Semantics in Japanese FrameNet: The Case of Risk and
Kakeru: 163–182. Berlin/New York: Mouton de
Gruyter.

9.

Language Resource References

FrameNet: https://framenet.icsi.berkeley.edu/fndrupal/
Glosbe: https://el.glosbe.com/
Merriam-Webster: https://www.merriam-webster.com/
Oxford Dictionaries: https://www.oxforddictionaries.com/
Portal for the Greek Language: http://www.greeklanguage.gr/greekLang/index.html
WordReference: http://www.wordreference.com/

55

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 56–62
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Using Verb Frames for Text Difficulty Assessment
John Lee, Meichun Liu, Tianyuan Cai
Department of Linguistics and Translation
City University of Hong Kong
{jsylee, meichliu, tianycai}@cityu.edu.hk
Abstract
This paper presents the first investigation on using semantic frames to assess text difficulty. Based on Mandarin VerbNet, a verbal
semantic database that adopts a frame-based approach, we examine usage patterns of ten verbs in a corpus of graded Chinese texts. We
identify a number of characteristics in texts at advanced grades: more frequent use of non-core frame elements; more frequent omissionof some core frame elements; increased preference for noun phrases rather than clauses as verb arguments; and more frequent metaphoric
usage. These characteristics can potentially be useful for automatic prediction of text readability.
Keywords: Mandarin VerbNet, verb frames, frame elements, readability

1. Introduction

such as those in FrameNet, Chinese Framenet (You and Liu,
2005), or Mandarin VerbNet (Liu, 2016; Liu and Chang,
2016; Liu, 2018; Liu, 2019). Based on Mandarin VerbNet,a verbal semantic database that adopts a frame-based approach, this paper investigates the correlation between verb
frames and text difficulty.

FrameNet (https://framenet.icsi.berkeley.edu) and othersimilar resources have supported a large range of natural
language processing (NLP) tasks including semantic rolelabeling (Gildea and Jurafsky, 2002), information extraction (Fader et al., 2011), sentiment analysis (Ruppenhofer
and Rehbein, 2012) and language learning (Carrión, 2006;
Xu and Li, 2011). However, they have yet to be exploited
for analyzing text difficulty, which is also known as readability assessment. Given any text, the system is to predict
its reading difficulty, by estimating the age or school grade(e.g., Grades 1 to 13) required for readers to understand thetext; by assigning it a difficulty score, such as Lexile (Stenner, 1996); or by locating it on a proficiency scale, such as
the six-level scale in the Common European Framework ofReference for Language (2001).
Previous research on automatic readability assessment has
mostly relied on lexical and syntactic features. A common
lexical feature is the level of vocabulary difficulty, for example according to the number of “difficult words” (Kincaid et al., 1975). Syntactic features may include parse tree
patterns or, as a proxy, average sentence length. While lexical complexity and syntactic complexity have been shown
to be effective predictors of text readability, they do not capture all aspects of reading difficulty. Consider the pairs of
example sentences in Table 1. The sentences in each pair
have comparable vocabulary difficulty and sentence length.
Sentences (1a) and (1b) both have the verb ‘worry’. The
verb in (1a) takes as object a short clause ‘you would get
sick’, but in (1b) it takes an abstract noun, ‘your health’,
which may be more difficult to process. Likewise, sentences (2a) and (2b) are semantically similar, but the reason construction ‘because [he] missed the exam’ in the latter may make it harder to read than the former. Finally,
sentence (3b) is likely more challenging to understand than(3a) due to a metaphorical usage.
Semantic analysis can be expected to improve the readability assessment for such sentences. While some existingassessment models already incorporate semantic features,
they are mostly limited to anaphora patterns, word senses
and semantic categories of individual words (Pilán et al.,2014; Sung et al., 2015; Schumacher et al., 2016). Salientfeatures may potentially be derived from semantic frames,

2.

Research Questions

We hypothesize that the verb usage patterns encoded inverb frames can be associated with different levels of reading difficulty. The distribution of frame-related attributes ina text may therefore be correlated with readability. Moreprecisely, this paper tests the following hypotheses:• H1: Non-core frame elements are more frequently
used in more difficult texts (Section 5);• H2: Core frame elements are more frequently omitted
in more difficult texts (Section 6);• H3: For verbs that can take either a noun phrase (NP)or a clause as argument, NPs are more frequently chosen in more difficult texts (Section 7).
• H4: Metaphor is more frequently used in more difficult texts (Section 8).
The rest of this paper is organized as follows. After asummary of previous research on readability assessment(Section 3), we describe our dataset (Section 4). We thenpresent results on the four hypotheses above (Sections 5 to
8).

3. Previous Work
This section reviews the variety of lexical, syntactic andsemantic features that have been explored for readability
assessment.

3.1.

Lexical Features

Most readability formulas rely on shallow features such as
word length, sentence length, and vocabulary lists (Kincaid et al., 1975). The Lexile framework incorporates
features derived from word frequencies, for instance lexical richness based on the type-token ratio (Stenner, 1996).

56

Sentence(1a) 我担心你会生病‘I worried you would get sick’(1b) 我担心你的健康‘I worried about your health’(2a) 他很后悔错过了考试‘He regretted missing the exam.’(2b) 他因为错过了考试十分后悔‘Because [he] missed the exam, he felt regretful.’(3a) 他没有把书本放在桌上‘He did not put the book on the desk’(3b) 他没有把问题放在心上‘He did not care about (lit., ‘put on heart’) this question’

ReadabilityLess difficult

Remarks
Clause argument for ‘worry’More difficult

Noun argument for ‘worry’Less difficult

Clause argument for ‘regret’More difficult
Less difficult

Use of reason construction
to express cause for regret
No metaphorical usage

More difficult

Metaphorical usage with ‘put’

Table 1: Sentences with varying reading difficulty due to semantic complexity, despite similar lexical and syntactic
complexity.

4.

More recent work in NLP has made use of n-gram language models (Collins-Thompson and Callan, 2004; Petersen and Ostendorf, 2009), inflectional and derivational
morphology (Hancke et al., 2012), verbal morphology, verb
tense and mood-based features (Dell’Orletta et al., 2011;
François and Fairon, 2012). Psycholinguistic properties,
such as the concreteness, imageability and meaningfulnessof words (Wilson, 1988), and the age of acquisition (Kuperman et al., 2012), have also been shown to be helpful.

3.2.

4.1.

Mandarin VerbNet

Mandarin VerbNet is a verbal semantic database with annotation of frame-based constructional features (Liu andChiang, 2008). In addition to frame elements, its frames
make use of a schema-based meaning representation andconstructional patterns. Adopting a hybrid approach tothe semantic analysis of the lexical-constructional behaviorof Chinese verbs, it incorporates tenets of Frame Semantics (Fillmore and Atkins, 1992) and Construction Grammar (Goldberg, 1995).
We selected ten verbs from three different frame categories
for this study (Table 2). For more reliable statistics onframe distribution with respect to grade, we have deliberately chosen common verbs that are used in a wide range of
grades.

Syntactic Features

Even if a sentence is composed of simple words, it can still
be difficult to understand because of complicated syntactic structure. Early models often use sentence length andclause length as proxies for syntactic complexity. Morerecent ones incorporate part-of-speech (POS) features, including the frequency of coordination and subordination;
the nominal ratio and the pronoun/noun ratio (Pilán etal., 2014); the number of different kinds of pronouns andconjunctions (Sung et al., 2015); and more generally, thepercentage and diversity of POS tags (Vajjala and Meurers, 2014). Parse tree depth, parse scores, subtree patterns (Heilman et al., 2008; Schumacher et al., 2016) anddependency distance (Liu, 2008) have also been found to
be useful.

3.3.

Data

This section first presents Mandarin VerbNet and the verbs
to be analyzed (Section 4.1), and then describes the corpus
of graded texts on which our analysis is based (Section 4.2).

4.2.

Corpus of Graded Text

We performed our analysis on a corpus of Chineselanguage textbooks constructed at Ludong University,
China.1 The 5-million-character corpus consists of morethan 6000 articles, taken from 368 textbooks spanning thetwelve grades in the curriculum for Chinese language inmainland China. For analysis purposes, the grades are divided into three categories:

Semantic Features

Lexical complexity and syntactic complexity do not cover
all factors that influence readability. As discussed in Section 1, the (b) sentences in Table 1 can be expected to be
more difficult to read than their (a) counterparts, despite
their similar lexical and syntactic complexity.
Many readability models have therefore incorporated measures on semantic complexity. Common features include
the average number of senses per word (Pilán et al., 2014);
the ratio of active/passive voice (Graesser et al., 2011); thenumber of content words and the number of semantic categories in a sentence (Sung et al., 2015); the number of
unique entities per document and the average number ofwords per entity; and the semantic probability of a sentence, according to a semantic network (vor der Brück etal., 2008).

• 1-3: Grades 1 through 3;• 4-6: Grades 4 through 6;• 7+: Grades 7 through 12.

Table 2 shows the number of sentences in which the ten
verbs appear. We manually and exhaustively annotated theverb frame usage in these sentences.

5.

Use of Non-core Frame Elements

Similar to FrameNet, Mandarin VerbNet distinguishes between “core” or “non-core” frame elements. Core frame
1

pus.

57

We thank Prof. Xu Dekuan for providing access to this cor-Frame Category

Verb

CAUSED-MOTION

放
丢
发现
注意到
思考
担心
吸引
感动
着急
后悔

COGNITIONEMOTIONfàng ‘put’
diū ‘cast away’fāxiàn ‘discover’zhùyı̀dào ‘notice’sı̄kǎo ‘reflect’dānxı̄n ‘worry’xı̄yı̌n ‘attract’gǎndòng ‘be moved’zhāojı́ ‘be anxious’hòuhuı̌ ‘regret’

Grades 1-3
24
13
125
5
7
16
19
4
26
9

# sentences
Grades 4-6 Grades 7+
26
49
17
24
323
404
18
56
30
37
48
56
50
49
20
30
28
24
18
18

Total
99
54
852
79
74
120
118
54
78
45

Table 2: Verbs used in our analysis, and the number of sentences in which they appear in our corpus among texts of thelower and upper grades (see Section 4.2).
Verbdānxı̄n‘worry’gǎndòng‘be moved’xı̄yı̌n‘attract’fāxiàn‘discover’hòuhuı̌‘regret’Frame Element
Exp
Target-PossibleSituation
Beneficiary
Target Entity
Stim
Affector
Affectee
ResultMeans
AffecteeAct
Affector
Means
Affectee Theme
ResultReasonCognizer
Phenomenon
Means
Topic
MediumInstrument
Exp
Expressor
Given-fact
ReasonStim
Given-factdescription

Lowest
1
1

Type
Core
Core

Selected examples
我Exp 担 心 我会生病Target-possible-situation
IExp worried I would get sickTarget-possible-situation

2
4
7
2
2
4
6
1
1
2
4
6
7
7
1
1
1
2
2
7
2
2
3
6
7
10

Non-coreCore
Non-coreCoreCore
Non-core
Non-coreCore
Non-coreCore
Non-coreCore
Non-core
Non-coreCoreCore
Non-coreCore
Non-core
Non-coreCore
Non-CoreCore
Non-CoreCoreCore

我Exp 担 心 你的健康Target-entity
IExp worried about your healthTarget-entity
观众Affectee 也都被感
感动了， 大家拍着手Result ...
The audienceAffectee were moved, clappingResult ...
他们Affector ... 便 以死Means 来感
感动 ...
Through their deathMeans , theyAffector moved ...
作者Affector 总是 用思想感情Means 吸引 你Affectee
The authorAffector attracts youAffectee with her emotionsMeans ...
他Affector 吸 引着 孩子的心Affectee , 让人总在想着他呢Result
HeAffector so attracted the kidsAffectee , making them thinkof himResult ...
我Cognizer 在桌上Medium 发 现 一本书Topic
ICognizer found a bookTopic on the tableMedium从 瞄准镜Instrument 里, 发 现 一个火球穿过Phenomenon ...
From the telescopeInstrument , [we] discovered a fireballshootingPhenomenon ...
如果 不复习Stim 他Exp 会后
后悔 的HeExp would regret if [he] didn’t reviewStim
他Affector 因为错过了考试Reason 十分后
后悔Because of missing the examReason , heExp felt regretful.

Table 3: Verbs and their frame elements, showing the lowest grade in which the frame element appears.

elements are fundamental; they commonly appear as a necessary argument in a sentence and plays an essential rolein the event frame. Non-core frame elements are optional;
they are “potentially relevant”, and can be added to a sentence as an adjunct (Liu and Chiang, 2008).

which a frame element occurs. As shown in Table 3, many
non-core frame elements are found only at higher grades.
For xı̄yı̌n ‘attract’, for example, Result and Reason do
not appear until Grade 7.
To test H1, we calculated the percentage of sentences with
non-core frame elements at each grade. The verb zhùyı̀dào‘notice’ does not employ non-core frame elements at any
grade level in our dataset. As shown in Table 4, the overall

According to the first hypothesis (H1), non-core frame elements are used more frequently in more difficult text. As apreliminary investigation, we identified the lowest grade at

58

Verb
zhùyı̀dào ‘notice’diū ‘cast away’sı̄kǎo ‘reflect’gǎndòng ‘be moved’fàng ‘put’
dānxı̄n ‘worry’hòuhuı̌ ‘regret’zhāojı́ ‘be anxious’xı̄yı̌n ‘attract’fāxiàn ‘discover’

1-3
0%
0%
0%
0%
4.2%
6.3%
11.1%
15.4%
26.3%
40.8%

Grades
4-6
0%
5.9%
7.0%
55.0%
7.7%
12.5%
0%
35.7%
18.0%
32.2%

Verb

7+
0%
8.3%
22.0%
40.0%
2.0%
10.7%
5.6%
20.8%
42.9%
46.0%

gǎndòng ‘be moved’xı̄yı̌n ‘attract’zhùyı̀dào ‘notice’hòuhuı̌ ‘regret’dānxı̄n ‘worry’zhāojı́ ‘be anxious’fàng ‘put’
fāxiàn ‘discover’sı̄kǎo ‘reflect’diū ‘cast away’

Table 4: (H1) Percentage of sentences with non-core frame
elements.

Verb
fàng ‘put’
diū ‘cast away’fāxiàn ‘discover’zhùyı̀dào ‘notice’gǎndòng ‘be moved’xı̄yı̌n ‘attract’dānxı̄n ‘worry’hòuhuı̌ ‘regret’sı̄kǎo ‘reflect’zhāojı́ ‘be anxious’

7+
100%
100%
85.7%
97.2%
71.4%
79.2%
42.9%
76.0%
67.6%
66.7%

1-3
100%
100%
100%
100%
100%
94.7%
75%
44.0%
14.3%
0%

Grades
4-6
100%
100%
100%
94.4%
70.0%
90.0%
66.7%
66.7%
23.3%
10.7%

7+
100%
100%
99.4%
98.2%
86.7%
98.0%
67.6%
66.7%
32.5%
16.7%

Table 6: (H2) on objects: Percentage of sentences withframe elements serving as the direct object of the verb.

Omission of Core Frame Elements

To reduce repetition, a writer may omit a verb argumentfrom a sentence, expecting the reader to infer the information from the context. This phenomenon is frequent in
Chinese even for some core arguments; for example, prodropped subjects account for more than 36% of the subjects
in Chinese sentences (Kim, 2000). The number of zero pronouns is likely correlated with the effort needed for resolution. According to the second hypothesis (H2), omissionof core frame elements is more frequent in more difficult
texts.

6.1.

Grades
4-6
100%
100%
100%
100%
95.8%
78.6%
69.2%
87.3%
80.0%
82.4%

Table 5: (H2) on subjects: Percentage of sentences withframe elements serving as the subject of the verb.

statistics of the remaining verbs lend support to the hypothesis. Eight of the verbs2 exhibit a lower percentage of sentences with non-core frame elements at grades 1-3 than at
higher grades. Consider gǎndòng ‘be moved’ as an example: non-core frame elements for this verb3 do not appear
in grades 1-3, but account for 55.0% of the sentences ingrades 4-6 and 40.0% in higher grades. The difference between grades 4-6 and 7+, however, is less clear-cut. Five of
the verbs exhibit higher rates of non-core frame elements
in grades 7+, while four exhibit lower rates. More finegrained analysis is necessary to account for the underlying
differences.

6.

1-3
100%
100%
100%
100%
100%
96.2%
91.7%
91.2%
85.7%
84.6%

gap between 1-3 and 4-6.5

6.2.

Objects

We next investigate frame elements that normally occupy
the object position after the verb. Table 6 shows the proportion of sentences containing these frame elements.6 Twoof the verbs, fàng ‘put’ and diū ‘cast away’, always have
explicit objects in sentences at all grades, as the frame element Figure is indispensable for their semantic expression. Among the remaining eight verbs, the trend is morenuanced compared to the omission of subjects. We will focus on comparing grades 1-3 with the higher grades. Consistent with H2, four of these verbs — fāxiàn ‘discover’,zhùyı̀dào ‘notice’, gǎndòng ‘be moved’ and dānxı̄n ‘worry’— have more sentences in grades 1-3 containing objects.
In contrast, for the other four verbs, the sentences in grades
1-3 are more likely to omit the object. These results sug-

Subjects

We first examine frame elements that normally occupy the
subject position before the verb. Table 5 shows the proportion of sentences containing these frame elements.4 For
the verbs gǎndòng ‘be moved’ and xı̄yı̌n ‘attract’, this proportion is constant since all of their sentences at all grades
contain subjects. The hypothesis is however supported by
the remaining eight verbs. Generally, more sentences lacksubjects in the higher grades than in the lower ones. The
gap between grades 4-6 and 7+ is usually larger than the5
For the verb dānxı̄n ‘worry’, for example, all sentences ingrades 1-3 have subjects, as do 95.8% of the sentences in grades
4-6. However, the figure drops to 71.4% at grades 7+. The only
exception to this trend is observed for zhāojı́ ‘be anxious’.
6
Among the ten verbs analyzed, depending on their frame
category, these frame elements can be Affectee, Figure,
Given Fact, Given Fact Description, Phenomenon,
Topic, Target, Target Empathy, Target-Entity,
Target-Situation or Target-Possible-Situation.

2

With the exception of hòuhuı̌ ‘regret’.
See Table 3 for example sentences for the non-core frame elements Result and Means.
4
Among the ten verbs analyzed, depending on their frame category, these frame elements can be Agent, Cognizer, Exp,
Placer, Affector or Affectee.
3

59

Verb
zhùyı̀dào‘notice’fāxiàn‘discover’dānxı̄n‘worry’hòuhuı̌‘regret’Argument
typeclauseNPclauseNPclauseNPclause
NP

1-3
80%
20%
78.4%
21.6%
75.0%
0%
44.4%
0%

Grades
4-6
55.6%
38.9%
57.6%
51.7%
58.3%
8.3%
66.7%
0%

Verb

7+
41.1%
53.6%
55.2%
44.1%
53.6%
14.3%
50.0%
16.7%

fàng ‘put’
diū ‘cast away’
Table 8:usage.

Grades4-6
7+19.2% 30.60%
11.8% 33.30%(H4) Percentage of sentences with metaphoric

Table 1. In (3a), the verb 放 fàng ‘put’ is used in its regularsense, ‘put a book on the table’. In (3b), however, it isused in the metaphorical sense in the verb phrase 放在心上(‘remember’; literally, “put on the heart”), which is moredifficult to interpret.
Our analysis centered on the two verbs in our dataset —fàng ‘put’ and diū ‘cast away’ — that are more productive in metaphorical usage. In non-metaphorical usage, theframe elements Ground-Location (for fàng ‘put’) and
Figure (for diū ‘cast away’) typically expect physical locations and objects. That is not necessarily the case inmetaphorical usage, which allows abstract entities such as‘worry’ (e.g., “cast away one’s worry”) or ‘heart’ (“put on
the heart”).
The fourth hypothesis (H4) predicts metaphorical usage to
be more frequent in more difficult texts. Table 8 presents
evidence for this hypothesis. For both verbs, no metaphor
is employed in the texts for grades 1-3. The percentage
of metaphorical usage increases to 19.2% and 11.8%, respectively, at grades 4-6. The higher grades see even more
substantial amount of metaphorical usage, at 30.60% and33.30%.

Table 7: (H3) Percentage of sentences with clause or nounphrase as argument to the verb.
gest that the impact of the object on text difficulty differs
according to usage patterns of individual verbs. The presence of objects in the verbs zhāojı́ ‘be anxious’ and hòuhuı̌‘regret’, for instance, can make a sentence harder to read.
Since these verbs do not take objects in most instances, theabsence of the object should perhaps not be considered anomission.

7.

1-3
0.0%
0.0%

Clause vs. Noun Phrase

As illustrated by the verb dānxı̄n ‘worry’ in sentences (1a)and (1b) in Table 1, some verb arguments can be either anoun phrase (NP) or a clause. The distinction is reflectedby the frame element. Sentence (1a), which contains theclause ‘you would get sick’ as object, has the frame element Target-possible-situation. In contrast,
sentence (1b), with the NP ‘your health’ as object, has theTarget-entity element. Similar distinctions are madein other frame categories, for example with Phenomenon(clause) vs. Topic (NP), and Given-fact (clause) vs.
Given-fact-description (NP).
According to the third hypothesis (H3), given a choice between NP and clause for an eventive complement, NP or
event nominal is more often used in difficult texts than ineasier ones. We analyzed the four verbs in our dataset thatoffer this choice, and the overall statistics support the hypothesis (Table 7). For all four verbs, sentences in grades
1-3 substantially prefer clause over NP, and the gap narrows in grades 7+; in the case of zhùyı̀dào ‘notice’, clauses
are even outnumbered by NPs in grades 7+. This observation suggests that for these verbs, a clause may be easier forless proficient readers to understand than a noun, especiallywhen it expresses an abstract meaning.
When taking grades 4-6 into account, the statistics are notalways consistent with H3. Consider the case of fāxiàn ‘discover’. While the preference for clause over NP decreases
from grades 1-3 (a difference of 56.8%) to grades 4-6 (adifference of 5.9%), it unexpectedly increases again from
grades 4-6 to grades 7+ (a difference of 11.1%).

9.

Conclusions

We have presented the first investigation on the correlation
between verb frames and text difficulty. Based on Mandarin
VerbNet (Liu, 2016; Liu and Chang, 2016; Liu, 2018; Liu,2019), our analysis of ten common Chinese verbs showedthat at higher grades, there is generally more frequent use ofnon-core frame elements; more frequent omission of coreframe elements that normally occupy the subject position
before the verb; increased preference for a noun phrase overa clause as verb argument; and more frequent metaphoricalusage. These patterns can potentially help improve a readability assessment model.
We plan to pursue two directions in future work. First, weplan to expand our analysis to a larger set of verbs from diverse frame categories. Second, we intend to incorporate
frame patterns as features in a system for readability prediction.

10.

Acknowledgements

We thank Prof. Dekuan Xu for providing access to the
corpus of Chinese-language textbooks, and Tianqi He andYingying Ye for their assistance. We gratefully acknowledge support from a grant (Project #9360163) from theHong Kong Institute for Data Science at City Universityof Hong Kong.

8. Metaphor
Metaphorical usage which involves cognitive transfer fromone domain to another tends to make a sentence harder toread, even when the vocabulary and syntactic structures are
simple. Consider the example sentences (3a) and (3b) in60

11. Bibliographical References

approach. In Minghui Dong, et al., editors, Proc. International Conference on Asian Language Processing(IALP), pages 30–36. New York: Institute of Electricaland Electronics Engineers (IEEE).
Liu, M. and Chiang, T. Y. (2008). The construction of
mandarin verbnet: A frame-based study of statement
verbs. Language and Linguistics, 9(2):239–270.
Liu, H. (2008). Dependency Distance as a Metric of Language Comprehension Difficulty. Journal of Cognitive
Science, 9(2):159–191.
Liu, M. (2016). Emotion in lexicon and grammar: lexicalconstructional interface of Mandarin emotional predicates. Lingua Sinica, 2.4.
Liu, M. (2018). A frame-based morpho-constructional approach to verbal semantics. In Chunyu Kit et al., editors,
Frontiers of Empirical and Corpus Linguistics. China
Social Sciences Press, Beijing, China.
Liu, M. (2019). The construction and annotation of a semantically enriched database: The mandarin verbnet andits nlp application. In From Minimal Contrast to Meaning Construct: Corpus-based, Near Synonym Driven Approaches to Chinese Lexical Semantics (Vol. 9, Frontiers in Chinese Linguistics Book Series), pages 257–
272. Springer-Nature and Peking University Press.
Petersen, S. E. and Ostendorf, M. (2009). A machinelearning approach to reading level assessment. Computer Speech and Language, 23(1):89–106.
Pilán, I., Volodina, E., and Johansson, R. (2014). Rulebased and Machine Learning Approaches for Second
Language Sentence-level Readability. In Proc. 9th Workshop on Innovative Use of NLP for Building Educational
Applications.
Ruppenhofer, J. and Rehbein, I. (2012). Semantic Frames
as an Anchor Representation for Sentiment Analysis.
In Proc. 3rd Workshop in Computational Approaches
to Subjectivity and Sentiment Analysis (WASSA), pages
104–109.
Schumacher, E., Eskenazi, M., Frishkoff, G., and CollinsThompson, K. (2016). Predicting the relative difficultyof single sentences with and without surrounding context. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1871–1881.
Stenner, A. J. (1996). Measuring reading comprehension
with the Lexile framework. In Proc. Fourth North American Conference on Adolescent/Adult Literacy.
Sung, Y.-T., Lin, W.-C., Dyson, S. B., Chang, K.-E., andChen, Y.-C. (2015). Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with
the CEFR. The Modern Language Journal, 99(2):371–
391.
Vajjala, S. and Meurers, D. (2014). On Assessing theReading Level of Individual Sentences for Text Simplification. In Proc. 14th Conference of the European
Chapter of the Association for Computational Linguistics (EACL).
vor der Brück, T., Hartrumpf, S., and Helbig, H. (2008).
A Readability Checker with Supervised LEarning using
Deep Syntactic and Semantic Indicators. In Proc. 11th
International Multiconference: Information Society.

Carrión, O. B. (2006). Framenet as a Corpus Tool for
the Learning of Second Languages and for the Lexical
Awareness of one’s First Language. Porta Linguarum,
6:67–76.
CEFR. (2001). Common European Framework of Reference for Languages: Learning, Teaching, Assessment.
Cambridge University Press, Cambridge.
Collins-Thompson, K. and Callan, J. (2004). A languagemodelling approach to predicting reading difficulty. In
Proc NAACL-HLT, Boston, MA.
Dell’Orletta, F., Montemagni, S., and Venturi, G. (2011).
Read-it: Assessing Readability of Italian Texts with aView to Text Simplification. In Proc. 2nd Workshop on
Speech and Language Processing for Assistive Technologies.
Fader, A., Soderland, S., and Etzioni, O. (2011). Identifying Relations for Open Information Extraction. In Proc.
EMNLP.
Fillmore, C. J. and Atkins, B. T. (1992). Towards a Framebased organization of the lexicon: the semantics of RISK
and its neighbors. In Adrienne Lehrer et al., editors,
Frames, Fields, and Contrasts: New Essays in Semantics, pages 75–102, Hillsdale. Lawrence Erlbuan.
François, T. and Fairon, C. (2012). An “AI Readability”
Formula for French as a Foreign Language. In Proc.
EMNLP-CONLL.
Gildea, D. and Jurafsky, D. (2002). Automatic Labeling of
Semantic Roles. Computational Linguistics, 28(3):245–
288.
Goldberg, A. E. (1995). Constructions: A Construction
Grammar Approach to Argument Structure. Universityof Chicago Press, Chicago.
Graesser, A. C., McNamara, D. S., and Kulikowich, J. M.
(2011). Coh-metrix providing multilevel analyses of text
characteristics. Educational Researcher, 40(5):223–234.
Hancke, J., Vajjala, S., and Meurers, D. (2012). Readability Classification for German using Lexical, Syntacticand Morphological Features. In Proc. 24th International
Conference on Computational Linguistics (COLING).
Heilman, M., Collins-Thompson, K., and Eskenazi, M.
(2008). An Analysis of Statistical Models and Features
for Reading Difficulty Prediction. In Proc. Third Workshop on Innovative Use of NLP for Building Educational
Applications.
Kim, Y.-J. (2000). Subject/object drop in the acquisition of
Korean: A cross-linguistic comparison. Journal of East
Asian Linguistics, 9:325–351.
Kincaid, J. P., Fishburne, R. P., Rogers, R. L., and Chissom,
B. S. (1975). Derivation of new readability formulas(automated readability index, fog count, and flesch reading ease formula) for Navy enlisted personnel. In Research Branch Report 8–75. Chief of Naval TechnicalTraining: Naval Air Station Memphis.
Kuperman, V., Stadthagen-Gonzalez, H., and Brysbaert, M.
(2012). Age-of-Acquisition ratings for 30,000 English
words. Behavior Research Methods, 44(4):978–990.
Liu, M. and Chang, J.-C. (2016). Semantic annotation for
Mandarin verbal lexicon: A frame-based constructional

61

Wilson, M. D. (1988). The MRC Psycholinguistic
Database: Machine Readable Dictionary, Version 2. Behavioral Research Methods, Instruments and Computers,
20:6–11.
Xu, F. and Li, T. (2011). Semantic Frame and EVT for
Chinese EFL Learners. Journal of Language Teaching
and Research, 2(3):649–654.
You, L. and Liu, K. (2005). Building Chinese FrameNet
Database. In Proc. IEEE NLP-KE.

62

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 63–69
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Deriving a PropBank Corpus from Parallel FrameNet and UD Corpora
Normunds Gruzitis, Roberts Dargis, Laura Rituma, Gunta Nespore-Berzkalne, Baiba Saulite
Institute of Mathematics and Computer Science, University of Latvia
Raina bulv. 29, LV-1459, Riga, Latvia{normunds.gruzitis, roberts.dargis, laura.rituma, gunta.nespore, baiba.valkovska}@lumii.lv
Abstract
We propose an approach for generating an accurate and consistent PropBank-annotated corpus, given a FrameNet-annotated corpus
which has an underlying dependency annotation layer, namely, a parallel Universal Dependencies (UD) treebank. The PropBank
annotation layer of such a multi-layer corpus can be semi-automatically derived from the existing FrameNet and UD annotation
layers, by providing a mapping configuration from lexical units in [a non-English language] FrameNet to [English language] PropBank
predicates, and a mapping configuration from FrameNet frame elements to PropBank semantic arguments for the given pair of a
FrameNet frame and a PropBank predicate. The latter mapping generally depends on the underlying UD syntactic relations. To
demonstrate our approach, we use Latvian FrameNet, annotated on top of Latvian UD Treebank, for generating Latvian PropBank incompliance with the Universal Propositions approach.
Keywords: PropBank, FrameNet, Universal Dependencies, Universal Propositions, Latvian

1.

Introduction and Related Workdencies (syntactic roles of the root tokens of the respective
subtrees). For some languages (e.g. Hindi/Urdu and Brazilian Portuguese) both kinds of syntactic representations andboth kinds of PropBank-treebank mappings are available.
While dependency trees are often considered a more convenient and straightforward intermediate representation forrobust automatic SRL, as it has been proved by state-ofthe-art SRL parsers (Cai and Lapata, 2019), the use of acommon inventory of PropBank framesets would facilitatecross-lingual SRL and the downstream applications likecross-lingual information extraction.
The Universal Propositions (UP) project1 proposes to use
the English PropBank framesets for universal SRL, on topof the Universal Dependencies (UD) syntax trees. The underlying UD representation (Nivre et al., 2016) facilitatescross-lingual semantic parsing even more.
Akbik et al. (2015) present a method for automatic projection of English framesets to a target language, and they haveapplied this method to generate UP propbanks for multiple languages. In this paper, we present our work which
contributes to the UP initiative. We propose an alternative
approach for generating accurate and consistent UP propbanks for languages that have a FrameNet-annotated corpus where FrameNet annotations are specified on top of a
UD treebank, or a dependency treebank in general.
To some extent, our approach is similar to the one applied
to convert the SALSA Corpus for German into a PropBanklike corpus for the CoNLL 2009 shared task (Hajič et al.,2009). The SALSA corpus (Burchardt et al., 2006) uses
semantic roles in the FrameNet paradigm (Fillmore et al.,2003), annotated on top of a treebank, which were semiautomatically converted to the respective PropBank arguments. The semantic predicates, however, remain Germanspecific in the converted SALSA corpus. In contrast, wereuse semantic predicates from the English PropBank (following the UP approach), which was the most challenging
part in the Latvian FrameNet-to-PropBank conversion. The

Proposition Bank or PropBank (Palmer et al., 2005) is(i) a shallow semantic representation for the annotationof predicate-argument structures, (ii) a lexicon of English
verbs and their semantic predicates (frames) and semantic
arguments (roles), and (iii) a large annotated text corpus
of English, where the semantic roles of each predicate instance are added to the syntactic structures of the underlying treebank.
Since PropBank uses a small set of semantic roles which
are defined on a verb-by-verb basis, and the annotated corpus provides broad-coverage training data, it is an attractive approach for robust automatic semantic role labelling,
SRL (Cai and Lapata, 2019). This has also encouraged
extensive use of PropBank framesets (coarse-grained verb
senses each having a specific set of semantic arguments orroles) in the whole-sentence Abstract Meaning Representation (AMR) approach (Banarescu et al., 2013).
Following the work on English, PropBank-style corpora
have been created for a number of languages. Apart fromother aspects, creation of a propbank depends on a fundamental decision: whether to define language-specificframesets or to re-use the English PropBank framesets.
In most projects, language-specific framesets have been
defined and used in the manual or semi-automatic corpus annotation workflow, e.g. for Chinese (Xue, 2008),
Hindi/Urdu (Bhatt et al., 2009) and Finnish (Haverinen etal., 2015). Few attempts have been made to create a nonEnglish propbank by reusing the English PropBank framesets. An example to the latter approach is Brazilian Portuguese PropBank (Duran and Aluisio, 2012), although theuse of English framesets was intended only as an intermediate step on the way to define language-specific framesets.
Another consideration is the underlying syntactic representation – syntactic structures to which the semantic roles areadded. In the case of phrase structure trees (e.g. the English and Chinese treebanks), semantic roles are added toconstituents (phrases). In the case of dependency trees (e.g.
the Finnish treebank), semantic roles are added to depen-1

https://github.com/System-T/
UniversalPropositions

63

LEMMAmācı̄tmācı̄ties
mācı̄ties
dzı̄vot
dzı̄vot
dzı̄vot

UPOS
VERB
VERB
VERB
VERB
VERB
VERB

PREDFN
Education teaching
Education teaching
Memorization
Residence
Dead or alive
Living conditions

PREDPBteach.01
study.01
learn.01
reside.01
live.01
live.02

PREDFNAPREDFN DEP PREDPB APREDPB
Education teaching Student
nsubj study.01 A0
Education teaching Student
obj teach.01 A2
iobj teach.01 A2
Education teaching Student
Education teaching Subject
obj study.01 A1
obj teach.01 A1
Education teaching Subject
Education teaching Teacher obl study.01 A2
Education teaching Teacher nsubj teach.01 A0
Education teaching Institution obl study.01 AM-LOC
Education teaching Institution obl teach.01 AM-LOC
Education teaching Levelobl study.01 AM-LOC
Education teaching Timeobl study.01 AM-TMP
obl teach.01 AM-TMP
Education teaching Time

Table 1: Sample mapping from lexical units (verb-frame
pairs) in Latvian FrameNet (FN) to English PropBank (PB)
predicates (verb sense-specific translation equivalents).

consecutive conversion of FrameNet roles into PropBank
roles is rather straightforward, although it depends on theunderlying UD roles.
On the one hand, FrameNet defines a set of more abstract
semantic frames (compared to PropBank predicates) thatcan be evoked by different target words. On the other hand,
FrameNet uses more fine-grained semantic roles (frame elements), some of which are often not expressed in a sentence as direct syntactic arguments of the predicate. Therefore our proposed FrameNet-to-PropBank conversion approach is unidirectional, i.e., a rather complete PropBank
corpus can be derived from an existing FrameNet corpus(with parallel dependency annotations), however, it would
not be possible to derive a complete FrameNet corpus froman existing PropBank corpus without additional annotationwork.
To demonstrate our approach, we use Latvian UD Treebank(Gruzitis et al., 2018b) and Latvian FrameNet (Gruzitis etal., 2018a) for generating Latvian PropBank, compliant to
the Universal Propositions approach.

2.

Table 2: Mapping from FrameNet (FN) frame elements to
PropBank (PB) semantic roles, taking UD dependency relations (syntactic roles) into account.

the underlying syntactic relations. The successive generation of a PropBank annotation layer is a straightforward
automation.
Since the FrameNet annotation is semantically richer, andit can be non-projective w.r.t. the underlying dependency
tree, some FrameNet frame elements are not transferred to
the PropBank layer, if they are not syntactic arguments ofthe target verb.
To ensure productive work on defining the languagespecific mapping configuration (Latvian FrameNet to English PropBank via Latvian UD Treebank), we have developed a convenient and predictive user interface that exploits a simple but efficient method for sorting candidate
suggestions for LU-to-predicate mapping (Section 2.1) andfor FE-to-argument mapping (Section 2.2). Note that bothkinds of mapping are done on the type level, i.e., no individual occurrences are mapped. Affected corpus examples,
however, are dynamically selected and displayed, whichhelps the annotator to verify the choices made.

Mapping Configuration

Semantic roles in PropBank are much more robust compared to FrameNet frame elements, and the overall PropBank annotation systematically follows the syntactic verbargument structure. Therefore the PropBank layer of sucha multi-layer text corpus can be semi-automatically derived from the existing FrameNet and UD layers of the corpus, by providing (i) a mapping configuration from lexicalunits (LU) in [a non-English language] FrameNet to [English language] PropBank predicates (see Table 1), and (ii)a mapping configuration from FrameNet frame elements
to PropBank semantic arguments for the given pair of a
FrameNet frame and a PropBank predicate, i.e., independently from LUs (see Table 2).
We are building on the previous work on SemLink (Palmer,2009) and Predicate Matrix (Lopez de Lacalle et al., 2016),
although none of the two data sets provide complete mapping suggestions, especially for less frequently used lexicalunits, since the suggestions are corpus-driven. We use thesuggested mapping alternatives between English FrameNet
and English PropBank as a draft configuration. The manualtask for a linguist is to map the LUs from Latvian FrameNetto the semantic predicates of English PropBank, and to verify the mapping between FrameNet frame elements (FE)and PropBank semantic roles, which generally depends on

2.1.

LU-to-predicate mapping

Figure 1 partially illustrates the interface for mapping
FrameNet lexical units (verb-frame pairs) to the corresponding PropBank predicates.
In total, there are nearly 11,000 English PropBank framesets, therefore an efficient method to narrow down the LUto-predicate mapping candidates is necessary.
Mapping suggestions are extracted from two existing data
sets. First, the SemLink data set was parsed to extractsuggested FrameNet frame candidates (if any) for each
PropBank predicate. Second, additional mapping alternatives between FrameNet frames and PropBank predicates were similarly extracted from the Predicate Matrix data set. Overall, the two data sets provide suggestions for about 90% of Berkeley FrameNet frames reusedin Latvian FrameNet. Although the ultimate mapping
must be provided between the language-specific LUs (verbframe pairs) and the PropBank predicates, not just between
FrameNet frames and PropBank predicates, the candidate
predicates are proposed based on the FrameNet frame.
In addition to SemLink and Predicate Matrix, we also used64

average. When this group of predicate candidates is available, a mapping suggestion proposed by this group is accurate in 79% of the cases.
The second group contains suggestions which are an intersection of suggestions proposed by both SemLink andPredicate Matrix. Although this group is provided for 71%
of all LUs in Latvian FrameNet, it is the first group of suggestions only in 23% of the cases (when no MT-supported
suggestions are available). On average, there are 5 suggestions per LU contained in this group. When this group is
the first available one, a suggestion proposed by this groupis accurate in 27% of the cases.
The remaining groups contain suggestions that are supported only by SemLink or by Predicate Matrix. One ofthese two groups is the first priority group only for 7% of
all LUs.
Note that for 10% of all LUs, the ultimately selected PropBank predicate was not present in any of the suggestion
groups, and the linguist had to find an appropriate predicate
on its own.
Also note that each FrameNet frame has 15 PropBank
predicate suggestions on average due to the highly abstract FrameNet frames that each can be evoked by different target verbs. Consequently, for all LUs of thesame frame, the same 15 candidates (on average) aresuggested for PropBank predicate mapping, except thatthese candidates are grouped differently, based on translation candidates of the target verb. For example, theFrameNet frame Body movement can be evoked by manytarget verbs, and therefore it has 70 PropBank predicate
suggestions, such as clap.01, close.01, kneel.01, nod.01,
and wave.01. However, if we consider, for instance, the LU
aizvērt.VERB.Body movement, the predicate close.01 is the
top suggestion, while for pamāt.VERB.Body movement the
top suggestion is wave.01.

Figure 1: User interface for mapping a LU in Latvian
FrameNet (the verb ‘ķerties’ in the Activity start sense) toa PropBank predicate (begin.01). Candidate predicates are
sorted depending on the sources that have suggested thiscandidate: translation candidates are the most probable, followed by suggestions by SemLink and Predicate Matrix (ifnot already proposed as translation candidates).

2.2.

a state-of-the-art Latvian-English machine translation (MT)system2 to acquire translation candidates for Latvian verbs(lexical units in Latvian FrameNet). Translation candidates
that correspond to PropBank predicates or their aliases (alternative lexical units, or target verbs, suggested by Predicate Matrix) are used to reorder the final suggestions. As it
has turned out, the MT-supported suggestions are the mostuseful and accurate ones.
To make the choice for a linguist easier, the PropBank predicate suggestions are split in the user interface into four priority groups (see Figure 1). Each lower priority group contains suggestions that are not contained by any of the higher
priority groups.
The first group contains LU-to-predicate suggestions thatare supported by Latvian-English verb translation candidates. This group is provided for 61% of all LUs in Latvian
FrameNet, suggesting 2.6 PropBank predicates per LU on
2

FE-to-argument mapping

When a lexical unit (LU) is mapped between FrameNet and
PropBank at the frame-predicate level, the next step is tomap FrameNet frame elements (FE) to PropBank semantic
arguments.
Figures 2 and 3 partially illustrate the interface for mapping
FrameNet frame elements to the corresponding PropBankarguments, depending on the underlying syntactic relations.
In case of FE-to-argument mapping, we consult only thePredicate Matrix data set (in addition to the PropBank data
set itself) to extract and group FE-to-argument mapping
suggestions, since Predicate Matrix is a more recent dataset, and it provides mapping suggestions directly between
FrameNet and PropBank, instead of the indirect SemLink
mappings via VerbNet (Schuler et al., 2000).
For each PropBank frameset, core and non-core arguments
are extracted and grouped separately. The group of corearguments is prioritised over the group of non-core arguments.
Suggestions supported by Predicate Matrix are separated
in the highest priority group. Such priority suggestions
are available for 51% of the required FE-to-argument mappings, typically containing only one suggestion. If thisgroup is present, it always contains the accurate mapping

https://hugo.lv

65

Figure 3: Candidate PropBank argument mappings for theselected pair of a FrameNet frame element (Agent) and a
UD dependency relation (nsubj).

As Figure 2 illustrates, the linguist who verifies the mapping configuration also sees all corpus examples for the
given LU. This not only helps to make decisions in both
LU-to-predicate mapping and FE-to-argument mapping,but also helps to notice inconsistencies and errors in the underlying annotation layers. Such sentences can be marked
with a FixMe tag, indicating the annotation layer and thetype of the issue.
We have identified three types of typical issues so far:Figure 2: User interface for mapping FrameNet frame elements to PropBank semantic arguments for the given LU(ķerties.VERB.Activity start) / predicate (begin.01). In general, the choice depends on the underlying UD relation.

• An incorrect UD relation associated with a FrameNet
frame element, which means that most likely there isan error in the UD annotation layer. The FrameNetto-PropBank mapping user interface allows to filter all
corpus examples (along with their sentence identifiers)
containing this error. The mapping configuration of
LUs containing such issues is left unfinished until theissues are fixed and the mapping can be finalised.

candidate, and a linguist had to make a choice between twoor three candidates only in 10% of the cases; the rest of thecases were unambiguous.
In 49% of all cases when no Predicate Matrix suggestions
are available for a given FE, a linguist first considers thegroup of PropBank core arguments defined for the particular frameset. This group contains 3 candidates on average.
Note that Latvian FrameNet mostly contains annotations ofcore FEs; non-core FEs are annotated rarely (for the time
being). Therefore one of these suggestions is typically thecorrect one, and it is easy to make the choice.
The remaining group of non-core PropBank arguments (theArgM roles) is always present, containing all the possible
ArgM options regardless the particular frameset (except if
already included in the priority group of Predicate Matrix
suggestions). However, this group of suggestions is seldom
consulted by a linguist because of the above mentioned nature of Latvian FrameNet.

2.3.

• An incorrect root node of a subtree of the underlying
UD tree is selected for a FrameNet frame element. It
can also be the case that the whole FrameNet frame
is chosen incorrectly for a particular sentence, and
the PropBank perspective has helped to notice that.
Again, the user interface allows to filter the problematic corpus examples, and the mapping configuration
for the particular LUs is left unfinished until the issues
are fixed.
• The mapping process also encourages to reconsider
the whole LU – whether the selected FrameNet frame
is best suited for the particular verb sense. For instance, we have observed that different verbs are annotated in Latvian FrameNet using the Give impression
frame, however, SemLink suggests the Appearance
frame for the respective PropBank predicates. This
helps to achieve a better consistency for both Latvian
FrameNet and Latvian PropBank.

Elimination of FrameNet and UD errors

A very important side result of the FrameNet to PropBank
mapping process is that it has unveiled a number of annotation errors and inconsistencies both in Latvian FrameNet
and in Latvian UD Treebank.

66

3. Results

dauzı̄ties.VERB.Impact was mapped to three different predicates: beat.02, bang.02 and thud.01. In general, all three
predicates represent a situation when something hits something making a sound. However, each alternative has aslightly different meaning. Another example – the LU
noslaucı̄t.VERB.Emptying – first annotator has selected arather abstract predicate (clear.01), second annotator – amore specific predicate (wipe.01), while third – even a more
specific one (wipe-off.03). These differences illustrate thatthe annotator’s sense of the second language plays an important role.
FE-to-argument Given that annotators have agreed on apredicate, the mapping of FrameNet frame elements to
PropBank semantic arguments is straightforward. Our IAA
experiment shows that annotators can agree in 95.2% of thecases. The remaining 4.8% are cases where at least one
annotator has tagged the given FrameNet frame element or
UD dependency relation as an annotation error to be fixedin the FrameNet or UD layer respectively.

The Latvian FrameNet to PropBank mapping process isnearly finished: we have so far specified mappings for 92%
of the LUs in the latest version of Latvian FrameNet. A
corresponding Latvian PropBank corpus is automaticallyderived, and all the annotation layers of the multi-layer corpus are released as open data.3

3.1.

Data set

Current statistics of the parallel Latvian FrameNet and Latvian PropBank corpora is as follows:• Lexical units For 2,377 (out of 2,577) LUs represented in Latvian FrameNet, a mapping configuration to PropBank has been specified (92.2%). TheseLUs represent word senses of 1,322 (out of 1,358) frequently used verbs represented in Latvian FrameNet(97.3%).
• FrameNet frames For 521 (out of 540) Berkeley
FrameNet frames reused in Latvian FrameNet, at least
one LU has been mapped to PropBank so far (96.5%).
Latvian FrameNet, in turn, covers 44.2% of 1,222
frames defined in Berkeley FrameNet v1.7.

4.

This section summarises discussion of linguistic issues regarding LU-to-predicate and FE-to-argument mapping.

• PropBank predicates Current LU-to-predicate mappings cover 1,033 (out of 10,687) English PropBank
v3.1 predicates (9.7%).

4.1.

Latvian PropBank consists of two data sets: (i) a machinereadable mapping configuration for each LU in Latvian
FrameNet, and (ii) a set of annotated corpus examples in an
extended CoNLL-U format, compliant to Universal Propositions. Latvian FrameNet is a single data set of annotatedcorpus examples in an extended CoNLL-U format.

Inter-annotator agreement

To conduct an inter-annotator agreement (IAA) experiment,
we selected 30 random LUs from Latvian FrameNet to bemapped to PropBank by three linguists experienced in treebanking as well as in frame semantics. The 30 LUs cover205 corpus examples (annotation sets). First, we measured
IAA w.r.t. LU-to-predicate mapping, then – w.r.t. FE-toargument mapping.
LU-to-predicate Statistically, only in 13 cases out of 30(43.3%) all 3 annotators have agreed on the corresponding PropBank predicate for a given LU. In 13 more cases,
at least 2 annotators agreed on the same predicate, thus,
at least 2 of 3 annotators could agree on a predicate in
86.6% of the cases. In the remaining 4 cases, no two annotators could agree on the same predicate. The qualitative analysis, however, shows that the cause of disagreement was mostly due to different preferences when deciding between close translation equivalents (having anequivalent argument structure). For instance, the LU
3

LU-to-predicate mapping

If there are several predicates with similar meaningin PropBank, it is not always clear which of themshould be chosen. If we consider, for instance, theLU parādı̄ties.VERB.Circumscribed existence, its meaning
roughly corresponds to PropBank predicates appear.01,
show up.02 and emerge.02. In such cases, we choose
the predicate with an argument structure that best matches
the argument structure of the Latvian verb, i.e., the predicate that covers as many core FEs of the corresponding
FrameNet frame as possible.
In some cases, more than one PropBank predicate corresponds to the meaning of a LU in Latvian FrameNet – themeaning of the FrameNet frame is more general than themeaning of the candidate PropBank predicates. For instance, the LU izvirzı̄t.VERB.Choosing covers corpus examples izvirzı̄t mērķi (‘to set a goal’) and izvirzı̄t kandidātu (‘to nominate a candidate’), but PropBank does notprovide a predicate that covers both meanings. In suchcases, we consider the possibility of making the sense
split at the FrameNet layer, if possible, by applying different FrameNet frames to represent these differences. Another example: the LU slēgt.VERB.Closure. The FrameNet
frame Closure covers LUs of both meanings: opening andclosing something. In PropBank, there are different predicates for each of the two meanings. In Latvian, however,these both meanings can be expressed by the same verb, using different adverbial modifiers: slēgt ciet (‘to close’) andslēgt vaļā (‘to open’). We do not have a good solution forthis issue yet, although such cases are quite rare.
There are some cases when a LU does not have an appropriate PropBank predicate for mapping, and would require
a constructicon kind of a sense inventory. For instance, theLU klusēt.VERB.Volubility is expressed in English as thepredicate adjective construction ‘to be/keep silent’.

• Corpus examples The LU-to-predicate and FE-toargument mappings specified so far cover 20,054(out of 20,879) annotation sets in Latvian FrameNet(96.0%).

3.2.

Discussion

https://github.com/LUMII-AILab/FullStack67

A related issue are several Latvian verbs with modal meaning, which are not considered as modal verbs. In English
PropBank, modal verbs like must and can are not annotatedas predicates, therefore we cannot select a verbal predicate
for a Latvian verb with such a meaning. However, we canchoose an adjectival predicate, for instance, able.01 or unable.01.
For around 25% of LUs in Latvian FrameNet, it was challenging to select the corresponding semantic predicate fromEnglish PropBank. In such cases, it took up to 1 hour fora linguist to decide the best fitting mapping, sometimes resulting in no mapping at all (see Section 3.1). In the remaining 75% of cases, it took up to 5 minutes for a linguist
to decide the mapping. Overall, it took around 1 person
month (PM) to map the easy cases (around 1,933) and 4
PMs to map the difficult cases (around 644).

4.2.

be consistently and rapidly derived from an existing multilayer corpus that contains both FrameNet and UD annotation layers (or equivalent annotation layers). While mapping lexical units from a non-English FrameNet to English
PropBank predicates is often (around 25% cases) a linguistically challenging task, the mapping at the semantic rolelevel is straightforward, although it depends on the syntactic roles in general. Note that neither SemLink nor Predicate Matrix mappings contain information about the corresponding syntactic roles. This kind of information is created in our approach, and it could be added to these resources.
Although it is often the case that a PropBank corpus iscreated before a FrameNet corpus, as a layer on top ofa treebank, since PropBank closely follows the syntactic
verb-argument structure, it has paid us off to start withthe manual creation of the more abstract FrameNet annotation layer from which the PropBank layer can be derived
semi-automatically. It would not be possible the other way
around.
It is also often the case that language-specific framesets aredefined in advance to create language-specific FrameNet or
PropBank annotations. Our design decision to reuse theexisting framesets of English FrameNet and English PropBank, although introduce some cross-lingual issues, allow
for cross-lingual linguistic studies and for the developmentof cross lingual semantic parsers.

FE-to-argument mapping

There are cases when it is impossible to assign a PropBankargument to a core FE of a FrameNet frame:• In a syntax tree, the potential argument of a PropBank
predicate is not a syntactic argument of this predicate. For example, consider the sentence ļauj man
paskatı̄ties ‘let me look’: the argument ARG0 of thepredicate look.01 semantically is man (‘me’), but syntactically this is an argument of the verb ļaut (‘to let’).
• Similarly, there are cases when a syntax subtree witha verb as its root node depends on another part of thesentence which represents a semantic argument of theverb but is not its syntactic argument. Consider, for instance, the sentence kā pastāstı̄ja organizācija, nebija‘
iespējams lietot elektrı̄bas generatoru(‘as it was toldby the organization, it was impossible to use the power
generator’). The verb pastāstı̄t (‘to tell’) corresponds
to the PropBank predicate tell.01 that has the argument ARG1: utterance, but the utterance itself is represented by the root node of the whole syntax tree on
which the instance of tell.01 depends.

6.

7.

Bibliographical References

Akbik, A., Chiticariu, L., Danilevsky, M., Li, Y.,
Vaithyanathan, S., and Zhu, H. (2015). Generating highquality proposition banks for multilingual semantic rolelabeling. In Proceedings of the 53rd Annual Meeting of
the Association for Computational Linguistics and the
7th International Joint Conference on Natural Language
Processing, pages 397–407, Beijing, China.
Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt,
K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M.,
and Schneider, N. (2013). Abstract Meaning Representation for Sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with
Discourse, pages 178–186, Sofia, Bulgaria.
Bhatt, R., Narasimhan, B., Palmer, M., Rambow,
O., Sharma, D., and Xia, F. (2009). A MultiRepresentational and Multi-Layered Treebank for
Hindi/Urdu. In Proceedings of the 3rd Linguistic Annotation Workshop, pages 186–189, Suntec, Singapore.
Burchardt, A., Erk, K., Frank, A., Kowalski, A., Padó, S.,
and Pinkal, M. (2006). The SALSA Corpus: a German
Corpus Resource for Lexical Semantics. In Proceedingsof the 5th International Conference on Language Resources and Evaluation (LREC), Genoa, Italy.

• A core FE of a FrameNet frame is not defined
as a core argument of the corresponding PropBank predicate. A typical example is the frame
Change position on a scale: in FrameNet, there are
two core FEs – Item (the entity that has a position on
the scale) and Attribute (a scalar property that the Item
possesses) – that both correspond to one argument of acorresponding PropBank predicate. Consequently, the
FE Item is not mapped to an argument, if both Item
and Attribute are present in the sentence.
The time spent to provide mapping at the semantic rolelevel is included in the estimated time spent to provide
mapping from lexical units in Latvian FrameNet to English
PropBank predicates (see Section 4.1).

5.

Acknowledgements

This work has received financial support from the European Regional Development Fund under the grant agreements 1.1.1.1/16/A/219 and 1.1.1.2/VIAA/1/16/188. Laterdevelopment was supported by the Latvian Council of Science under the grant agreement lzp-2018/2-0216.

Conclusion

We have demonstrated in practice that a quality PropBankcompliant lexical database and annotated text corpus can68

Cai, R. and Lapata, M. (2019). Semi-Supervised Semantic Role Labeling with Cross-View Training. In Proceedings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 1018–1027.
Duran, M. S. and Aluisio, S. M. (2012). Propbank-Br: a
Brazilian Treebank annotated with semantic role labels.
In Proceedings of the 8th International Conference on
Language Resources and Evaluation, pages 1862–1867,
Istanbul, Turkey.
Fillmore, C. J., Johnson, C. R., and Petruck, M. R. (2003).
Background to FrameNet. International Journal of Lexicography, 16(3):235–250.
Gruzitis, N., Nespore-Berzkalne, G., and Saulite, B.
(2018a). Creation of Latvian FrameNet based on Universal Dependencies. In Proceedings of the International
FrameNet Workshop 2018: Multilingual FrameNets andConstructicons (IFNW), pages 23–27, Miyazaki, Japan.
Gruzitis, N., Pretkalnina, L., Saulite, B., Rituma, L.,
Nespore-Berzkalne, G., Znotins, A., and Paikens, P.
(2018b). Creation of a Balanced State-of-the-Art Multilayer Corpus for NLU. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC), pages 4506–4513, Miyazaki, Japan.
Hajič, J., Ciaramita, M., Johansson, R., Kawahara, D.,Martı́, M. A., Màrquez, L., Meyers, A., Nivre, J., Padó,S., Štěpánek, J., Straňák, P., Surdeanu, M., Xue, N.,
and Zhang, Y. (2009). The CoNLL-2009 Shared Task:
Syntactic and Semantic Dependencies in Multiple Languages. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL): Shared
Task, pages 1–18.
Haverinen, K., Kanerva, J., Kohonen, S., Missilä, A., Ojala,S., Viljanen, T., Laippala, V., and Ginter, F. (2015).
The Finnish Proposition Bank. Language Resources and
Evaluation, 49(4):907–926.
Lopez de Lacalle, M., Laparra, E., Aldabe, I., and Rigau, G.
(2016). A Multilingual Predicate Matrix. In Proceedings of the 10th International Conference on Language
Resources and Evaluation, pages 2662–2668, Portoroz,
Slovenia.
Nivre, J., de Marneffe, M.-C., Ginter, F., Goldberg, Y.,
Hajič, J., Manning, C. D., McDonald, R., Petrov, S.,
Pyysalo, S., Silveira, N., Tsarfaty, R., and Zeman, D.
(2016). Universal Dependencies v1: A Multilingual
Treebank Collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation, pages 1659–1666.
Palmer, M., Gildea, D., and Kingsbury, P. (2005). The
Proposition Bank: An Annotated Corpus of Semantic
Roles. Computational Linguistics, 31(1):71–106.
Palmer, M. (2009). SemLink: Linking PropBank, VerbNet
and FrameNet. In Proceedings of the Generative Lexicon
Conference, pages 9–15, Pisa, Italy.
Schuler, K. K., Dang, H. T., and Palmer, M. (2000). ClassBased Construction of a Verb Lexicon. In Proceedingsof the 17th National Conference on Artificial Intelligence(AAAI).

Xue, N. (2008). Labeling Chinese Predicates with Semantic Roles. Computational Linguistics, 34(2):225–255.

69

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 70–76
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Building the Emirati Arabic FrameNet
A. Gargett(1), T. Leung(2)(1) Hartree Centre (Science & Technology Facilities Council),
United Kingdom (andrew.gargett@stfc.ac.uk)(2) United Arab Emirates University,
United Arab Emirates (leung@uaeu.ac.ae)
Abstract
The Emirati Arabic FrameNet (EAFN) project aims to initiate a FrameNet for Emirati Arabic, utilizing the Emirati Arabic Corpus. The
goal is to create a resource comparable to the initial stages of the Berkeley FrameNet. The project is divided into manual and automatic
tracks, based on the predominant techniques being used to collect frames in each track. Work on the EAFN is progressing, and we herereport on initial results for annotations and evaluation. The EAFN project aims to provide a general semantic resource for the Arabic
language, sure to be of interest to researchers from general linguistics to natural language processing. As we report here, the EAFN iswell on target for the first release of data in the coming year.
Keywords: Emirati Arabic, FrameNet, corpus linguisticsc)

1.

Easily accessible and semantically organizedexample sentences, especially useful for language
learning and teaching.
d) Detailed annotations in a gloss language, such as
English in the case of the EAFN project, also asignificant resource for language learning andteaching.
The EAFN will be an invaluable resource for primarytheoretical research on Emirati Arabic, as well as for
additional forms of research crossing a number ofdisciplines, including natural language processing,information retrieval, corpus linguistics, second language
acquisition teaching and research, machine translation,
psycholinguistics, and artificial intelligence. FNs are
currently available for such major languages as English,
German (Rehbein et al. 2012) and Japanese (Ohara 2012).
FNs typically accompany a corpus resource of somedescription, in the target language, and the EAFN will
employ data from the EAC for this purpose.

Introduction

The Emirati Arabic FrameNet (EAFN) project aims toinitiate a FrameNet for Emirati Arabic, utilizing the Emirati
Arabic Corpus (EAC, Halefom et al. 2013). The goal is tocreate a resource comparable to the initial stages of theBerkeley FrameNet (Baker et al. 1998). A FrameNet (FN)
is a corpus-based resource, documenting the semantics of anatural language by linking the “lexical units” (or formmeaning pairings) of the language, such as words, to“frames”. Frames represent the background knowledge
against which lexical units are understood. This background knowledge typically surfaces in how a lexical unit
is used in some situation, together with syntactically related
units, termed “frame elements”. For example, lexical units
such as accuse, blame and esteem all have in common a
JUDGEMENT frame, since they typically involve “aCognizer making a judgment about an Evaluee” (suchframe elements are usually presented capitalized).

1.1This notion of a “Frame Semantics” has been pursued by
Charles Fillmore and colleagues for over 4 decades, with avast body of research to support the approach (e.g. Fillmore
1982. Fillmore et al. 2003), much of which can be accessedfrom the Berkeley FrameNet website.1 Fillmore’s key
insight is that an individual’s use of specific items in theirlanguage is structured by the background knowledge
referred to above. Thus, expressing notions of judging
draws upon a “‘domain’ of vocabulary whose elements
somehow presuppose a schematization of human judgment
and behavior involving notions of worth, responsibility,
judgment, etc.’’ (Fillmore 1982). This enables
generalizations to be made about natural language patterns
in terms of frames, which the FN seeks to capture.

The Corpus of Emirati Arabic (EAC) was established andlicensed by the Department of Linguistics at the United
Arab Emirates University (Halefom et al. 2013). The EAC
is a three-million-word corpus of Emirati Arabic. The dataof the EAC was drawn from various naturalistic sources
such as radio and TV interviews, and daily conversations.
It also consists of some scripted conversations such as TV
dramas and documentaries.
While the current size of the EAC is incomparable withother full-fledged corpora (e.g. British National Corpus),
the EAC is the first annotated corpus of spoken Arabic (cf.
other annotated corpora which are based on Modern
Standard Arabic). It also serves as a useful tool for otherpotential research.

A FN for a natural language thereby provides a rich andhighly nuanced model of the syntactic and semantic
patterns of the language. A FN project has the potential to
add a number of valuable component resources to any
existing corpus:

The EAC is fully annotated using the International
Phonetic Alphabet (IPA). Narrow transcriptions are used inwhich detailed phonetic information instead of the citation
form is described. In addition to the phonetic details, theEAC also provides further annotation including
morphological boundaries (\mb), glossing (\ge), part ofspeech (\ps), and translation (\ft). For in-stance, Tables 1
and 2 contain two annotated examples from the EAC.

a)

Fine-grained information about grammatical roles
and relations.
b) A searchable database of semantically oriented
annotations.

1

The Emirati Arabic Corpus

https://framenet.icsi.berkeley.edu/fndrupal/

70

\ref
\tx
\mb
\ge
\ps
\ft

EAC002.3
do:khawi:hʃu:jalɛsjso:lɛfdo:khawi:hʃu:jalɛsjso:lɛf
look
Hawih
what
doing
talk
v
N
wh
v
v
look, hawih is talking with the donkey

wja
wja
with
prep

lɛħma:rlɛ-ħma:r
the-donkey
d-nTable 1: Example from the EAC
\ref
\tx
\mb
\ge
\ps
\ft

EAC002.59wεsˁaltwεsˁal-t
arrived-2sp
v-pro
Did you arrive?

Table 2: Example from the EAC
1.2

The Emirati Arabic FrameNet Project

searchable database of such patterns in this language. The
information stored in this database will include:

The EAFN project aims to describe the range of semantic
and syntactic combinations of each word in our collectionin each of its senses. As mentioned earlier, this collection
is a sub-corpus extracted from the EAC. To add FN
information, annotators perform computer-assisted
annotation of example sentences, these annotations thenbeing collected in the EAFN database.

1) Raw sound files (from the current Emirati Arabic
Corpus).
2) Arabic and English Transcriptions of the data (avariety of texts in Emirati Arabic).
3) Annotations in the International Phonetic
Alphabet of the files listed in (1) above (from thecurrent Emirati Arabic Corpus).
4) FrameNet annotations, including Frame Element(FE) components for each lexical unit:a) Frame Element (FE) name for lexical unitb) Grammatical function (e.g. subject, object,etc)c) Phrase type (e.g. noun phrase)

Currently, the Berkeley English FrameNet (BEFN), whichbegan in 1997, consists of in excess of 13,000 entries forsenses of lexical units, over 190,000 manually annotatedsentences, rep-resenting more than 1200 frames. Started in2015, the EAFN project originally planned for the initial 2year phase to collect 1000 senses of lexical units, and up to
10,000 annotated sentences, and an expectation of over 100
frames for Emirati Arabic. However, while a substantial
amount of these initial objectives was met, the project wasdelayed until earlier this year, due to a change incircumstances for the first author. We are currently
planning to complete the project by the end of this current
year. Delivery of this database will represent a hugeadvance in knowledge about the language, and lay thegroundwork for development of a rich array of corpusbased and other resources, including descriptive,
computational and teaching and learning resources, for
Emirati Arabic.

2.

MethodThe annotation in this project combines manual andautomatic annotation techniques, and integrates these at
several points, as explained below.

2.1

FrameNet Annotation

Formally, FN annotations are sets of triples that represent
the FE realizations for each annotated sentence, eachconsisting of the frame element’s name (for example,
Food), a grammatical function (say, Object) and a phrase
type (say, noun phrase). Working these out for a newly
encountered language requires a range of decisions to be
made. The first stage of our project involved developing amanual annotation protocol, as well as preparing the subcorpus of EAC texts for annotation (e.g. extracting citation
forms for lexical units).

Our project aims to make a significant contribution to thelevel of resources for Arabic, and especially Emirati
Arabic. The only comparable work to date is from outside
the region, for example, the Leeds University Corpus,
where within the Computer Science Department, theCorpus of Quranic Arabic has been developed. However,
our project differs from such previous work, in that it aims
to deliver large-scale information about deep-level
syntactic (grammatical roles) as well as semantic(argument roles) information for this dialect of Arabic. This
will involve developing novel collection materials, much ofwhich involves using the BEFN.

Developing a FN typically proceeds as follows (Fillmore
and Atkins 1998, Fillmore et al. 2003, Boas 2009):
1) Select the words to be analyzed.
2) Starting from the primary corpus (for the proposed
project, this is the Emirati Arabic Corpus), define
frame descriptions for these words by:a) first, providing in simplified terms a
description of the kind of entity or situation
represented by the frame,

Regarding research outcomes, the project aims to deliver a
store of primary linguistic information about syntactic andsemantic patterns of Emirati Arabic, in a detailed and71b) next, choosing labels for the frame elements(entities or components of the frame),c) finally, collecting words that apparently
belong to the frame.
3) Next, focus on finding corpus sentences in the
primary corpus that illustrate typical uses of thetarget words in specific frames.
4) Then, the sentences from (3) are annotated bytagging them for frame elements.
5) Finally, lexical entries are automatically preparedand stored in the database.
Building a FN for a language from scratch involves a range
of decisions, both linguistic and non-linguistic, raisingquestions about having sufficient data, about the kind of
information to include (dependent on the size and scale ofthe project aims), and also about the tools required to carryout the work. Relatedly, there are questions about theoverall approach to building the FN, such as, whether to
employ largely manual or automatic techniques, there
being advantages and disadvantages on both sides. As canbe seen from the above outlines of a procedure forannotating frames, the complexity of annotating se-mantic
information means manual annotation would be expected
to yield higher quality data, although relatively much more
expensively, whereas automatic annotation would
potentially yield much more, lower quality data albeit far
more cheaply.

range of activities, such as buying, selling, bartering,
trading, and the like. The automatic side of the project aims
to build resources able to leverage this generality of frames,
and thereby interface the English FN with an Arabic
language resource, in order to capture frames common
across each language. Of course, this generality is known
to be limited (e.g. Boas 2009), although, we haveanticipated this with the manual annotation side of our
project, which provides a capacity within our project fordiscovering frames unique to (Emirati) Arabic. Of course,
we acknowledge the difficulty of the challenge involved inbeing able to build such a resource for generating frames
across distinct languages (on this, see e.g. recent work by
Czulo et al. 2019).2 However, we are heartened by a rangeof results, particularly using more recent, scaled up datadriven approaches to Machine Translation, where Deep
Neural Networks are making significant gains in
automating the task of relating the semantics of one
language to another,3 and such work is already yielding
impressive results (e.g. ElJundi et al. 2019).

In our project, we have combined manual and automatic
annotation procedures, to maximize quality and yield, overthe longer term of the project itself. Having a foundation ofmanually annotated frames provides for the EAFN a solidcore on which to build our database. On the other hand, wefaced a lengthy lead-in time for developing suitablesoftware tools for the automatic annotation, and so having
the manual annotation track enabled an immediate start onframe collection. Further, and perhaps more importantly,
the manually collected gold-standard can be used toevaluate the output of automatic annotation, and in turn,manual annotators are able to evaluate the results ofautomatic annotation.

In spring 2014, a research collaboration was establishedbetween the UAE University and the University ofBirmingham with the aim of enriching the EAC byproviding frame annotations. In particular, the research
purpose is to annotate the EAC by adopting the framework
laid out by the Berkeley FN (Baker et al. 1998).
Researchers at the UAEU manually annotated the EAC
with frames. Manual annotation was initiated with native
Arabic speaker annotators being trained by the main EAFN
researchers in frame annotation, in line with the protocols
established by the Berkeley English FN (see section 2.1
above). Annotators then carried out annotation of sentences
sample from the EAC.

2.1.1
Manual annotation
One standard approach to building a large-scale resource
like a FN is to construct a representative sample of thelanguage, to carry out any required corpus analysis. Manual
annotation on the EAFN follows this route, and starts froma sub-corpus specially selected from the EAC for this task.

It might at first seem counter-intuitive that such a resource
can indeed be constructed automatically, given thesemantic complexity of natural language. Ambiguity
abounds in daily communication, making the proposal thata computer system could somehow automatically performaccurate and reliable annotation a somewhat dubious one.
However, it turns out that a key factor in being able toachieve this is the generality of the notion of frame, inparticular its definition in usage-based terms: this definition
leads us to expect that there is a significant overlap between
the set of frames in one language and a completely
unrelated language, since a frame consists of knowledge
about the situations in which a specific language is used,
and a significant number of such situations are common
across languages. For example, while currencies and evenprotocols for proper financial arrangements may differ
from country to country, the Transaction frame, wherein
goods are exchanged for tokens or other goods of equal
worth, is ubiquitous across language settings, covering a

Below are two examples of the same lexical unit ʔəmʃii
which stems from the tri-consonantal root mʃʔ. Allconceptual frames are arrived at through corpus-driven
techniques, rather than through native speaker
introspection. Note that for these initial stages of the
EAFN, labels for frames and FEs have been largely drawn
from the Berkeley English FN, although we fully anticipate
this will need to be revised as the project further develops.

2

3

We would like to thank an anonymous reviewer for
pointing this out (complete with reference).

For a very recent example of this, see work by the
Tsinghua University Natural Language Processing Group(https://github.com/THUNLP-MT/THUMT/)72

\entryid\root\lexeme\gloss\pos\frame\corefe1_label\corefe1_item\corefe1_gloss\corefe2_label\corefe2_item\corefe2_gloss\example\free_trans

EAC001.2mʃʔʔəmʃiiWalk
verb
Self-Motion
Goal
DeliilaDeliilaPathxəTTi
along withʔəmʃii ʕalaa xəTTi deliila
I walk on the path to Deliila

use the Berkeley English FN to help build FNs in otherlanguages (De Cao et al. 2008, Tonelli et al. 2009), often
by linking existing electronic resources, such as adictionary, in a target language to the English FN in someway, in order to label items from this language with frames
from the English FN.
Along these lines, our approach makes use of the English
FN (i.e. the Berkeley English FN), and the English andArabic Wiktionaries. In order to link these resources, wehave customized available NLP tools, and also built suchtools from scratch, in order to use these resources to derive
candidate frames for the EAFN, based on those from theEnglish FN. A major part of this work has involved using
the tools made available by the Ubiquitous Knowledge
Processing (UKP) Lab at the University of Darmstadt inGermany.4 In particular, we employed tools for parsing theEnglish and Arabic Wiktionaries, the Java-based
Wiktionary Library (JWKTL),5 and the UBY database6(Gurevych et al. 2012).

Table 3: Example from the EAFN
\entryid\root\lexeme\gloss\pos\frame\corefe1_label\corefe1_item\corefe1_gloss\example\free_trans

EAC0016mʃʔʔamʃiiWalk
verb
Self_Motion
Pathfi ha Siiħra
in the desertʔamʃi fi ha Siiħra
I will walk in this desert.

Considering the UBY database first, we built tools for
extracting information from the UBY database, in order tobridge the English Wiktionary and the English FN. This
database stores a wealth of Wiktionary-related information
across a range of languages, such as English and Arabic, as
well as links to other resources, in particular the English
FN. We extracted the following information from this:1) For each English Wiktionary lexeme:a) Its written formb) Its sense
2) For each English FrameNet lexical unit matchedto an English Wiktionary lexeme:a) Its index in the English FNb) Its UBY definition [essentially a gloss]

Table 4: Example from the EAFN
The initial annotation process was carried out iteratively in
two phases. During the initial development phase,annotators built the database using backslash entries, as
demonstrated in Tables 1 to 4. In the second phase thisbackslash database was converted into an XML databaseusing custom built parsers; for this phase, the initial
annotation protocol can be refined, involving
reconsideration of the range of categories required forannotating frames in Emirati Arabic, as well as theprocedures for this annotation. For this first round ofannotations, these phases gave rise to the foundation of the
EAFN database; subsequent rounds of annotations
continue to employ both phases, enabling a relatively
flexible arrangement. Furthermore, this approach tobuilding a database requires a minimal setup of a laptop on
which to run a text editor, making the task highly mobile
and relatively technology independent, with annotators
employing relatively lightweight tools. Note that theflexibility of such a set-up potentially facilitates collecting
such data in a more typical fieldwork type setting. Finally,
by extending the custom parsers for the backslash database,
we can extract the required information as XML, thereby
making our database (re)usable in a range of ways.

As well as supplying a ready-made parser for the English
Wiktionary, the JWKTL library provides the means forcustomizing a parser for the Arabic Wiktionary; while
wiktionaries largely overlap in their format, there can besignificant differences from one language to another.
Actual entries in individual language wiktionaries contain
information about a specific lexeme in that language, butalso, importantly for our purposes, links to translations of
this lexeme in wiktionaries of other languages; e.g. theEnglish Wiktionary entry for book links to the Arabic
Wiktionary entry for ‫( ِكتَاب‬this Arabic word being a directtranslation of the English).
Using the newly customized parser for the Arabic
Wiktionary, and the one already available for the English
Wiktionary, we were able to collect information from both
wiktionaries, as follows – for each lexeme in the English
Wiktionary, we collected:

2.1.2
Automatic annotation
This side of the project brings together a variety of Natural
Language Processing tools, aiming to construct a state-ofthe-art system for automatically generating frames for
Emirati Arabic. There have been a variety of attempts to451) Word form
2) Part-of-speech3) All possible definitions for this lexeme
6

https://www.informatik.tu-darmstadt.de/ukp/
https://dkpro.github.io/dkpro-jwktl/

73

https://dkpro.github.io/dkpro-uby/4) The lexeme in the Arabic Wiktionary which theEnglish lexeme has been linked to. For each ofthese Arabic lexemes, we also collected:a) Word formb) Part-of-speechc) Definition [supplied in English]

LUs. Of course, such results need to be treated with a great
deal of caution, and indeed initial evaluation of this datasuggests only a fraction of this data is expected to be of
sufficient quality to justify its being retained for the initial
release of the EAFN database.
While we are listing manually and automatically collected
entries separately at this stage, these will be collected
together for the initial release of the database.

Now, these links between the English to Arabic
Wiktionaries are one-to-many, in that there are many
possible Arabic word forms for each English lexeme. Thismeans we need to carry out a disambiguation of some kind,
if we are to properly align the FN and Wiktionary
resources. Taking this need for disambiguation into
account, we proceed with the alignment in two stages:

Finally, we should also emphasize that the two sources oflanguage are different in dialectal terms: the manual track
works directly from the EAC, and so the yield is dialectbased, whereas the automatic track works from the
Wiktionary, which is in fact closer to the Modern Stand
Arabic dialect. This combination of dialects within thesame resource raises many issues, and we intend to begin
addressing these during the latter part of the current project,which constitutes the initial development stage of the
EAFN. However, it is likely that more comprehensive
solutions to the issues raised will be solved in later stagesof the EAFN, once we have completed the initial release of
the database.

1) First, for each English Wiktionary lexeme fromthe UBY database, we split the list of English
Wiktionary definitions, and calculate a measure of
the similarity between this lexeme’s UBY
definition and its Wiktionary definition. For thiswork, we used the Gensim word2vec tools,7 andtrained models for this based on the so-called “1Billion Word Language Model Benchmark”.8 We
use this similarity measure as part of an
automatically derived overall confidence score,
which we later use when comparing competing
frame entries in the database.
2) Second, we align the English Wiktionary
definitions with the Arabic Wiktionary
definitions, again calculating a similarity measure
between these definitions (with the same set-up
for Gensim word2vec referred to above), as
another automatically derived component of theabove-mentioned confidence score.

3.

The automatically collected frame annotations of items
from the Arabic Wiktionary, currently consist of lexicalunits (i.e. pairing of lemma and frame), including
confidence measure derived from measuring the strength ofthe match between the English FN and Wiktionary
definitions, on the one hand, and between English andArabic gloss-es/definitions, on the other. Future work will
involve extending this work to include annotations ofFrame Elements.

2.2

Evaluation

Semantic annotation is fraught with issues regarding lackof reliability and accuracy, making quality control of dataa key component of any project in this area. While our
project is still at an early stage of development, we areworking toward an initial release of our data, for which weare developing a comprehensive evaluation regime,
incorporating both the manual and automatic annotation
tracks. A description of this, as well as some early results,
are included in the rest of this section.

3.1

Manual track

3.1.1
Procedure
We are currently piloting several evaluation tasks, targeting
accuracy of judgements about frames and the core elementsof those frame. For these tasks, we first extract a random
sample from the EAC, and annotators then carry out
annotation of this data according to the annotation
protocols we have developed (see Section 2 above). We
then proceed to apply various measures of agreement
between the annotators

Corpus progress

While the initial release of the EAFN is still un-derdevelopment, immediately below we provide a snapshot of
the current data collection, for the initial stages of eachcollection track. In the next section, we present moredetailed evaluations of both the automatic and manualcollection efforts.

We have several measures of the quality of this data,
centering on degrees of overlap in the annotations of twoof the annotators currently involved in the collection efforts
at the UAEU. The statistic we are using here is Cohen’s
kappa coefficient 𝒌:

Currently the EAFN covers verbs only. For manuallygathered entries, we have collected 29 frames, and 360
LUs. As we show later in this section, in initial evaluation
studies, we have found reasonably high inter-annotator
agreement for the manual annotation. We have also
implemented a fully automatic procedure for collecting
entries, for which we have gathered 630 frames and 2100

Pr(𝑎) − Pr(𝑒)1 − Pr(𝑒)Where Pr(𝑎) models the probability of observed agreement
among raters, and Pr(𝑒) captures chance agreement; thehigher the value for 𝒌, the better the agreement between
annotators. There are various interpretations of such scores,
for example, 0.60 is often considered a threshold, with

7

8

𝒌=

https://radimrehurek.com/gensim/

74

http://www.statmt.org/lm-benchmark/scores above this being taken to indicate “substantial
agreement” (Landis & Koch 1977). 𝒌 enables quantifying
the inter-annotator agreement (IAA), particular for
qualitative data, which is closer to our evaluation task,
involving as it does detailed semantic knowledge.9

3.2.2
Results
Table 5 presents the results of this evaluation, withevaluation categories used by both annotators across the
top and down the leftmost column, and inside the tableshowing how scores matched for each item. From this, we
can see that by far the largest number of matches is where
annotators agree that an item is “completely correct”, andthe next highest being where one an-notator thought that an
item was “mostly correct” and the other annotator thought
the same item was “completely correct”.

3.1.2
Results
For comparison of frame annotations on our sample, we
achieve the following: 𝒌 = 0.790 (p-value ≪ .001, 𝑁 =31). For annotation of core FEs, we achieve, 𝒌 = 0.899 (pvalue ≪ .001, 𝑁 = 31). This shows that using the protocol
we have devised, annotators are achieving very good levelsof agreement for judgements about FEs, and acceptableagreement for judgements about frames.

3.2

When ignoring the vowel vs. no-vowel condition, we
achieve the following: 𝒌 = 0.443 (p-value value ≪ .001,
𝑁 = 198). However, when taking into consideration thevowel vs. no-vowel condition, this score improved
somewhat: 𝒌 = 0.602 (p-value value ≪ .001, 𝑁 = 83).

Automatic track

3.2.1
Procedure
Evaluating the automatic annotation provides a key pointof convergence between the two tracks. For this, themanual annotators evaluated the output of the automatic
system, their responses to the automatically generated
frames requiring them to draw on their intuitions, whichhave their foundations in their direct experience building
the manual collection of frames. Feedback from theannotators is crucial to pinpoint where further developmenton the automatic system will be required. In this way, ouraim is that the automatic track more closely approximates
the results from the manual track.

Overall, we can see that general agreement be-tween
annotators is quite low, despite the overall largest matchbeing “completely correct”. This suggests possible
problems and indeed errors for many of the automaticallycollected frames. On the other hand, when we partition thedata set, and extract those items with vowel information,
for this subset, the IAA improves considerably, suggestingthat such information is an important component toincorporate in future automatically acquired collections for
the EAFN.
1
2
3
4
5

The procedure we followed here involved manual
annotators going through individual, automaticallygenerated LUs, complete with brief information about thetarget LU, as well as the frame assigned to this LU. Eachannotator was given a total of 198 randomly sampledlexical units to evaluate. Annotators rated this on thefollowing 5-point scale: 1 = Completely correct, 2 = Mostly
correct, 3 = Acceptable, 4 = Mostly incorrect, 5 =Completely incorrect. The sample was further split
according to two conditions: either (1) the rendering of thelexical unit in Arabic script included vowel information, or(2) it did not. For Arabic script, information about vowels
can help disambiguate LUs, and potentially influence theratings assigned for any specific LU. We are interested ininvestigating such aspects of the automatic collection
process more closely.

1
24
2
1
1
2

2
15
7
1
3
2

3
4
0
0
1
0

4
1
2
2
3
1

5
1
0
0
1
9

Table 5: Evaluation of automatic track (1 = Completely
correct, 2 = Mostly correct, 3 = Acceptable, 4 = Mostly
incorrect, 5 = Completely incorrect)

4.

Conclusion

We have presented early results for the first iteration of the
Emirati Arabic FrameNet (EAFN). The EAFN is a generalsemantic resource for the Arabic language, which is sure to
be of interest to a range of researchers, from those inlinguistics, to others working within natural language
processing. The project is divided into manual andautomatic tracks, based on the predominant techniques
being used to collect frames in each track. Despite a hiatus,
work on the EAFN has recommenced; we have herereported on initial results for annotations and evaluation ofthese annotations which have been carried out in bothtracks. The EAFN is well on target for the first release ofdata in the coming year.

The key statistic we are using here is Cohen’s kappa
coefficient, the same statistic used for measuringagreement during evaluation of the manual annotation task.
The difference for the task of evaluating the automatic
annotation, is that this task results in ordered data (a likert
scale), and so we need to use weighted kappa coefficients;
specifically, we are using squared weights, whereby
disagreements are weighted according to their squared
distance from perfect agreement.

5.

Acknowledgements

We would like to thank S. Al Helou and W. Ghaban forhelp with the evaluation. We would also like to thank M.

9For all of this, we have used the irr package in R, which
has been specifically designed for modelling “interrater
reliability and agreement.”75

Elkaref for some early technical help with the JWKTLlibrary, as well as valuable discussions about carrying out
computational linguistic modelling of Arabic. Finally, we
would also like to thank J. Ruppenhofer for discussions
about the English FrameNet.

6.

Processing: Volume 1-Volume 1 (pp. 276-285).
Association for Computational Linguistics

Bibliographical References

Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998). The
Berkeley Framenet project. In Proceedings of the 17th
international conference on Computational linguisticsVolume 1 (pp. 86-90). Association for Computational
Linguistics
Boas, H. C. (Ed.). (2009). Multilingual FrameNets incomputational lexicography: methods and ap-plications(Vol. 200). Walter de Gruyter.
Czulo, O., Torrent, T. T., da Silva Matos, E. E., da Costa,
A. D., & Kar, D. (2019). Designing a Frame-Semantic
Machine Translation Evaluation Metric. In 2nd
Workshop on Human-Informed Translation andInterpreting Technology (HiT-IT 2019) (pp. 28-35).
De Cao, D., Croce, D., Pennacchiotti, M., & Basili, R.
(2008). Combining word sense and usage for modelingframe semantics. In Proceedings of the 2008 Conference
on Semantics in Text Processing (pp. 85-101).
Association for Computational Linguistics.
ElJundi, O., Antoun, W., El Droubi, N., Hajj, H., El-Hajj,
W., & Shaban, K. (2019). hULMonA: The Universal
Language Model in Arabic. In Proceedings of the Fourth
Arabic Natural Language Processing Workshop (pp. 6877).
Fillmore, C. (1982). Frame semantics. Linguistics in themorning calm, (pp. 111-137). Seoul, South Korea:
Hanshin Publishing Co.
Fillmore, C. J., & Atkins, B. S. (1998). FrameNet andlexicographic relevance. In Proceedings of the FirstInternational Conference on Language Resources and
Evaluation, Granada, Spain (pp. 28-30).
Fillmore, C.J., C.R. Johnson, and M.R. L. Petruck. (2003).
"Background to FrameNet." International Journal of
Lexicography. 16.3: 235-250.
Gurevych, I., Eckle-Kohler, J., Hartmann, S., Matuschek,
M., Meyer, C. M., & Wirth, C. (2012). Uby: A largescale unified lexical-semantic resource based on LMF. In
Proceedings of the 13th Conference of the European
Chapter of the Association for Computational
Linguistics(pp.
580-590).
Associationfor
Computational Linguistics.
Halefom, G, T. Leung and D. Ntelitheos. (2013). A Corpus
of Emirati Arabic. NRF Grant 31 H001, United ArabEmirates University.
Landis, J.R. and G.G. Koch (1977). The measurement ofobserver agreement for categorical data. Biometrics.
Vol. 33, pp. 159—174.
Ohara, K. (2012). Semantic Annotations in Japanese
FrameNet: Comparing Frames in Japanese and English.
In LREC (pp. 1559-1562).
Rehbein, I., Ruppenhofer, J., Sporleder, C., & Pinkal, M.
(2012). Adding nominal spice to SALSA-framesemantic annotation of German nouns and verbs. In
Proceedings of the 11th Conference on Natural
Language Processing (KONVENS’12) (pp. 89-97).
Tonelli, S., & Giuliano, C. (2009). Wikipedia as frame
information repository. In Proceedings of the 2009
Conference on Empirical Methods in Natural Language76Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 77–84
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Exploring Crosslinguisitic FrameNet Alignment
Collin F. Baker, Arthur Lorenzi

International Computer Science Institute1947 Center St. Suite 600
Berkeley, CA, 94704, U.S.A.
{collinb,lorenzi}@icsi.berkeley.edu
Abstract
The FrameNet (FN) project at the International Computer Science Institute in Berkeley (ICSI), which documents the core vocabularyof contemporary English, was the first lexical resource based on Fillmore’s theory of Frame Semantics. Berkeley FrameNet has
inspired related projects in roughly a dozen other languages, which have evolved somewhat independently; the current Multilingual
FrameNet project (MLFN) is an attempt to find alignments between all of them. The alignment problem is complicated by thefact that these projects have adhered to the Berkeley FrameNet model to varying degrees, and they were also founded at different
times, when different versions of the Berkeley FrameNet data were available. We describe several new methods for finding relationsof similarity between semantic frames across languages. We will demonstrate ViToXF, a new tool which provides interactive
visualizations of these cross-lingual relations, between frames, lexical units, and frame elements, based on resources such as multilingual dictionaries and on shared distributional vector spaces, making clear the strengths and weaknesses of different alignment methods .
Keywords: frame semantics, lexical semantics, semantic frames, multilingual lexical resources

1. Introduction
1.1.

The FrameNet lexical database, in XML format, has been
downloaded more than 3,000 times by researchers anddevelopers around the world; the well-known NLP library
NLTK (Loper and Bird, 2002) also provides API access to
FrameNet (Schneider and Wooters, 2017). FrameNet’s main
publications have been cited over 2,500 times according toGoogle Scholar, and it has been an important basis for atleast 14 PhD dissertations.

Frame Semantics and FrameNet

NLP researchers have long sought to develop tools andresources to build meaning representations beyond theword or syntax level, and many have looked to Charles J.
Fillmore’s theory of Frame Semantics (Fillmore, 1977b;
Fillmore, 1977a) as part of the solution. Fillmore and hiscolleagues founded the FrameNet (FN) Project (Fillmore
and Baker, 2010; Fontenelle, 2003) at the International
Computer Science Institute (ICSI) in 1997 with the goal ofestablishing a general-purpose resource for frame semantic
descriptions of English language text. FrameNet’s lexicon
is organized not around words, but semantic frames(Fillmore, 1976), which are characterizations of events,
static relations, states, and entities. Each frame provides
the conceptual basis for understanding a set of word senses,
called lexical units (LUs), that evoke the frame in the mindof the hearer; LUs can be any part of speech, although mostare nouns, verbs, or adjectives. FrameNet now contains
roughly 1,200 frames and 13,600 LUs.

The wide use of FrameNet in NLP depends on the successof systems for automatic semantic role labeling (ASRL) of
unseen text, trained on the FrameNet annotation data. ASRL
then enables (or improves) downstream NLP applications,such as• Question Answering (Shen and Lapata, 2007; Sinha,
2008)• Information Extraction (Surdeanu et al., 2003)• Text-to-scene generation (Coyne et al., 2012)• Dialog systems (Chen et al., 2013)• Social network extraction (Agarwal et al., 2014)• Knowledge Extraction from Twitter (Søgaard et al.,2015)
In fact, automatic semantic role labeling has become oneof the standard tasks in NLP, and several freely available
FrameNet-based ASRL systems have been developed, including SEMAFOR (Das et al., 2010; Das et al., 2014) andopen-sesame (Swayamdipta et al., 2018). The latter jointly
exploits PropBank-based (Palmer et al., 2005) semantic rolelabeling and FrameNet to train a neural net (NN) to do frameand FE discrimination without run-time parsing. Other recent FrameNet-based ASRL systems have tried a variety of
new approaches:• FitzGerald et al. (2015) train a NN representing joint
embeddings of PropBank and FrameNet roles,• Kshirsagar et al. (2015) use structured features and theframe hierarchy,• Roth and Lapata (2015) predict roles based both on the

FrameNet provides very detailed information about thesyntactic-semantic patterns that are possible for each LU,
derived from expert annotations on naturally occurring
sentences. These annotations label the phrases that
instantiate the set of roles involved in the frame, known
as frame elements (FEs). An example of a simple frame
is Placing, which represents the notion of someone (orsomething) placing something in a location; this frame isevoked by LUs like place.v, put.v, lay.v, implant.v, andbillet.v and also bag.v, bottle.v, and box.v, The core frame
elements of Placing are the AGENT who does the placing(or the C AUSE of the placing), the T HEME that is placed,
and the G OAL. An example of a more complex frame is
Revenge, which has FEs AVENGER, I NJURED PARTY,I NJURY, O FFENDER, and P UNISHMENT.

77

entire document and on recent role assignments, and• Peng et al. (2018) jointly model FrameNet roles anddependency parses.

1.2.

However, the projects have adhered to the Berkeley
FrameNet (BFN) model to different degrees: The Spanish,
Japanese, and Brazilian Portuguese FNs have followed BFN
rather closely, using BFN frames as templates, whereas
the SALSA Project (for German), French FN, Swedish
FrameNet++ and Chinese FN have diverged further from
BFN, either adding many new frames and/or modifying the
BFN-derived ones.3

Multilingual FrameNet

Since the beginning of Frame Semantics, the question has
arisen as to whether semantic frames represent “universals”
of human language or are language specific. Despite many
language-specific patterns of expression, the conclusionfrom the FrameNet experience has been that many frames
are applicable across different languages, especially thosefor basic human experiences, like eating, drinking, andsleeping. Even some cultural practices are similar acrosslanguages: e.g. in every culture, commercial transactions
involve the roles BUYER , SELLER , MONEY, and GOODS (orservices).

More fundamentally, there is no reason to assume thatcross-linguistic frame relations will be limited to equivalence. Frames in other languages can be broader or narrower than the nearest English frame, or similar situations
may require a different point of view in different languages.
For example, English I like X, where like.v is in the Experiencer focused emotion frame (along with adore, dread
and regret), is regularly translated as Spanish Me gusta X,‘X pleases me’ where gustar.v is in the Experiencer object
frame (along with asombrar ‘astound’, chocar ‘shock’,
and molestar ‘bother’, cf. Subirats-Rüggeberg and Petruck(2003)). Other well-attested differences in information structure between languages are similarly reflected in differencesin choice of frames, such as that between satellite-framed
languages like English and German and verb-framed languages like Spanish and Japanese (Slobin, 1996). There will
also be cultural differences, which may mean that equivalent frames do not exist, such as frames for religions orlegal processes, which differ widely from country to country.4 The Multilingual FrameNet project (Gilardi and Baker,2018) is studying the relations between frames in different
languages and will distribute a database of alignments between FrameNets. We have developed several approaches
to calculating frame similarity to produce the cross-lingualalignments; these are described in Sections 2.1. through 2.4..
In order to compare these approaches and to evaluate theirstrengths and weaknesses under various settings of parameters, we have also built an interactive tool for visualizationof frame alignments, called ViToXF (for “visualization toolacross FrameNet”). We describe this tool in Sec. 3., andwill demonstrate it at the workshop. Finally, Sec. 4. offerssome qualitative evaluations of the alignment methods anddiscusses directions for future research.

Since the Berkeley FrameNet (hereafter BFN) project
began releasing its data, researchers in many countries haveexpressed interest in creating comparable resources forother languages; in fact, the BFN team is in contact with
about a dozen FrameNets in languages other than English
1. The methods used in building these FrameNets havediffered, and each has created frames based on their own linguistic data, but all at least have an eye to how their frames
compare with those created for English at ICSI (Boas, 2009).
Given that so much research has been conducted in building
separate lexical databases for many languages, it is natural
to ask whether these lexical databases can be aligned to form
a multilingual FrameNet lexical database connecting all of
the languages (as well as new FrameNets that may arisein the future), while also accounting for language-specificdifferences and domain-specific extensions to FrameNet.
The results produced so far suggest that this is possible. It isalso urgent to carry out this harmonization process as soon
as possible, to take better advantage of the experience ofeach language project, to avoid duplication of effort, and tounify the representational format as much as possible. A
number of FrameNet groups, led by FrameNet Brasil havealso established the Global FrameNet2 project to improve
communication between FrameNets.

2.

Despite differences among the FrameNet projects, all agree
on the concept of semantic frames as their organizing
principle and all have found the set of frames defined in
BFN to be generally applicable to their language. For
example, all languages have ways to express directed
self-motion, which involves the frame elements M OVER,S OURCE, PATH and G OAL (although it is rare for all of these
to be expressed in the same clause). Likewise, whenevera communicative act occurs, we can identify the FEs
S PEAKER, A DDRESSEE, and T OPIC or M ESSAGE, which
are common to all the communication frames. Semantic
frames thus should provide useful generalizations both overlexical units within a language and across languages.

Cross-lingual alignment and Visualization
Techniques

Table 1 gives counts for frames and LUs for the six languages included in the preliminary version of the visualization tool; in some cases, these numbers may understate thetotal in each project, due to certain difficulties in importing
the data.

2.1.

Alignment by frame name/ID

At first glance, the alignment problem seems trivial: if theother FrameNets have largely used BFN frames, one might3
At this time, the MLFN effort is not trying to align the Italian,
Arabic, or Hebrew data, for various reasons, including availability
and coverage, among others.
4
Of course, languages frequently develop, borrow, or calc terms
and frames for concepts that are not “native” to the language, inorder to discuss other cultures.

1

https://framenet.icsi.berkeley.edu/
fndrupal/framenets_in_other_languages
2
https://www.globalframenet.org

78

Figure 1: Visualization tool, showing user controls and resulting Sankey diagram of English Judgment frames aligned with
Spanish Frames using LU translation by linking synsets at a high score threshold (see Sec. 2.2. for details).
Project
FrameNet (ICSI) = BFN
Chinese FN
FN Brasil (PT)French FN (Asfalda)
German FNJapanese FN
Spanish FN
Swedish FN

Frames
1,224
1,259
1,092
148
1,023
984
1,196
1,186

Lexical Units
13,67520,551
2,896
2,590
1,826
3392
11,352
38,749translations; however, exactly how to use information fromframes and frame relations in the translation process is still
to be determined.
The Open Multilingual WordNet (OMWN) (Bond and Foster, 2013) contains multilingual synsets, combining lemmas
from WordNets for dozens of languages, data from Wiktionary, and the Common Locale Data Repository.5 We
are currently using it to find a set of translation equivalents
between languages. The first step is to create a mappingS(`) from each LU in each language to a set of OMWN
synsets that represent its senses. That mapping is createdby searching OMWN for synsets that contain the lemma(with the correct part of speech) of the LU. More formally,
let e be a frame in the source language and f a frame in thetarget language; let Le and Lf be the lists of the LUs inframes e and f respectively. Then any two LUs a and b in
Le and Lf (respectively) match if they occur together in atleast one synset; this matching function can be expressed by
Equation 1.

Table 1: Frame and LU counts of FrameNets now in ViToXF
just assume that a frame in another language with the same
name as a BFN frame represents the same concept, andignore any that don’t have matching names. However, as
might be expected, some of the other languages have usedframe names in the target language, rather than English; thiswould mean aligning the frame names themselves acrosslanguages. In some cases, their frame data also includes afield for the BFN name or BFN ID, which can be used foralignment, even when the frame names are not in English.
Furthermore, even when the names (or IDs) match, thenon-English frame may be defined differently or have moreor fewer core frame elements than the BFN frame, which,strictly speaking, makes it a different frame.

2.2.

m(Le , Lf ) = {a ∈ Le | b ∈ Lf : S(a) ∩ S(b) 6= ∅} (1)When evaluating the alignment between two frames, thisfunction was used to calculate three different scores. The
first is a metric that takes into consideration LUs from both
frames (Equation 2); however, this gives frames containing
more LUs more influence over the result. To avoid thisproblem, we decided to break the alignment into two otherscores taking into account the direction of alignment, i.e., thescore of the alignment from English to the target language
can be different from the reverse. The basic formula forthose scores is presented in Equation 3. (Note that the twoscores can be obtained by simply swapping the arguments

Alignment by LU translation

A second way of approaching alignment is to take all
the lexical units from a source language frame and findtranslation equivalents in the target language. To the extent
that frames are equivalent across languages, we would
expect all the translations of LUs in one source language
frame to fall into one target language frame. Of course, thisdepends on the accuracy of the translations. By definition, alexical unit in a frame represents one sense of a lemma, soin theory that should greatly narrow the range of possible

579

http://cldr.unicode.org/

in the equation.)6
s1 (Le , Lf ) =|m(Le , Lf )| + |m(Lf , Le )||Le | + |Lf |(2)|m(Le , Lf )|
|Le |

(3)

s2 (Le , Lf ) =two different scoring techniques. The first,“LU translations
using MUSE”, like those discussed above based on OMWN,
uses the word embeddings as a way to obtain translationequivalents: we define the neighborhood around the vector
embedding of a target language word as n(~v , k, t), that is,
the k-neighborhood of ~v in the target language with a cosine
similarity greater than t. Then we define the alignment score
between a pair of frames given their LU lists Le and Lf by
Equation 5.

We also explored another alternative scoring method that isavailable in the visualization tool by selecting the ”Synset
count” scoring method. This score is calculated using Equation 3.
s3 (Le , Lf ) =

2.3.

S

a∈Le S(a) ∩SSb∈Lf S(b)a∈Le S(a)s4 (Le , Lf ) =|{a ∈ Le | b ∈ Lf : ~v (b) ∈ n(~v (a), k, t)}|
|Le |

(4)

The second scoring technique, ”LU centroid similarity using
MUSE”, calculates the alignment between two frames byfinding the average of the vectors of their LUs (i.e. thecentroid vector of each frame) and computing the cosine
similarity of those two centroids, similar to the approach of
Sikos and Padó (2018).

Alignment by frame element similarity

By definition, for two frames to be the same across languages, they must have the same number and type of frame
elements (FEs). Some FrameNets (such as Spanish FN andJapanese FN) have simply copied the FEs from Berkeley
FrameNet, so that their names and definitions are still identical to BFN. Others, such as Chinese FN, have translated
or created both the names and the definitions in the target
language; in those cases, we need to align the FEs by using
the proximity of the names and definitions from the two
languages in a shared vector space. French FN created FE
names and definitions in English, even though many of theirframes do not correspond to BFN. Swedish has FE names inEnglish, but no definitions; since they state that the framesand FEs with English names are intended to be identicalto the BFN frames and FEs of the same name, the English
definitions should also apply to them. Finally, both Brazilian
Portuguese and German (SALSA) have FEs in a mixtureof English and the target language. In those two cases, wegroup the FEs according to whether they are in English or
the target language,7 calculate the similarity separately forthe two groups, and then combine the scores.

2.4.

(5)

3.
3.1.

Alignment Visualization Tool

Frame Alignment example

We will demonstrate the alignment of three related
English frames with Spanish, Judgment, Judgment communication, and Judgment direct address.
The Judgment frame applies whenever a person (theC OGNIZER) forms an opinion (good or bad) about
someone or something (the E VALUEE). In the Judgment communication frame, the C OGNIZER, now called
the C OMMUNICATOR expresses that opinion, possibly to an
A DDRESSEE. In the frame Judgment direct address, theA DDRESSEE is also the one being evaluated, so this frame
contains LUs like congratulate,harangue, scold, take to task
and tell off. The relations between these frames and theirframe elements are spelled out in detail in FrameNet; theJudgment communication frame uses two frames, Judgment frame and Statement, and Judgment direct address
inherits from Judgment communication.

Alignment by distributional similarity of
LUs

3.2.

Another approach to alignment is to use cross-lingual wordembeddings to obtain translations equivalents. The current
iteration of the visualization tool uses the FastText wordembeddings from FaceBook Research, which were trainedon Wikipedia data from various languages and aligned toa single embedding space (Bojanowski et al., 2017). The
spaces were aligned by an unsupervised method that uses anadversarial approach, where the discriminator tries to predict
the embedding origin and the generator aims to create transformations that the former is not able to accurately classify(Conneau et al., 2017). The transformed FastText vectorsof many languages mapped to English space were madepublicly available in the MUSE library.8 We are currently
using these pre-trained cross-lingual word embeddings for

Visualization modes

In its current iteration, the system has two visualization
modes, one that uses a Sankey diagram to show alignments
between frames and another that displays the translations
between the LUs of a frame pair in a different type of graph.
Frame Alignment Visualization: Fig. 1 shows the mainvisualization mode of the tool. It is an interactive bipartite
Sankey diagram where English frames are displayed on theleft side and target language frames on the right. The widthof each band in the diagram is proportional to the alignmentscore between the frame pair.
Due to the number of lexical units in the FrameNet projects,
the resulting diagram can be very dense, making analysis
difficult. To alleviate this problem, ViToXF allows both
frame selection and band filtering. Frames to be shown canbe selected from a list of all the English and target language
frames, (marked with a two-letter suffix, en for English,
es for Spanish and so forth). By default, the system will
display any match that includes one of the selected frames.

6

Both scores can also be visualized with the ”LU translations
using WordNet” options, cf.Sec. 2.2.
7
Using Michal Danilak’s python library for language recognition https://pypi.org/project/langdetect/8
https://github.com/facebookresearch/MUSE

80

target language LU both match a lemma+POS in a synset,
the name of that synset (or LU depending on the scoring
method) is shown in green, and the overall matching scoreis raised. If an LU from one language matches the synset
but not from the other language, the synset name or LU isyellow; this adds to the denominator of Equations 2, 3, 4
and 5 reducing the overall matching score. Synsets whichmatch no LU in the source language are colored black; they
do not influence the score.
When the vector embedding method is being used, the left
and right columns are as described above, but the center
column now represents wordforms; edges are connected
to LUs in either language whose lemma lies within theneighborhood of the wordform in the embedding space. If
the FastText vectors are used, this means that subparts ofwords play a role, and that may help connect the various
wordforms of a lemma, but may also lead to false positives.
Part of speech is not used. The meanings of the colors in
the central column are as described above.

Figure 2: Frame Alignment: English ⇒ Spanish for
Judgment-related Frames, matched using synsets at a lower
score threshold
This can be further restricted by checking ”Show ONLY
selected frames”; in that case, it displays only connections
where both frames are selected.

Continuing with our example of aligning from English to
Spanish in the Judgment-related frames using LU translation
via synsets, Fig. 3 shows how the LUs in each language
link to the lemmas in the synsets of OMWN. Fig. 3 also
shows what can go wrong: the lemma charge.v appears in
the BFN Judgment communication frame, but in OMWN
it also appears in a synset with English load.v and Spanish
cargar.v, defined as ’provide (a device) with somethingnecessary’. Thus it links erroneously to the Filling frame inboth languages; this problem is discussed further in the next
section.

To filter which bands are displayed in the diagram, the user
can also set an alignment score threshold (so that weakeralignments will not be shown) and/or set a limit on thenumber of connections from each frame. When the numberof connections is restricted, those with the highest scores
will always be displayed first.
When the “LU-based using MUSE vectors” is selected as
the scoring technique for LU matching between frames, theparameters k (neighborhood size) and t (distance threshold)of the function n(~v , k, t) described in Subsection 2.4., Eq. 5
can be modified, potentially changing alignment scores andhence, the graph displayed. Fig. 1 shows the sidebar, where
all of these parameters can be controlled by the user.

4.

Discussion

Each new FrameNet constitutes an experiment in crosslinguistic Frame Semantics. Motivated by the fundamental research question ”To what extent are semantic framessimilar across languages?”, ViToXF provides an intuitive,
graphical, interactive tool to study a variety of methods forfinding the relations between frames and lexical units acrossdifferent languages. It also highlights some of the problems that need to be solved to create meaningful alignmentsthat are useful for a wide range of NLP tasks. We do not
yet have not results from testing these alignments against
standard NLP tasks, but this section offers some qualitativeevaluations of the methods and results so far.

Fig. 2 shows the same English-Spanish alignment of frames
related to judgment, with a slightly lower similarity threshold than in Fig. 1. Note that alignments to two additionalframes Placing and Filling have now appeared; this will be
explained in the following section.
Lexical Unit Translation Visualization: This visualization mode is intended to demonstrate exactly howtranslations were found for the LUs of a frame pair, and canbe accessed by clicking on any band in the Sankey diagram.
ViToXF provides two methods for aligning frames and LUs
across languages, one based on synsets, the other on vector
embeddings; depending on which method is used for the
Sankey diagram, the LU translation visualization will besomewhat different.

4.1.

Evaluating Synset-based methods

The alignment methods that depend on WordNet synsets
have the merit that they take advantage of large-scale curated
groupings of lemmas (by part of speech). However, theyalso make clear one problem with WordNets: for many
common words, the number of senses given is simply too
high. We noted in the preceding section some problems withthe polysemy of charge.v; in fact, Princeton WordNet lists
31 senses for the verb alone! Some of the major divisions
are clear:• charge#1, bear down#3 (to make a rush at or sudden
attack upon, as in battle) He saw Jess charging at him
with a pitchfork

In both cases, the translation visualization is a tripartite
graph with vertices organized in three columns: the left
column is composed of the LUs of the BFN frame, the rightcolumn of the LUs of the target language frames. In thecase of the synset-based LU translation method, the middle
column lists the names of synsets, and edges are drawn
between the synsets and the LUs in each language whoselemma+POS occurs in that synset. If an English LU and a

81

Figure 3: Lexical unit translation: English ⇒ Spanish for Judgment-related Frames, matched using synsets at a lower scorethreshold. In addition to the expected alignment to Judgment communication, charge.v is also mapped to a synset in the
Filling frame.
• charge#3, bill#1 (demand payment)Will I get charged
for this service?
• appoint#2, charge#5 (assign a duty, responsibility or
obligation to) She was charged with supervising thecreation of a concordance• charge#6, lodge#3, file#4 (file a formal charge against)
The suspect was charged with murdering his wife
• charge#24 (energize a battery by passing a current
through it in the direction opposite to discharge) I needto charge my car battery
But some senses are hard for humans to distinguish, let alone
algorithms; compare for example:• charge#2, accuse#2 (blame for, make a claim of wrongdoing or misbehavior against) He charged the director
with indifference and• charge#7 (make an accusatory claim) The defense attorney charged that the jurors were biased.
Are these separate from each other? How are they related to#6?
• charge#8 (fill or load to capacity) charge the wagon
with hay and• load#2, charge#16 (provide (a device) with somethingnecessary) He loaded his gun carefully.
Are these the same as #24? Is #24 just a special case of
charge#16?
The Ontonotes lexical resource (Pradhan et al., 2013), which
is based on combining WordNet senses so that annotators
can reliably distinguish the classes, may provide a coarser
but more reliable list of senses for English and Chinese, but
it does not include the other FrameNet language pairs.

4.2.

(e.g. most lexicographers would expect the verb go to
be represented by a single vector that covers go, went,gone, goes, and going, rather than separate vectors for thefive word forms). These embeddings also do not include
Chinese and Japanese.
The major shortcoming of the current distributional embeddings, however, is that they provide only one vector per wordform, with no distinction of senses. However, there havebeen encouraging results on finding embeddings for word
senses, such as Upadhyay et al. (2016) who use multilingual
corpora to learn sense-specific embeddings. They point out
that often patterns of polysemy are similar across languages;
continuing with the preceding example, English charge andSpanish cargar can both mean either ’file charges in court’
or ’fill a battery with electricity’. However, adding an unrelated language such as Chinese often gives completely
different translations:

4.3.

Applications and Future WorkA major limitation of ViTOXF is simply that most
FrameNets are rather small in comparison with otherlexical resources, primarily because of the amount of
human curation needed to produce them. However,interest in Frame Semantics continues to grow and new
FrameNet projects are appearing frequently, so there may
be a continuing interest in finding alignments for them.
There are also numerous approaches to automatically or
semi-automatically adding lexical units to FrameNets (e.g.
Pavlick et al. (2015), Fossati et al. (2013),Hartmann andGurevych (2013),Green (2004)), offering the prospect ofmuch larger, if less precise, lexical inventories.

Evaluating Vector-based methods

The alignment methods based on vector embeddings havethe advantage of making it possible to measure distances
between uses, distances which are arguably semantic;
however these distances are not easily converted to “senses”
that humans can understand. Also, the MUSE embeddings,
like most distributional embeddings, are based on wordforms, and do not generalize to the level of lexemes

We expect that alignments produced by the methods outlined
here and refined by the use of ViTOXF will prove useful to• translators and second language learners seeking tounderstand cross-linguistic differences in framing;• developers of MT systems, parsers, and grammars (especially for languages for which FrameNets already82

exist) (e.g. Czulo et al. (2019)); and, of course,• cognitive linguists and researchers creating new
FrameNets.
Since FastText does not provide cross-linguistic embeddings
for English-Japanese and English-Chinese, we will attempt
to train some ourselves, to make that type of alignment
available for them. We may be able to find ways to use theannotated sentences themselves to align frames, possiblyusing methods related to BERT vector embeddings, such as
those of Zhang et al. (2020).
As just mentioned, instances of similar polysemy canusually be split apart by looking simultaneously at morelanguages, especially if the languages are unrelated.
We therefore plan to look for frames which align wellacross three or more languages, making for highly robustalignments. Our immediate goal is to incorporate as manyof the current FrameNet projects as possible.
We also plan to explore methods for creating sense-specificvectors in all the languages, and better techniques for finding translation equivalents; for example, a smaller numberof translations from an MT system may prove more accurate than those from OMWN synsets. Finally, it should
be clear that there are many ways to combine the similarity scores from the different methods to get an overall
score between two frames. We plan to test the advantages and disadvantages of various weighted linear combinations of scores for different applications. The current code for the visualizer is essentially an alpha version; we welcome suggestions for improving the userinterface. We will make the code for ViToXF available on Github; a demo version of the visualizer isavailable now at https://icsi-berkeley.github.
io/framenet-multilingual-alignment/.

Acknowledgments
This material is based in part upon work supported by the
U.S. National Science Foundation under Grant No. 1629989.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors anddo not necessarily reflect the views of the National Science
Foundation. We also received support in the form of accessto virtual machines from the NSF-funded XSEDE programand machine time and systems support from Ely Matos andTiago Torrent of FrameNet Brasil, which supported a separate experiment in cross-lingual annotation, reported in
Gilardi and Baker (2018). Finally, of course, we are especially grateful to all the FrameNets mentioned above forproviding their data for this study.

Bibliographical References
Agarwal, A., Balasubramanian, S., Kotalwar, A., Zheng, J.,
and Rambow, O. (2014). Frame semantic tree kernels
for social network extraction from text. In EACL, pages
211–219.
Hans C. Boas, editor. (2009). Multilingual FrameNets in
Computational Lexicography: Methods and Applications.
Mouton de Gruyter.

83

Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T.
(2017). Enriching word vectors with subword information. Transactions of the Association for Computational
Linguistics, 5:135–146.
Bond, F. and Foster, R. (2013). Linking and extending anOpen Multilingual Wordnet. In Proceedings of the 51st
Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1352–1362,
Sofia, Bulgaria, August. Association for Computational
Linguistics.
Chen, Y.-N., Wang, W. Y., and Rudnicky, A. I. (2013). Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing. In
Automatic Speech Recognition and Understanding Workshop, pages 120–125. IEEE.
Conneau, A., Lample, G., Ranzato, M., Denoyer, L., andJégou, H. (2017). Word translation without parallel data.
arXiv preprint arXiv:1710.04087.
Coyne, B., Klapheke, A., Rouhizadeh, M., Sproat, R., andBauer, D. (2012). Annotation Tools and Knowledge Representation for a Text-To-Scene System. In COLING
2012, 24th International Conference on Computational
Linguistics, Proceedings of the Conference: TechnicalPapers, pages 679–694.
Czulo, O., Torrent, T., Matos, E., Costa, A. D. d., and Kar, D.
(2019). Designing a frame-semantic machine translation
evaluation metric. In Proceedings of the Human-Informed
Translation and Interpreting Technology Workshop (HiTIT 2019), pages 28–35, Varna, Bulgaria, September. Incoma Ltd., Shoumen, Bulgaria.
Das, D., Schneider, N., Chen, D., and Smith, N. A. (2010).
Probabilistic Frame-Semantic Parsing. In Proceedings of
the North American Chapter of the Association for Computational Linguistics Human Language Technologies
Conference, Los Angeles, June.
Das, D., Chen, D., Martins, A. F. T., Schneider, N., andSmith, N. A. (2014). Frame-Semantic Parsing. Computational Linguistics, 40(1).
Fillmore, C. J. and Baker, C. F. (2010). A Frames Approach
to Semantic Analysis. In Bernd Heine et al., editors, Oxford Handbook of Linguistic Analysis, pages 313–341.
OUP. This is Chapter 13 in 1st edition, 33 in 2nd edition.
Fillmore, C. J. (1976). Frame semantics and the nature oflanguage. Annals of the New York Academy of Sciences:
Conference on the Origin and Development of Language
and Speech, 280(1):20–32.
Fillmore, C. J. (1977a). Scenes-and-frames semantics. In
Antonio Zampolli, editor, Linguistic Structures Processing, number 59 in Fundamental Studies in Computer Science. North Holland Publishing.
Fillmore, C. J. (1977b). The need for a frame semantics inlinguistics. In Hans Karlgren, editor, Statistical Methods
in Linguistics. Scriptor.
FitzGerald, N., Täckström, O., Ganchev, K., and Das, D.
(2015). Semantic role labeling with neural network factors. In EMNLP, pages 960–970.
Thierry Fontenelle, editor. (2003). International Journalof Lexicography–Special Issue on FrameNet, volume 16.
Oxford University Press.

Fossati, M., Giuliano, C., and Tonelli, S. (2013). Outsourcing FrameNet to the Crowd. In Proceedings of the 51st
Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), pages 742–747,
Sofia, Bulgaria, August. Association for Computational
Linguistics.
Gilardi, L. and Baker, C. F. (2018). Learning to alignacross languages: Toward Multilingual FrameNet. In
Tiago Timponi Torrent, et al., editors, Proceedings of theInternational FrameNet Workshop 2018: Multilingual
FrameNets and Constructicons, pages 13–22. LREC.
Green, R. (2004). Inducing Semantic Frames from Lexical
Resources. Ph.D. thesis, University of Maryland, College
Park.
Hartmann, S. and Gurevych, I. (2013). FrameNet on the
Way to Babel: Creating a Bilingual FrameNet Using Wiktionary as Interlingual Connection. In Proceedings of the
51st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1363–1373,
Sofia, Bulgaria, August. Association for Computational
Linguistics.
Kshirsagar, M., Thomson, S., Schneider, N., Carbonell, J.,
Smith, N. A., and Dyer, C. (2015). Frame-Semantic Role
Labeling with Heterogeneous Annotations. In Proceedings of the 53rd Annual Meeting of the Association for
Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume
2: Short Papers), pages 218–224, Beijing, China, July.
Association for Computational Linguistics.
Loper, E. and Bird, S. (2002). NLTK: The natural language toolkit. In In Proceedings of the ACL Workshop
on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics.
Philadelphia: Association for Computational Linguistics.
Palmer, M., Gildea, D., and Kingsbury, P. (2005). The
proposition bank: An annotated corpus of semantic roles.
Computational Linguistics, 31(1):71–106.
Pavlick, E., Wolfe, T., Rastogi, P., Callison-Burch, C.,
Dredze, M., and Van Durme, B. (2015). FrameNet+: Fast
Paraphrastic Tripling of FrameNet. In Proceedings of the
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short
Papers), pages 408–413, Beijing, China, July. Association
for Computational Linguistics.
Peng, H., Thomson, S., Swayamdipta, S., and Smith, N. A.
(2018). Learning joint semantic parsers from disjoint
data. In Proceedings of the 2018 Conference of the North
American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1(Long Papers), pages 1492–1502. Association for Computational Linguistics.
Pradhan, S., Moschitti, A., Xue, N., Ng, H. T., Björkelund,A., Uryupina, O., Zhang, Y., and Zhong, Z. (2013). Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational
Natural Language Learning, pages 143–152, Sofia, Bulgaria, August. Association for Computational Linguistics.
Roth, M. and Lapata, M. (2015). Context-aware Frame-

Semantic Role Labeling. Transactions of the Association
for Computational Linguistics, 3:449–460.
Schneider, N. and Wooters, C. (2017). The NLTK
FrameNet API: Designing for discoverability with a richlinguistic resource. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 1–6, Copenhagen,Denmark, September. Association for Computational Linguistics.
Shen, D. and Lapata, M. (2007). Using Semantic Roles to
Improve Question Answering. In Proceedings of the 2007
Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language
Learning (EMNLP-CoNLL), pages 12–21.
Sikos, J. and Padó, S. (2018). Using embeddings to compare framenet frames across languages. In Proceedingsof the First Workshop on Linguistic Resources for Natural Language Processing, pages 91–101. Association
for Computational Linguistics. Collocated with COLING
2018.
Sinha, S. (2008). Answering Questions about Complex
Events. Ph.D. thesis, EECS Department, University ofCalifornia, Berkeley, Dec.
Slobin, D. I. (1996). Two ways to travel: Verbs of motion
in English and Spanish. In M. S. Shibatani et al., editors,
Grammatical constructions: Their form and meaning,
pages 195–220. Clarendon Press, Oxford.
Subirats-Rüggeberg, C. and Petruck, M. R. L. (2003). Surprise: Spanish FrameNet! In Eva Hajičová, et al., editors,
Proceedings of the Workshop on Frame Semantics, XVIIInternational Congress of Linguists (CIL), Prague. Matfyzpress.
Surdeanu, M., Harabagiu, S., Williams, J., and Aarseth,P. (2003). Using Predicate-Argument Structures for Information Extraction. In Erhard Hinrichs et al., editors,
Proceedings of the 41st Annual Meeting of the Association
for Computational Linguistics, pages 8–15.
Swayamdipta, S., Thomson, S., Lee, K., Zettlemoyer, L.,
Dyer, C., and Smith, N. A. (2018). Syntactic scaffolds
for semantic structures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3772–3782, Brussels, Belgium, OctoberNovember. Association for Computational Linguistics.
Søgaard, A., Plank, B., and Alonso, H. M. (2015). Using
Frame Semantics for knowledge extraction from Twitter.
In Proceedings of the Twenty-Ninth AAAI Conference on
Artificial Intelligence.
Upadhyay, S., Faruqui, M., Dyer, C., and Roth, D. (2016).
Cross-lingual models of word embeddings: An empiricalcomparison. CoRR, abs/1604.00425.
Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., andZhou, X. (2020). Semantics-aware BERT for language
understanding. In Proceedings of the Thirty-Fourth AAAI
Conference on Artificial Intelligence (AAAI-2020).

84

Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 85–92
Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020c European Language Resources Association (ELRA), licensed under CC-BY-NC

Building Multilingual Specialized Resources Based on FrameNet:
Application to the Field of the Environment
Marie-Claude L’Homme*, Benoit Robichaud*, Carlos Subirats***Observatoire de linguistique Sens-Texte (OLST), Université de Montréal, C.P. 6128, succ. Centre-ville, Montréal
Québec, H3C 3J7, CANADA**Departamento de Filología Española. Facultad de Filosofía y Letras. Campus Universidad Autonoma de Barcelona,
Edifici B, Carrer de la Fortuna, 08193 Bellaterra, SPAIN{mc.lhomme, benoit.robichaud}@umontreal.ca
carlos.subirats@uab.cat
Abstract
The methodology developed within the FrameNet project is being used to compile resources in an increasing number of specialized
fields of knowledge. The methodology along with the theoretical principles on which it is based, i.e. Frame Semantics, are especiallyappealing as they allow domain-specific resources to account for the conceptual background of specialized knowledge and to explain
the linguistic properties of terms against this background. This paper presents a methodology for building a multilingual resource that
accounts for terms of the environment. After listing some lexical and conceptual differences that need to be managed in such a resource,
we explain how the FrameNet methodology is adapted for describing terms in different languages. We first applied our methodology to
French and then extended it to English. Extensions to Spanish, Portuguese and Chinese were made more recently. Up to now, we havedefined 190 frames: 112 frames are new; 38 are used as such; and 40 are slightly different (a different number of obligatory participants;
a significant alternation, etc.) when compared to Berkeley FrameNet.
Keywords: terminology, FrameNet, Frame Semantics, multilingual resource, environment1.

Authors also point out that the process of acquiring
specialized concepts require “a lengthy chain of priorlearning as a prerequisite to attaining the new concept.”(Fillmore and Baker 2010:317)

Introduction

Frame Semantics (Fillmore 1982), and more specificallythe methodology developed within the FrameNet project(Fillmore and Atkins 1992; Ruppenhofer et al. 2016) isbeing used for the development of resources in an
increasing number of specialized fields of knowledge.
Projects can lead to stand-alone resources based on
FrameNet or to proposals to increase the lexical coverageof Berkeley FrameNet with the addition of specialized
terminology. These projects deal with various fields ofknowledge, such as biology (Dolbey et al. 2006), football(Schmidt 2009; Dicionário da copa de mundo 2020), law(Pimentel 2013), computing (Ghazzawi 2016), linguistics(Malm et al. 2018), and the environment (the resource
presented in this article). Other terminology projects, suchas EcoLexicon (Faber et al. 2016) in the field of theenvironment, apply Frame Semantics without referring
explicitly to the FrameNet methodology.

Additionally, the FrameNet methodology allows designersof specialized resources to account for the linguistic
properties of terms and to connect these properties to aconceptual background. Traditionally, specialized
resources have focused on providing explanations (in somecases, very detailed ones) about the knowledge conveyed
by terms, giving very few details about the linguistic
behavior of these terms. It is often assumed that thisinformation can be found in other, perhaps more general
resources. Things are changing though as work on corporaemphasizes the need for a deeper understanding of
linguistic behavior.
This paper presents a methodology for the development ofa multilingual resource that accounts for environment
terms. The resource, called Framed DiCoEnviro (2020),
covers various topics, such as climate change, renewableenergy, transport electrification, endangered species, andsustainable agriculture. It includes Chinese, English,
French, Portuguese (Lamberti 2019) and Spanish terms.1(The coverage varies considerably from one language toanother as some projects started only recently).
Additionally, besides French and English, topics covered indifferent languages vary.

For designers of specialized resources, the FrameNet
methodology is especially appealing as it first allows themto capture the conceptual background of domain-specificknowledge. As pointed out by Fillmore and Baker (2010),
acquiring new specialized concepts requires a backgroundof other, more familiar concepts:[…] as with the mathematical concept mantissa, which
requires previous familiarity with such concepts as base,power, logarithm, decimal point, and, of course, theconceptual prerequisites of each of these in turn. (Fillmore
and Baker 2010:317)

After listing some lexical and conceptual differences thatmust be managed in domain-specific resources (Section 2),
we describe the steps of our methodology and explain towhat extent the FrameNet methodology needed to be

1

The terminological resource also contains Italian terms, butthese have not been linked to the Framed DiCoEnviro yet.

85

adapted (Section 3). We conclude by providing somefigures regarding the work we have done up to now andmention some directions we wish to take in the future.

look soon reveals that many differences can be spotted
between the situation as it is described in FrameNet and asituation in which species are involved. The first one is thatthe perspective taken in FrameNet concerns human beings.
Other differences are listed below.

2. Why Developing “FrameNets” inSpecialized Fields of Knowledge is not a
Trivial Task

Lexical content of frames. It appears that some lexicalunits that can evoke the RESIDENCE frame cannot apply to
species. Indeed, some lexical units only apply to humanbeings (e.g. resident, squatter) and others only to species(e.g. nest) (Table 1).

Specialized knowledge and terms used to express it can be
unknown to the layperson, as is the case with mantissa
mentioned in the previous section. However, such clear-cut
cases do not exhaust all possible ways to convey
specialized knowledge. Indeed, specialized and common
knowledge interact in various situations and influence eachother in different ways. In addition, terms are often basedon the lexical stock of languages and share with lexicalunits a complex set of relationships (L’Homme andPolguère 2008).
A common mechanism to create terms consists in adding
new meanings to existing lexical items (e.g. green meaning“that is less damageable to the environment”). Other, much
subtler changes can also affect the way lexical items areused in specialized situations. Fillmore pointed out that,
although the concepts of INNOCENT and GUILTY can be
known to laypeople, they are conceptualized differently
when considered from the perspective of law.

RESIDENCE in FrameNet

RESIDENCE in theenvironment

bivouac.n, bivouac.v, camp.n, camp.v,
camped.a, camper.n, dwell.v,
dweller.n, inhabit.v, inhabitant.n,live.v, lodge.v, occupant.n, occupy.v,
reside.v, resident.n, room-mate.n,room.v, shack up.v, squat.v,
squatter.n, stay.v, tenant.n

inhabit, live, nest,
nesting, occupy

Table 1. Different lexical contents for the RESIDENCE frameCore and non-core frame elements (FEs). Core and noncore frame elements can vary when considering the
RESIDENCE situation from the perspective of endangered
species. The Co-resident core FE has no correspondence inthe field of endangered species. On the other hand, therange in which a given species can be found is often
specified.

In both everyday language and legal language there is acontradictory opposition between INNOCENT andGUILTY. In everyday language, the difference depends
on whether the individual in question did or did notcommit the crime in question. In legal language, bycontrast, the difference depends on whether the individualin question has or has not been declared guilty by the court
as a result of a legal action within the criminal system.
(Fillmore 1982:127)

Relations between frames. When considering a residence
situation from the point of view of endangered species, thefact of living in a given area is closely linked to othersituations that concern the state of this species. Species canspread in small or large areas; they can also settle inlocations in larger and larger numbers. Species are also
vulnerable to certain threats that will cause them to be lessabundant in an area or even disappear. Finally, measures
can be taken to place species in an area so they can start
occupying it again. In other words, the relations shared bysituations from the point of view of endangered species
differ sharply from those described in FrameNet (Figure 1).

Such differences can be observed in many other fields ofknowledge. In the environment, which is the field that weare concerned with in this article, situations that may seemfamiliar at first sight are considered in a way that contrasts
with what we will call everyday situations. We examine a
simple example, i.e. a situation that involves species thatlive in a specific location, and consider it from theperspective of endangered species. We will compare thissituation to a similar one captured in FrameNet (which isconsidered here, although this is not entirely the case, as areference for “everyday situations”).2

This simple example is by no means an exceptional case.
Many more examples could be mentioned in whichsituations are similar or do not differ drastically from othermore common situations. However, differences emerge atmany descriptive levels (lexicon, frame as was discussed
above) given that entities and events can be conceptualized
differently in specialized fields of knowledge.

In contrast with the mantissa example, everybody has atleast basic knowledge about this situation.3 We could
assume that it is closely related to the situation captured in
the RESIDENCE Frame, which is defined as follows:This frame has to do with people (the Residents) residing
in Locations, sometimes with a Co-resident (FrameNet
2020).

On a lexical level, differences in conceptualization result in
subtle “meaning modulations”4 that we differentiate frompolysemy per se and are much more difficult to pinpoint
using standard lexico-semantic criteria. For instance, theIn both cases, living entities make use of locations for
shelter and to carry out daily activities. However, a closer

24More differences between Berkeley FrameNet and the Framed
DiCoEnviro are listed in L’Homme et al. (2016).

These phenomena are linked to what Cruse (2011) calledmicrosenses.

3This being said, some technical aspects of the situation might be
only known by the expert.

86Figure 1 Relations shared by the RESIDENCE frame in FrameNet and the Framed DiCoenviro

verb introduce denotes an activity whereby someone places
a species in an area where it can live and reproduce (Toadpopulations, predatory fish should not BE INTRODUCED
into breeding ponds). It is related to reintroduce andintroduction and is opposed to eliminate and extirpate. In
everyday language, introduce includes many more
activities in which someone places something in a given
location. Given the use of introduce in everyday language,
it is linked to a much larger set of different lexical units(LUs) (such as imbed, implant, insert, place, etc.). It would
be difficult to contend that we are dealing with a completely
new meaning in the field of endangered species (as was thecase with green mentioned earlier).
Furthermore, some distinctions can be relevant when
considering lexical units from the perspective of a
specialized domain but would not be made in othercontexts. For instance, the verb hunt lends itself to two
different uses in corpora that deal with the environment.
One corresponds to the activity whereby a meat eaterchases, captures and kills other animals for food; thesecond corresponds to the activity carried out by modern
human beings that consists in chasing animals for otherkinds of reasons, this activity having a negative impact on
the conservation of species. Hunt1 is linked to other terms,
such as predation, and to prey; while hunt2 is linked topoach, capture, and fish (Figure 2).

Figure 2 Distinguishing hunt in the environment

Another approach – which is the one taken in this work –consists in situating terminology within the broader
spectrum of the lexicon. This will allow us to connect
specialized meanings and usage to a more “general”
lexicon. By doing so, we must also attempt to better define
what is specific to a given field of knowledge and what thisfield of knowledge shares with language in general.

3.

In order to account for these phenomena, designers ofspecialized resources can build stand-alone resources for
specific fields of knowledge. This strategy certainly has theadvantage of allowing designers and users of theseresources to focus exclusively on the way situations areconceptualized in a given area of knowledge. However, therelationship with common knowledge and the generallexical stock of languages is lost. This strategy also implies
that stand-alone resources need to be compiled each time a
new domain or topic is targeted.

A Methodology Adapted to Specialized
Fields of Knowledge

Our methodology is bottom-up (Schmidt 2009) as in manyother terminology projects. Terminologists are not expertsof the fields they are asked to account for, especially when
they recently embarked on a project. They must rely onknowledge sources to identify relevant terms and describe
them. The first five steps of the methodology consist incompiling terminological entries. This part of the work isheavily based on Explanatory Combinatorial Lexicology(Mel’čuk et al. 1995) and is implemented in aterminological resource called the DiCoEnviro (2020).
Once these entries contain enough linguistic data, we

87

proceed to define frames based on the knowledge acquired
during these first steps. Frames are modelled in a resource
called the Framed DiCoEnviro (2020). Both resources areinterlinked.

is that unusually frequent units correspond to terms. Table
2 shows the first results of this method applied to our
corpus of endangered species.

Compiling terminological entries:
1.
2.
3.
4.
5.

Canonical
form

Compilation of specialized corpora
Identification of terms (semi-automated)
Selection and extraction of contexts
Definition of the argument structure
Annotation of contexts

Finding frames among lexical entries:
6.
7.
8.

Definition of semantic frames
Encoding of frames
Definition of relations between frames

As was mentioned above, our resources account for terms
in different languages and more could be added in the
future. The first five steps of the methodology are applied
to languages separately. Native speakers or near native
speakers of each language are responsible for building
lexical entries. The definition of semantic frames can take
into account terms in different languages.

3.1

Compilation of Corpora

Since terminologists are seldom experts of the field theydescribe, they rely heavily on the contents of corpora to
locate relevant terms and information about their uses.
Hence, all terminological projects start with thecompilation of a corpus. Since the field of the environment
encompasses a wide range of subjects and that theterminology and the number of occurrences of given terms
can vary quite drastically from one subject to another, we
work on separate topics and compile a corpus accordingly.

Specificity
score

Variants

specie53710

202.96 specie, species

species3046

185.74 species

habitat2614

173.96 habitat, habitats

conservation

1388

112.76 conservation

recovery1142

108.22 recovery, recoveries

endangered928

103.71 endangered

population

1621

98.61

threaten943

threaten, threatens,
84.46 threatened,threatening

extinction

603

81.63 extinction, extinctions

endanger504

endanger,
72.48 endangered,endangering

status866

71.86 status

nest

422

69.94

threat789

67.43 threat, threats

population,
populations

nest, nests, nested,nesting

Table 2. First term candidates extracted from a corpus on
endangered species

Terminologists must then analyze this list, keep thosecandidates that correspond to relevant terms, and ignore
other lexical items. Although some cases do not raise
problems (e.g. species, habitat), others might be more
problematic (e.g. recovery). Terminologists look up
problematic cases in the corpus to examine the context inwhich they appear.

When we embark on a new project, we start with corporaof about 500,000 words (this corresponds roughly to 30-40
different texts of varying sizes ranging between 1,000 to
50,000 occurrences). Corpora are often enriched at a laterstage (for instance, our English corpus on endangered
species now amounts to around 1,060,000 tokens andcomprises 88 different texts).

3.2

Frequency

3.3

Extraction of Contexts

The third step of the methodology consists in going back to
the corpus and retrieving contexts that will be placed inlexical entries. These contexts are extremely useful toanalyze terms and complete other parts of their description.
Contexts are also annotated, as will be seen further on.

Identification of Terms

Once a corpus on a specific topic is compiled, we proceed
to identify relevant terms. We first approach this task with
an automated method that produces a list of candidateterms.

For each term, terminologists extract 15 to 20 different
contexts. These are selected according to the richness of theinformation they contain (presence and number ofparticipants, argumental or circumstantial status of eachparticipant, explanations of the meaning, etc.). Experiencehas shown that 15 to 20 contexts per meaning are sufficient
to give a clear picture of how terms behave in a specialized
corpus. Beyond that point, the information becomesredundant.

We submit our corpus to a term extractor, called TermoStat
and developed by Drouin (2003), and have it search fornouns, verbs, adjectives and adverbs. The term extractor
compares the content of a specialized corpus to a reference
corpus. For English, the latter is a combination of the
British National Corpus (BNC) and the American National
Corpus (ANC). More specifically, the extractor compareslemmatized and part of speech tagged units in both corpora
and produces a list of candidate terms ranked according totheir specificity in the specialized corpus. This specificity
is a reflection of the unusual frequency of the unit in thespecialized corpus. The hypothesis underlying this method

At this stage, terminologists might make meaning
distinctions they missed during the previous step. Since
different meanings are described in separate entries,

5

Since lemmatization and POS tagging are automated, there
might be some erroneous entries

88Figure 3: Entries in English, French, Portuguese and Chinese in the DiCoEnviro (2020)

contexts must reflect these distinctions and be placed in theright entry.

Simon Cienega, where it[Patient] INHABITS riparian
thickets and similar native habitats[Location].

3.4

Young or small fish[Patient] are noted to INHABIT gravel
riffles[Location], and all individuals may move to deeper waters
to overwinterDefinition of Argument Structure

The fourth step consists in defining the argument structureof terms. This step – albeit central in our methodology –does not apply to terms that are non-predicative (e.g.,
animal, organism, plant, wolf). At this stage, terminologists
determine how many arguments a term has and state thesearguments in the entry (habitat of X; X inhabits Y).

Current populations, especially young[Patient], are much
reduced and INHABIT more restricted areas of thelake[Location].
Other species of pupfish in the Pecos River system[Patient]
INHABIT more saline waters[Location].

Arguments are represented with two different systems. We
first label them with semantic roles that express therelationship between the term and its arguments. The labels
used for the arguments of inhabit are: Patient inhabits
Location.6 An additional label states what we call the
typical term (Species inhabits Area). This typical term isdesigned to give the user an idea of the kinds of terms that
can instantiate the arguments.

3.5

The highly endangered Alabama beach mouse[Patient]
once[Time] INHABIT most of Alabama’s Gulf
Coast[Location].

Once these first five steps are completed, we obtainterminological entries that contain a statement of theargument structure and up to 20 annotated contexts (Figure3). Terminologists then proceed to identify terms that arelikely to evoke the same frame.

Annotation of Contexts

Once the argument structure is defined, terminologists
proceed to annotate the set of contexts based on themethodology devised for the FrameNet project(Ruppenhofer et al. 2016).

3.6

Definition of Semantic Frames

Given that our methodology is bottom-up, theidentification of frames is first guided by different lexicosemantic properties of terms that are described in theirentries:

The examples selected below represent a sample of theannotated contexts for the term inhabit.
This is primarily a species of the lowlands of central andsouthwest Arizona and adjacent areas, where it is apermanent resident along desert rivers and streams (Tweit
and Finch 1994). It is found in New Mexico only in Grant
and Hidalgo counties primarily in the Gila Valley and at San

The same number of arguments: e.g., inhabit, live
and occupy have two arguments.
Arguments of a similar nature: e.g. the arguments
of inhabit, live and occupy are labelled as Patient(L’Homme (2015). Frame elements in FrameNet are relevant
within a specific frame. In our resources, labels should be applied
to large sets of terms.

6

It should be said at this point that the labels used in our
terminological resources differ from those used in FrameNet

89

Figure 4. Comparison of terms and lexical units contained in FrameNetand Location, and are instantiated by terms that
denote living organisms and/or habitats (Patient;animal, fish, species, etc.; Location: coast,environment, habitat, reserve, etc.).
Shared circumstantials (found in annotations).

Of course, there are many cases for which nocorrespondence can be established and we must also create
frames that account for our specific data. More than half ofthe frames that appear in the Framed DiCoEnviro were
defined specifically to account for situations in the field of
the environment. For instance, a new frame needed to becreated to account for the meaning distinction that was
mentioned above for the verb hunt. In these cases, we createa name that attempts to be evocative of the situation that it
represents.

Of course, shared participants are useful clues to identify
terms evoking the same frame, but terminologists must
define the content of frames based on much more
information. Terms must denote the same general situationand share presuppositions about it (Ruppenhofer et al.
2016). Hence, based on our descriptions, we could establishthat the terms inhabit, live and occupy evoke the same
situation, whereby a living entity finds itself in a given
location that should provide it with what it needs to feed,
reproduce and survive.

3.7

Encoding of Frames

Once frames are defined, they are encoded in an entry that
accounts for the following:To help them define frames, terminologists also refer toBerkeley FrameNet (2020). They look for corresponding
data in the English data. To assist them in this process, atool compares the XML versions of both resources (Figure4), locates corresponding lemmas and extracts relevant
information. If a frame was already encoded in FrameNet
and that the data it describes fits the properties of the terms
in the field of endangered species, the frame defined in
FrameNet is used and adapted. For instance, this was
possible for the terms inhabit, live and occupy. We thus
based our frame on the one in FrameNet. Of course, manydifferences appear in the descriptions given in eachresource (different lexical content, labels used forparticipants, etc., see Section 2). Furthermore, when webase our frame on an existing one in FrameNet, we use thesame name and provide a link that will lead users to its
description in the original FrameNet resource.7






3.8

The name of the frame.
A definition formulated for the field of theenvironment and stating the obligatory
participants.
Example(s) for each of the languages described.
An indication of the reference to FrameNet with ahyperlink to FrameNet wherever relevant.
The list of participants (obligatory and optionalones).
The list of terms that evoke this frame in different
languages; hyperlinks to the DiCoEnviro areprovided to visualize terminological entries andcontextual annotations.

Definition of Relations Between Frames

Situations are connected in different ways, and frames that
capture these situations can be linked so as to make theseconnections
explicit.
For
instance,the
SPECIES_COLONIZATION frame (with LUs such as colonize,in the Framed DiCoEnviro when selecting the “Click here to see
associated FrameNet infos”. More explanations are given about
this feature in L’Homme et al. (2016).

7

Users can also view the similarities and the differences between
frames as they are represented in FrameNet and those that appear

90

recolonization) is connected via a Causation relation with
the ADDING_SPECIES_IN_LOCATION frame (with LUs suchas introduce, introduction, reintroduce) (Figure 1).
Relations described in the Framed DiCoEnviro are basedon those defined in FrameNet (Inheritance, Perspective,
Use, Subframe, Precedence, Causation, See also).
However, two additional relations were introduced:

Opposition:Is
opposedto(e.g.
Removing_trees_from_location Is opposed toAdding_trees_in_location).
This
relation
capturesdomain-specificoppositions.
Ithighlights oppositions such as reversiveness andcontrastivity.
Property: Is a property of, Has property (e.g.
Sustainability is a property of Human_activity).
Again, this relation captures recurrent domainspecific relations. It might be refined in the futurewhen more data is described.

Frames

FR

EN

ES

ZH

PT

New

112

337

240

20

34

11

Unchanged

38

124

96

8

11

3

Differences

40

149

114

18

20

3

TOTAL

190

610

450

46

65

17

Table 3. Frames in the Framed DiCoEnviro and LUs in different
languages

Table 4 gives a summary of the annotations that wererevised up to now in all five languages. (Some annotated
LUs have not been assigned to frames yet.)Once linked, frames can lead to larger scenarios that givean overview of how events are connected in the field of theenvironment. For instance, one scenario describes thedifferent activities that species undergo or carry out (livesomewhere, feed, reproduce, etc.) (part of this scenario isshown in Figure 1) as opposed to another one that accounts
for human activities. Another scenario, called
Understanding life, shows the different connections
between living organisms according to the terms used toexpress them (species, population, predator, offspring,etc.).

Annotated contexts
FR

12.262

EN

9,004

ES

2,267

ZH

1,150

PT

635

Table 4. Annotations in different languages

4.

Summary

5.

This article presented a bottom-up methodology to compile
FrameNet-like domain-specific resources. We applied thismethodology to different environmental topics and todifferent languages. Our descriptions are first placed in aterminological resource, called DiCoEnviro (2020). Basedon these terminological descriptions, we proceed to
identify frames. Once a frame is identified and defined,
terminological entries are linked to a frame module that issuperimposed on terminological entries and that is visible
in another resource called the Framed DiCoEnviro (2020).
Up to now, 190 frames were defined.

Future Work

The work reported in this article is ongoing. New
terminological entries are added in different languages ona regular basis. Some of these entries can be assigned toexisting frames or lead to the definition of new frames. We
also extend the coverage of the DiCoEnviro by adding
terms linked to new environmental topics.
Our methodology and infrastructure can easily be extended
to new languages. However, as was mentioned above, sometools might not be available for some languages; in thesecases, adjustments need to be made. As the descriptive
work progresses in different languages, we should get aclearer picture of interlinguistic differences and the levels
at which they occur (lexicon, frame).

We first applied our methodology to French and thenextended it to English. Our infrastructure can easily be
adapted other languages and entries in Spanish, Portuguese
and Chinese are currently being added. However, the nonavailability of tools can raise problems in certain
languages. For instance, the term extractor TermoStat has
not yet been adapted to Chinese. In this case, an alternative
solution needed to be sought.

The relationship with Berkeley FrameNet is visible throughthe Framed DiCoEnviro and this allows users to visualize
similarities and differences between domain-specific and“everyday” situations to a certain extent. It would also beinteresting to establish a connection the other way around,i.e. allow users of FrameNet to view how situations can beconceptualized differently in specialized areas ofknowledge. For the time being, interrelationships between
the two resources can only be made manually due to several
methodological differences that exist between them.
However, it would be useful to attempt to mitigate thesedifferences in order to capture most of them automatically.

When defining frames, we refer to Berkeley FrameNet. In
many cases, an existing frame can be used or adapted to ourdata. However, in many other cases (more than half), a newframe is created. More specifically, 112 new frames werecreated; 38 are used as such; 40 are slightly different (adifferent number of obligatory participants; a significant
alternation, etc.). Table 3 gives an overview of the work
carried out in different languages and of the frames defined
up to now. Most of these frames were added to 17
scenarios.

91

6.

langue générale, In Lexicographie et terminologie :histoire de mots. Hommage à Henri Béjoint, ed. by F.
Maniez, and P. Dury. 191-206. Lyon: Presses del’Université de Lyon.

Acknowledgments

The authors wish to thank collaborators who work on the
different versions of the DiCoEnviro and Framed
DicoEnviro: Maria Francesca Bonnadona, Flávia Lamberti
Arraes, and Zheng Ying. They also extend their thanks to
two anonymous reviewers who raised many interesting
points and whose comments helped clarify parts of thepaper.

7.

L’Homme, M.C., B. Robichaud, and C. Subirats. (2014).
Discovering frames in specialized domains, In Language
Resources and Evaluation, LREC 2014, Reykjavik,
Iceland.
L’Homme, M.C., C. Subirats, and B. Robichaud. 2016. A
Proposal for combining “general” and specialized
frames. In Proceedings of the Workshop on Cognitive
Aspects of the Lexicon. 156–165, Osaka, Japan.

Bibliographical References

Cruse, D.A. (2011). Meaning in Language, Cambridge:
Cambridge University Press.

Malm, P., S. Mumtaz Virk, L. Borin, and A. Savera (2018).
LingFN. Towards a FrameNet for the Linguistics
Domain. In International FrameNet Workshop. 2018.
Multilingual
FrameNets
andConstructions.
Proceedings. Language Resources and Evaluation(LREC 2018), ed. by T. Timponi Torrent, L. Borin, and
C. Baker, 37-43. Miyazaki, Japan.

Dolbey, A., M. Ellsworth, and J. Scheffczyk (2006).
BioFrameNet: A Domain-specific FrameNet Extension
with links to Biomedical Ontologies. In KR-MED 2006.
Biomedical Ontology in Action, 87-94. Baltimore,
Maryland.
Drouin, P. (2003). Term Extraction Using Non-Technical
Corpora as a Point of Leverage. Terminology 9(1): 99–
117.

Mel’čuk. I., A. Clas, and A. Polguère (1995). Introduction
à la lexicologie explicative et combinatoire, Duculot:
Louvain-la-Neuve.

Faber, P., P. León-Araúz, and A. Reimerink. (2016).
EcoLexicon: New features and Challenges. In
GLOBALEX 2016: Lexicographic Resources for Human
Language Technology in conjunction with the 10thedition of the Language Resources and Evaluation
Conference, ed. by I. Kernerman, I. Kosem Trojina, S.
Krek, and L. Trap-Jensen, 73-80. Portorož, Slovenia.

Pimentel, J. (2013). Methodological bases for assigningterminologicalequivalents.
A
Contribution.
Terminology 19(2): 237-257.

Fillmore, C. (1982). Frame Semantics. In Linguistics in theMorning Calm, ed. by The Linguistic Society of Korea,
111-137. Seoul: Hanshin.

Ruppenhofer, J, M. Ellsworth, M. Petruck, C. Johnson, C.
Baker, and J. Scheffczyk, 2016. FrameNet II: Extended
Theory and Practice.
(https://framenet.icsi.berkeley.edu/fndrupal/index.php?q=the_book). Accessed January 27, 2017.

Fillmore, C. J., and B.T. Atkins. 1992. Toward a Framebased Lexicon: The Semantics of RISK and its
Neighbors.” In Frames, Fields and Contrasts, ed. by A.
Lehrer, and E. Feder Kittay, 75-102. Hillsdale, New
Jersey: Lawrence Erlbaum Assoc.

Schmidt, T. (2009). The Kicktionary – A Multilingual
Lexical Resources of Football Language. In Multilingual
FrameNets in Computational Lexicography. Methods
and Applications, ed. by H. C. Boas, 101-134.
Berlin/NewYork: Mouton de Gruyter.

Fillmore, C.J., and C. Baker (2010). A Frames Approach to
Semantic Analysis. In The Oxford Handbook of
Linguistic Analysis, ed. by B. Heine and H. Narrog,
313-339. Oxford: Oxford University Press.

8.

Language Resource References

American National Corpus (ANC). (http://www.anc.org/).
Accessed March 30, 2020.
British National Corpus (BNC).
(http://www.natcorp.ox.ac.uk/corpus/index.xml).
Accessed March 30, 2020.

Ghazzawi, N. (2016). Du terme prédicatif au cadresémantique : méthodologie de compilation d’une
ressource terminologique pour des termes prédicatifs
arabes en informatique. Thèse présentée à l’Universitéde Montréal : Montréal.

Dicionário da copa de mundo(https://www.ufjf.br/framenetbr/dicionario/).
Accessed February 18, 2020.

Lamberti Arraes, F. (2019). DiCoEnviro, a Multilingual
Terminological Resource on the Environment: The
Brazilian Portuguese Experience. In Electronic
Lexicography in the 21st Century. Proceedings of theeLex 2019 Conference, ed. by I. Kosem, T. Zingano
Kuhn, M. Correia, J.P. Ferreria, M. Jansen, I. Pereira, J.
Kallas, M. Jakubíček, S. Krek, and C. Tiberius, 291-304.
Sintra, Portugal. Brno: Lexical Computing CZ.

DiCoEnviro. Dictionnaire fondamental de l’environnement(http://olst.ling.umontreal.ca/cgibin/dicoenviro/search.cgi). Accessed February 21,
2020.
A Framed Version of DiCoEnviro(http://olst.ling.umontreal.ca/dicoenviro/framed).
Accessed February 18, 2020.

L’Homme, M.C. (2015). Predicative lexical units interminology. In Recent Advances in Language
Production, Cognition and the Lexicon, ed. by N. Gala,
R. Rapp, and G. Bel-Enguix. 75-93. Berlin: Springer

FrameNet (https://framenet.icsi.berkeley.edu/fndrupal/).
Accessed February 21, 2020.

L’Homme, M.C., and A. Polguère (2008). Mettre en bonstermes les dictionnaires spécialisés et les dictionnaires de92

Author Index
Baker, Collin, 77
Belcavello, Frederico, 23
Cai, Tianyuan, 56
Christopoulos, Hephaestion, 48
Czulo, Oliver, 1‘Dargis,Roberts, 63
Diniz da Costa, Alexandre, 23

Fokkens, Antske, 31
Gargett, Andrew, 70
Giouli, Voula, 48
Gruzitis, Normunds, 63
Ilievski, Filip, 31
L’ Homme, Marie-Claude, 85
Lee, John, 56
Leung, Tommi, 70
Liu, Meichun, 56
Lorenzi, Arthur, 77
Marzinotto, Gabriel, 41
Matos, Ely, 23
Minnema, Gosse, 13
Nešpore-Bērzkalne, Gunta, 63
Ohara, Kyoko, 8
Pilitsidou, Vera, 48
Postma, Marten, 31
Remijnse, Levi, 13, 31
Rituma, Laura, 63
Robichaud, Benoît, 85
Saulite, Baiba, 63
Subirats, Carlos, 85
Timponi Torrent, Tiago, 1, 23
Titarsolej, Sam, 31
Viridiano, Marcelo, 23
Vossen, Piek, 31
Ziem, Alexander, 193