Venue: NAACL 2018

# The Alexa Meaning Representation Language

Thomas Kollar, Danielle Berry, Lauren Stuart, Karolina Owczarzak, Tagyoung Chung {kollart, danjjean, Isstuar, kawoc, tagyoung}@amazon.com Lambert Mathias, Michael Kayser, Bradford Snow, Spyros Matsoukas {mathiasl, mikayser, brsnow, matsouka}@amazon.com

###### Abstract

This paper introduces a meaning representation for spoken language understanding. The Alexa meaning representation language (AMRL), unlike previous approaches, which factor spoken utterances into domains, provides a common representation for how people communicate in spoken language. AMRL is a rooted graph, links to a large-scale ontology, supports cross-domain queries, fine-grained types, complex utterances and composition. A spoken language dataset has been collected for Alexa, which contains \(\sim 20\)k examples across eight domains. A version of this meaning representation was released to developers at a trade show in 2016.

## 1 Introduction

Amazon has developed Alexa, a voice assistant that has been deployed across millions of devices and processes voice requests in multiple languages. This paper addresses improvements to the Alexa voice service, whose core capabilities (as measured by the number of supported intents and slots) has expanded more than four-fold over the last two years. In addition more than ten thousand voice skills have been created by third-party developers using the Alexa Skills Kit (ASK). In order to continue this expansion, new voice experiences must be both accurate and capable of supporting complex interactions.

However, as the number of features has expanded, adding new features has become increasingly difficult for four primary reasons. First, requests with a similar surface form may belong to different domains, which makes it challenging to add features without degrading the accuracy of existing domains. For example, similar linguistic phrases such as _"order me an echo dot"_ (e.g., for Shopping) have a similar form to phrases used for a ride-hailing feature such as, _"Alexa, order me a taxi"_. The second challenge is that a fixed flat structure is unable to easily support certain features Gupta et al. (2006b), such as cross-domain queries or complex utterances, which cannot be clearly categorized into a given domain. For example, _"Find me a restaurant near the sharks game"_ contains both local businesses and sporting events and _"Play hunger games and turn the lights down to 3"_ requires a representation that supports assigning an utterance to two intents. The third challenge is that there is no mechanism to represent ambiguity, forcing the choice of a fixed interpretation for ambiguous utterances. For example, _"Play Hunger Games"_ could refer to an audiobook, a movie, or a soundtrack. Finally, representations are not reused between skills, leading to the need for each developer to create a custom data and representations for their voice experiences.

In order to address these challenges and make Alexa more capable and accurate, we have developed two key components. The first is the Alexa ontology, a large hierarchical ontology that contains fine-grained types, properties, actions and roles. Actions represent a predicate that determines what the agent should do, roles express the arguments to an action, types categorize textual mentions and properties are relations between type mentions. The second component is the Alexa Meaning Representation Language (AMRL), a graph-based domain and language independent meaning representation that can capture the meaning of spoken language utterances to intelligent assistants. AMRL is a rooted graph where action, operators, relations and classes are labeled vertices and properties and roles are labeled edges. Unlike typical representations for spoken language understanding (SLU), which factors language understanding into the prediction of intents (non-overlapping actions) and slots (e.g., named entities) Gupta et al. (2006a), our representation is

grounded in the Alexa ontology, which provides a common semantic representation for spoken language understanding and can directly represent ambiguity, complex nested utterances and cross-domain queries. Unlike similar meaning representations such as AMR (Banarescu et al., 2013), AMRL is designed to be cross-lingual, explicitly represent fine-grained entity types, logical statements, spatial prepositions and relationships and support type mentions. Examples of AMRL and the SLU representations can be seen in Figure 1.

The AMRL has been released via Alexa Skills Kit (ASK) built-in intents and slots in 2016 at a developers conference, offering coverage for eight of the \(\sim\)20 SLU domains 1. In addition to these domains, we have demonstrated that the AMRL can cover a wide range of additional utterances by annotating a sample from all first and third-party applications. We have manually annotated data for 20k examples using the Alexa ontology. This data includes the annotation of \(\sim\)100 actions, \(\sim\)500 types, \(\sim\)20 roles and \(~{}\sim\)172 properties.

Footnote 1: [https://amzn.to/2qDjNcJ](https://amzn.to/2qDjNcJ)

## 2 Approach

This paper describes a common representation for SLU, consisting of two primary components:

* A large-scale hierarchical ontology developed to cover all spoken language usage.
* A rooted graph that provides a common semantic representation, is compositional and can support complex user requests.

These two components are described in the following sections.

### The Alexa ontology

The Alexa ontology provides a common semantics for SLU. The Alexa ontology is developed in RDF and consists of five primary components:

* **Classes** A hierarchy of Classes, also referred to as types, is defined in the ontology. This hierarchy is a rooted tree, with finer-grained types at deeper levels. Coarse types that are children of Thing include Person, Place, Intangible, Action, Product, CreativeWork, Event and Organization. Fine-grained types include MusicCreording and Restaurant.
* **Properties** A given class contains a list of properties, which relate that class to other classes. Properties are defined in a hierarchy, with finer-grained classes inheriting the properties of its parent. There are range restrictions on the available types for both the domain and range of the property.
* **Actions** A hierarchy of actions are defined as classes within the ontology. Actions cover the core functionality of Alexa.
* **Roles** Actions operate on entities via roles. The most common role for an Action is the **.object** role, which is defined to be the entity on which the Action operates.
* **Operators and Relations** A hierarchy of operators and relations represent complex relationships that cannot be expressed easily as properties. Represented as classes, these include ComparativeOperator, Equals and Coordinator (Figure 2).

The Alexa ontology utilized schema.org as its base and has been updated to include support for spoken language. In addition, using schema.org as the base of the Alexa Ontology means that it shares a vocabulary used by more than 10 million websites, which can be linked to the Alexa ontology.

### Alexa meaning representation language

AMRL leverages classes, properties, actions, roles and operators in the main ontology to create a compositional, graph-based representation of the meaning of an utterance. The graph-based representation conceptualizes each arc as a property and each node as an instance of a type; each type can have multiple parents. Conventions have been developed to annotate the AMRL for an utterance accurately and consistently. These conventions focus primarily on linguistic annotation, and only consider filled pauses, edits, and repairs in limited contexts. The conventions include:

* [noitemsep]
* **Fine-grained type mentions** When an entity type appears in an utterance, the most fine-grained type will be annotated. For _"turn on the light"_, the mention _'light'_ could be annotated as a Device. However, there is a more appropriate finer-grained type, Lighting which will be selected instead.
* **Ambiguous type mentions** When more than one fine-grained type is possible, then the annotator will utilize a more coarse-grained 

type in the hierarchy. This type should be the finest-grained type that still captures the ambiguity. For example, in the utterance _"play thriller'_, "thriller" can either be a MusicAlbum or a MusicRecording. Instead of selecting one of these a more coarse-grained type of MusicCreativeWork will be chosen. When the ambiguity would force fallback to the root class of the ontology Thing, AMRL annotation chooses a sub-class and marks the usage of it as _uncertain_.
* **Properties** Properties are annotated when they are unambiguous. For example, _"find books by truman capote"_, the use of the **.author** property on the Book class is unambiguous. Similarly, for _"find books about truman capote"_ the use of the **.about** property on the Book class is unambiguous.
* **Ambiguous property usage** When there is uncertainty in the property that should be selected for the representation, the annotator may fall back to a more generic property.
* **Property inverses** When a property can be annotated in two different directions, a canonical property is defined in the ontology and used for all annotations. For example, **.parentOrganization** has an inverse of **.subOrganization**. The former is selected as canonical for annotation flexibility and to eliminate cycles in the graph.

A few of these properties have special meaning at annotation time. Specifically, for the annotation of textual mentions there exist three primary properties: **.name**, **.value** and **.type**. The conventions for these properties are as follows:

* **.name** This is a nominal mention in the utterance, the **.name** property links the text to an instance of a class. **.name** is only used for mentions that are not a numeric quantity or enumeration. An example of **.name** for a Musician class would be _"madonna"_.
* **.value** This is defined in the same way as **.name** but is used for mentions that are numeric quantities or enumerations. For instance, _"two"_ would be a **.value** of an Integer class.
* **.type** This is a generic mention of an entity type. For example, _"musician"_ is a **.type** mention of the Musician class.

One action (NullAction) has a special meaning. This is annotated whenever a SLU query does not have an associated action or the action is unclear. This happens, for example, when someone says, _"temperature"_. In contrast, _"show me the temperature"_ is annotated with the more specific DisplayAction.

Figure 1: This figure shows the SLU representation on the left and the AMRL representation on the right. The three components of the SLU representation, domain (D), intent (I) and slots (S) are shown. The intent is different (e.g., “ListenMedialIntent” vs. “ActivateIntent”), despite the presence of“_turn on”_. On the right are the same utterances represented in the AMRL. The nodes represent the instances of classes defined in an ontology, while the directed arcs connecting the class instances are properties. The root node of both graphs is the action, ActivateAction is shared across these two utterances, providing the domain-less annotation with a uniform representation for the same carrier phrase. “-0” indicates the first mention of a type in the utterance, and can be used used to denote co-reference across multiple dialog turns.

 



### Expanded Language Support

AMRL has been used to represent utterances that are either not supported or challenging to support using standard SLU representations. The following section describes support for anaphora, complex and cross-domain utterances, referring expressions for locations and composition.

#### 2.3.1 Anaphora

AMRL can natively support pronominal anaphora resolution both within the same utterance or across utterances. For example:

* Within utterance: _"Find the highest-rated toaster and show me its reviews"_
* Across utterances: _"What is Madonna's latest album" "Play it."_ Terminal nodes refer back to the same (unique) entity. An example annotation across multiple utterances can be seen in Figures 20a and 20b. Similar to the above, it can handle bridges within discourse, such as, "find me an italian restaurant" and "what's on its menu."

#### 2.3.2 Inferred nodes

AMRL contains nodes that are not grounded in the text. For example, for the utterance, in Figure 2a there are two inferred nodes, one for the address of the restaurant and another for the address of the sports event. Not explicitly representing types has two primary benefits. First, certain linguistic phenomena such as anaphora are easier to support. Second, the representation is aligned to the ontology, which enables direct queries against the knowledge base. Inferred nodes are the AMRL way to perform reification.

#### 2.3.3 Cross-domain utterances

Using the common semantics of AMRL means that parses do not need to obey domain boundaries. For example, these utterances would belong to two domains (e.g., sports and local search): _"Where is the nearest restaurant"_ and _"What is happening at the Sharks game"_. AMRL, as in Figure 2a, can handle utterances that span multiple domains, such as the one shown in Figure 2a.

#### 2.3.4 Conjunctions, disjunctions and negations

AMRL can cover logical expressions, where there can be an arbitrary combination of conjunctions, disjunctions, or conditional statements. Some examples of object-level or clause-level conjunctions include:

* Object-level conjunction: _"Add milk, bread, and eggs to my shopping list"_
* Clause-level conjunction: _"Restart this song and turn the volume up to seven"_ Conjunctions and disjunctions are represented using a Coordinator class. The "value" property defined which logical operation is to be performed. Examples of the AMRL representation for these is shown in Figure 2b and 2c.

#### 2.3.5 Conditional statements

Conditional statements are not usually represented in other formalisms. An example of a conditional statement is, _"when its raining, turn off the sprinklers"_. Time-based conditional statements are special cased due to their frequency in spoken language. For time-based expressions (e.g., _"when it is three p.m., turn on the lights"_), a start-Time (or endTime) property is used on the action to denote the condition of when the action should start (or stop). For all other expressions, we use the ConditionalOperator, which has a "condition" property as well as a "result" property. When the condition is true, then the result would apply. The constrained properties are defining the arguments of the Equals operator. An example can be seen in Figure 4. A deterministic transformation from the simplified time-based scheme to ConditionalOperator form when greater consistency is desired.

#### 2.3.6 Referring expressions for locations

AMRL can represent locations and their relationships. For simpler expressions that are common, such as "on" or "in," properties are used to represent the relationship between two entity mentions. For other spatial relations, such as "between" or "around," an operator is introduced. Two examples of spatial relationships can be seen in Figure 2d. In this example "beside" grounds to the relation being used (e.g., "beside") and uses two properties (e.g., constrained and target), which are the the first and second arguments to the spatial preposition.

#### 2.3.7 Composition

AMRL supports composition, which enables reuse of types and subgraphs to represent utterances with similar meanings. For example, Figures 2e and 2f show the ability to create significantly different actions only by changing the type of the object of the utterance. Such substitution can occur

anywhere in the annotation graph. PlaybackAction is used to denote playing of the entity referred to by the object role.

#### 2.3.8 Unsupported features

Although many linguistic phenomena can be supported in AMRL, there are a few that have not been explicitly supported and are left for future work. These include existential and universal quantification and scoping and conventions for agency (most requests are imperative). In addition, there is currently no easy way to convert to first order logic (e.g., lambda calculus), due to conventions that simplify annotation, but lose information about operators such as spatial relationships.

## 3 Dataset

Data has been collected for the AMRL across many spoken language use-cases. The current domains that are supported include music, books, video, local search, weather and calendar. We have prototyped mechanisms to speed up annotation via paraphrasing Berant and Liang (2014) and conversion from our current SLU representation, in order to leverage the much larger data available. The primary mechanism we have for data-acquisition is via manual annotation. Tools have been developed in order to acquire the full graph annotated with all the properties, classes, actions and operators.

AMRL manual annotation is performed by data annotators in four stages. In the first stage an action is selected, for example ActivateAction in Figure 0(b). The second stage defines the text spans in an utterance that link to a class in the ontology (e.g., "michael jackson" is a Musician type and "thriller" and "song" are MusicRecording types, the first is a **name** mention, while the latter is a **type** mention. The third stage creates connections between the classes and defines any missing nodes in the graph. In the final stage a skilled annotator reviews the graph for mistakes and and re-annotates it if necessary. There is a visualization of the semantic annotation available, enabling an annotator to verify that they have built the graph in a semantically accurate manner. Manual annotation happens at the rate of 40 per hour. The manually annotated dataset contains \(\sim\)20k annotated utterances and contains 93 unique actions,

Figure 3: (a) shows the first turn of a conversation, _“play songs by modonna”_ (b) shows the second turn of a conversation, _“what’s her address”_. Because the node Singer–0 has the same “-0” ID in both turns, the previous turn can be directly used to infer that the address should be for the person whose name is “Madonna.”

Figure 2: Examples of complex queries. In (a) is the utterance _“find restaurants near the sharks game.”_. In (b) is the utterance _“find red and silver toasters”_. In (c) is _“play charlie brown and turn the volume up to 10”_. In (d) is _“find the wendy’s on 5th avenue beside the park.”_ In (e) and (f) are an illustration composition for, _“play girl from the north country”_ and _“play blue velvet.”_.

 

448 types, 172 properties and 23 roles.

## 4 Parsing

Any graph parsing method can be used to predict AMRL given a natural language utterance. One approach is to use hyperedge replacement grammars Chiang et al. (2013) Peng et al. (2015), though these require large datasets in order to train accurate parsers. Alternatively, the graph can be linearized, as in Gildea et al. (2017) and sequence to sequence or sequential models can be used to predict AMRL Perera and Strubell (2018). We have shown that AMRL full-parse accuracy is at 78%, though the serialization, use of embeddings from related tasks can improve parser accuracy. More details can be found in Perera and Strubell (2018).

## 5 Related Work

FreeBase Bollacker et al. (2008) (now WikiData) and schema.org Guha et al. (2016) are two common ontologies. Schema.org is widely used on the web and contains actions, types and properties. The Alexa ontology expands schema.org to cover types, properties and roles used in spoken language.

Semantic parsing has been investigated in the content of small domain-specific datasets such as GeoQuery Wong and Mooney (2006) and in the context of larger broad-coverage representations such as the Groningen Meaning Bank (GMB) Bos et al. (2017), the Abstract Meaning Representation (AMR) Banarescu et al. (2013), UCCA Abend and Rappoport (2013), PropBank Kingsbury and Palmer (2002), Rainment Baker et al. (1998) and lambda-DCS Kingsbury and Palmer (2002). OntoNotes Hovy et al. (2006), lambda-DCS s Liang (2013) Baker et al. (1998), FrameNet Baker et al. (1998), combinatory categorial grammars CCG Steedman and Baldridge (2011)Hockenmaier and Steedman (2007), universal dependencies Nivre et al. (2016) are all related representations. A comparison of semantic representations for natural language semantics is described in Abend and Rappoport. Unlike these meaning representations for written language, AMRL covers question answering, imperative actions, and a wide range of new types and properties (e.g., smart home, timers, etc.).

AMR and AMRL are both rooted, directed, leaf-labeled and edge-labeled graphs. AMRL does not reuse PropBank frame arguments, covers predicate-argument relations, including a wide variety of semantic roles, modifiers, co-reference, named entities and time expressions Banarescu et al. (2013). There are more than 1000 named-entity types in AMRL (AMR has around 80). Retrancy is not used in AMRL notation. In addition to the AMR "name" property, AMRL contains a "type" property for mentions of a type (or class) and a "value" property for the mention of numeric values. Anaphora is handled in AMRL for spoken dialog Poesio and Artstein Gross et al. (1993). Unlike representations used for spoken language understanding SLU Gupta et al. (2006b), AMRL represents both entity spans, complex natural language expressions, and fine-grained named-entity types.

## 6 Conclusions and Future Work

This paper develops AMRL, a meaning representation for spoken language. We have shown how it can be used to expand the set of supported usecases to complex and cross-domain utterances, while leveraging a single compositional semantics. The representation has been released at AWS Re:Invent 2016 2. It is also being used as a representation for expanded support for complex utterances, such as those with sequential composi

Figure 4: Two examples of conditional statements. In (b) are the annotation for time-based conditions, while in (a) is a non-time based trigger.

 

tion. Continued development of a common meaning representation for spoken language will enable Alexa to become capable and accurate, expanding the set of functionality for all Alexa users.

## References

* Abend and Rappoport (2013) Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In _ACL (1)_. pages 228-238.
* Abend and Rappoport (2017) Omri Abend and Ari Rappoport. 2017. The state of the art in semantic representation. In _Proceedings of the Association for Computational Linguistics_. Vancouver,CA.
* Volume 1_. Association for Computational Linguistics, Stroudsburg, PA, USA, ACL '98, pages 86-90. [https://doi.org/10.3115/980845.980860](https://doi.org/10.3115/980845.980860).
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_. Association for Computational Linguistics, Sofia, Bulgaria, pages 178-186. [http://www.aclweb.org/anthology/W13-2322](http://www.aclweb.org/anthology/W13-2322).
* Berant and Liang (2014) Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing.
* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_. ACM, New York, NY, USA, SIGMOD '08, pages 1247-1250. [https://doi.org/10.1145/1376616.1376746](https://doi.org/10.1145/1376616.1376746).
* Bos et al. (2017) Johan Bos, Valerio Basile, Kilian Evang, Noortije Venhuizen, and Johannes Bjerva. 2017. The groningen meaning bank. In Nancy Ide and James Pustejovsky, editors, _Handbook of Linguistic Annotation_, Springer, volume 2, pages 463-496.
* Chiang et al. (2013) David Chiang, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, Bevan Jones, and Kevin Knight. 2013. Parsing graphs with hyperedge replacement grammars. In _ACL (1)_. pages 924-932.
* Gildea et al. (2017) Daniel Gildea, Nianwen Xue, Xiaochang Peng, and Chuan Wang. 2017. Addressing the data sparsity issue in neural amr parsing. In _EACL_.
* Gross et al. (1993) Derek Gross, James Allen, and David Traum. 1993. The trains 91 dialogues .
* Guha et al. (2016) R. V. Guha, Dan Brickley, and Steve Macbeth. 2016. Schema.org: Evolution of structured data on the web. _Commun. ACM_ 59(2):44-51. [https://doi.org/10.1145/2844544](https://doi.org/10.1145/2844544).
* Gupta et al. (2006a) N. Gupta, G. Tur, D. Hakkani-Tur, S. Bangalone, G. Riccardi, and M. Gilbert. 2006a. The at&t spoken language understanding system. _IEEE Transactions on Audio, Speech, and Language Processing_ 14(1):213-222. [https://doi.org/10.1109/TSA.2005.854085](https://doi.org/10.1109/TSA.2005.854085).
* Gupta et al. (2006b) Narendra Gupta, Gokhan Tur, Dilek Hakkani-Tur, Srinivas Bangalore, Giuseppe Ricciardi, and Mazin Gilbert. 2006b. The AT&T spoken language understanding system. _Audio, Speech, and Language Processing, IEEE Transactions on_ 14(1):213-222.
* Hockennaier and Steedman (2007) Julia Hockennaier and Mark Steedman. 2007. Ccgbank: A corpus of ccg derivations and dependency structures extracted from the penn treebank. _Comput. Linguist._ 33(3):355-396. [https://doi.org/10.1162/coli.2007.33.3.355](https://doi.org/10.1162/coli.2007.33.3.355).
* Hovy et al. (2006) Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In _Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers_. Association for Computational Linguistics, Stroudsburg, PA, USA, NAACL-Short '06, pages 57-60. [http://dl.acm.org/citation.cfm?id=1614049.1614064](http://dl.acm.org/citation.cfm?id=1614049.1614064).
* Kingsbury and Palmer (2002) Paul Kingsbury and Martha Palmer. 2002. From treebank to propbank. In _LREC_. pages 1989-1993.
* Liang (2013) Percy Liang. 2013. Lambda dependency-based compositional semantics. _CoRR_ abs/1309.4408. [http://arxiv.org/abs/1309.4408](http://arxiv.org/abs/1309.4408).
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silviera, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari (Conference Chair), Khalid Choukiri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LHEC 2016)_. European Language Resources Association (ELRA), Paris, France.
* Peng et al. (2015) Xiaochang Peng, Linfeng Song, and Daniel Gildea. 2015. A synchronous hyperedge replacement grammar based approach for amr parsing. In _CoNLL_. pages 32-41 

Vittorio Chung Tagyoung Kollar Thomas Perera and Emma Strubell. 2018. Multi-task learning for parsing the alexa meaning representation language. In _American Association for Artificial Intelligence (AAAI)_.
* Poesio and Artstein [2008] Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the arran corpus. In _LREC_.
* Steedman and Baldridge [2011] Mark Steedman and Jason Baldridge. 2011. _Combinatory Categorial Grammar_, Wiley-Blackwell, pages 181-224. [https://doi.org/10.1002/9781444395037.ch5](https://doi.org/10.1002/9781444395037.ch5).
* Wong and Mooney [2006] Yuk Wah Wong and Raymond J Mooney. 2006. Learning for semantic parsing with statistical machine translation. In _Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics_. Association for Computational Linguistics, pages 439-446.