Learning to Generate Naturalistic Utterances Using
Reviews in Spoken Dialogue Systems
Ryuichiro Higashinaka†
NTT Communication Science Laboratories, NTT Corporation
andMarilyn A. Walker‡
Department of Computer Science, University of Sheﬃeld
andRashmi Prasad§
Institute for Research in Cognitive Science, University of Pennsylvania
Spoken language generation for dialogue systems requires a dictionary of mappings between thesemantic representations of concepts the system wants to express and the realizations of thoseconcepts. Dictionary creation is a costly process; it is currently done by hand for each dialogue
domain. We propose a novel unsupervised method for learning such mappings from user reviewsin the target domain, and test it in the restaurant and hotel domains. Experimental results showthat the acquired mappings achieve high consistency between the semantic representation and therealization, and that the naturalness of the realization is signiﬁcantly higher than the baseline.
Categories and Subject Descriptors: I.2.7 [Artificial Intelligence]: Natural Language Processing—Language generation; I.2.1 [Artificial Intelligence]: Applications and Expert Systems—
Natural Language Interfaces; H.5.2 [Information interfaces and presentation]: User interfaces—Natural Language
General Terms: Languages, Human Factors
Additional Key Words and Phrases: Natural language generation, generation dictionary, user
reviews, spoken dialogue systems

1. INTRODUCTION
Spoken dialogue systems are beginning to achieve some commercial success [Heisterkamp 2001; Pieraccini and Lubensky 2005; Feng et al. 2005], but a remaining
obstacle to their widespread deployment is the cost involved in hand-crafting the
spoken language generation module. Spoken language generation requires a dictionary of mappings between the semantic representations of concepts the system† 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan. rh@cslab.kecl.ntt.co.jp ‡ Regent

Court, 211 Portobello Street, Sheﬃeld S1 4DP, U.K. walker@dcs.shef.ac.uk § 3401 Walnut Street,
Suite 400A Philadelphia, PA 19104-6228, USA. rjprasad@linc.cis.upenn.edu This paper is a
modified and augmented version of our earlier reports [Higashinaka et al. 2005; 2006].
Permission to make digital/hard copy of all or part of this material without fee for personalor classroom use provided that the copies are not made or distributed for proﬁt or commercial
advantage, the ACM copyright/server notice, the title of the publication, and its date appear, andnotice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish,
to post on servers, or to redistribute to lists requires prior speciﬁc permission and/or a fee.
c 20YY ACM 0000-0000/20YY/0000-0001 $5.00

ACM Journal Name, Vol. V, No. N, Month 20YY, Pages 1–23.

2·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad
An example user review (we8there.com)
Food=5, Service=5, Atmosphere=5, Value=5,
Overall=5
Review
The best Spanish food in New York. I am fromcomment
Spain and I had my 28th birthday there and we all
had a great time. Salud!
Ratings↓

Review comment after named entity recognition
The best {NE=foodtype, string=Spanish} {NE=food,string=food, rating=5} in {NE=location, string=New York}.
...

↓

Mapping between a semantic representation (a set of relations) and a syntactic
structure (DSyntS)• Relations:
RESTAURANT has FOODTYPERESTAURANT has foodquality=5
RESTAURANT has LOCATION(foodtype, food=5, location for shorthand.)• DSyntS:32
lexeme : food7
6class : common noun
7
6
7
6number : sg
7
6
7
6article : def
»
–
7
6
7
6
lexeme
:
best
7
6ATTR
7
6
class
:
adjective
6
3 7
2
7
6
lexeme : FOODTYPE
76
66class : common noun7 7
7 7
6ATTR66
5 7
4number : sg
76
76
article:no-art7632
76lexeme : in
7
6
6
77
6class : preposition
7
6
7
6
3
2
7
6
lexeme : LOCATION 7
7
6ATTR6
7
6
7
6
7
6
7
6
6
6II6class : proper noun777
4
4 4number : sg
555
article : no-art

Fig. 1.

Example of procedure for acquiring a generation dictionary mapping.

wants to express and realizations of those concepts [Reiter and Dale 2000; Stentet al. 2004]. Dictionary creation is a costly process: an automatic method for
creating them would make dialogue technology more scalable.
Generation dictionaries can be based on generation templates or syntactic trees,
and rules. Generation templates are widely used in many practical systems for
simplicity [Seneﬀ and Polifroni 2000; Theune 2003; Higashinaka et al. 2006]. They
are composed of pairs of communicative goals/dialogue acts and templates (surface
strings with variables). For example, a dialogue act refer-info-place, which conﬁrms
an information type and a place name in the weather information service domain,
may be mapped to the template “Are you interested in the [info=X] in [place=Y]?”,
where X and Y are provided by the dialogue manager [Higashinaka et al. 2006].
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·

For systems to assert more complex propositions, templates may be extended toincorporate syntactic structures, such as the Deep Syntactic Structure (DSyntS) in
Fig. 1 [Melčuk 1988], so that aggregation, and other syntactic transformations ofutterances, as well as context-speciﬁc prosody assignment, can be realized [Stentet al. 2004; Moore et al. 2004]. Generation rules are also used in systems where
semantic representations have complex or hierarchical structures bearing various
arguments. The generation process works by converting input such as E-forms[Goddeau et al. 1996] or frames [Bobrow et al. 1977] through the application ofrewrite rules in a cascading manner [Baptist and Seneﬀ 2000].
Whether templates or rules are employed, it is widely acknowledged that creating
and maintaining good-quality mappings is a costly process. This is because dialoguesystems, especially those that are task-oriented, have to convey the information
requested by users as accurately as possible, and in a natural enough form forusers to be able to intuitively converse with such systems to access information. To
satisfy these objectives, both signiﬁcant human eﬀort in corpus study and linguistic
expertise have been necessary [Reiter and Dale 2000; Reiter et al. 2003; Reiter andSripada 2002].
Recently, corpus-based approaches have been proposed as one way to reduce theeﬀort involved in developing natural language generators (NLGs). This work isbased on over-generating many candidate utterances and then ranking them, using
either user feedback or corpus models to generate automatic ranking functions[Langkilde-Geary 2002; Oh and Rudnicky 2000; Rambow et al. 2001; Walker et al.
2002; Stent et al. 2004; Chen et al. 2002; Chambers and Allen 2004]. The advantageof this approach is that the candidate utterances can be generated through many
combinations of potential generation procedures, without needing to guarantee thatevery candidate is high quality. However, candidate utterances are still generated
with a hand-crafted dictionary.
To date, little work has been done on automatically acquiring generation dictionaries, which requires that an accurate semantic representation be derived for
sentences in the target domain. The related problem of automatically ﬁnding sentences with the same meaning has been extensively studied in the ﬁeld of automaticparaphrasing. This work typically uses parallel corpora and corpora with multiple
descriptions of the same events to extract interchangeable sentences [Barzilay and
McKeown 2001; Barzilay and Lee 2003]. Other work has found predicates of similar
meanings by using the similarity of contexts around the predicates [Lin and Pantel
2001]. These studies ﬁnd a set of sentences with the same meaning, but do not
associate a speciﬁc meaning with the sentences.
An exception is work by [Barzilay and Lee 2002; Barzilay and Lapata 2006; 2005;
Snyder and Barzilay 2007], which is more in the spirit of our own work. This work
is based on supervised techniques with parallel corpora consisting of both complex
semantic input and corresponding natural language verbalizations in the domainsof mathematical proofs or sports writing. It derives mappings between semantic
representations and realizations, and explores the training of automatic content
selection and aggregation algorithms using these mappings. However, since thistechnique requires parallel corpora or previously existing semantic transcriptions
or labeling, it does not address the problem of the cost involved with dictionary
ACM Journal Name, Vol. V, No. N, Month 20YY.

3

4

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

creation.
Other research has begun to explore automatically obtaining semantic representations corresponding to particular linguistic phrases, but this research has notconsidered whether the learned phrases can be used to generate coherent utterancesfrom the semantic representations, and it has been primarily focused on a small setof semantic relations, such as is-a or part-of, [Pantel and Ravichandran 2004; Gildea
and Jurafsky 2002; Etzioni et al. 2005], although recent work has begun to expand
beyond this small set of relations [Soderland 2007].
However, when we look at the increasing number of language resources available on the web, we notice that some of them have speciﬁc structures that might
be used to facilitate the automatic understanding of the content. For example,
tables and lists marked by words such as “pros” and “cons” can be used to collect positive/negative sentence instances for training sentiment classiﬁers [Kaji andKitsuregawa 2006], and good/bad votes, such as those at amazon.com, have beenfound useful for determining the characteristics of helpful texts [Kim et al. 2006].
Previous work on mining user reviews aim at summarizing reviews so that users canmake decisions easily. This work can ﬁnd adjectives to describe products [Hu andLiu 2005], and automatically ﬁnd features of a product together with the polarity ofadjectives used to describe them [Popescu and Etzioni 2005]. Thus we hypothesizethat it may be possible to use the structured information typically available in user
review websites to induce semantic representations for review sentences.
In this paper, we propose a novel method for mining user reviews on the web toautomatically acquire a domain-speciﬁc generation dictionary for information presentation in a spoken dialogue system, and apply it to user reviews in the restaurant
and hotel domains. Our hypothesis is that reviews that provide individual ratingsfor various distinguished attributes of review entities can be used to map review
sentences to semantic representations. Figure 1 shows a user review in the restaurant domain, where we hypothesize that the user rating food=5 indicates that thesemantic representation for the sentence “The best Spanish food in New York” includes the relation ‘restaurant has foodquality=5.’ In Section 2, we describe ourmethod in detail. Section 3 describes an evaluation experiment based on the application of the method to the two domains. We present results of both objective andsubjective evaluations using the learned dictionaries to generate recommendations
for hotels and restaurants. Section 4 concludes and describes future work.
2. METHOD: LEARNING A GENERATION DICTIONARY
We propose mining user reviews on the web to automatically acquire a domain speciﬁc generation dictionary for information presentation in a spoken dialogue system.
The basic idea is that the ratings given to the reviews indicate the meaning of sentences, making it possible to derive accurate semantic representations for sentencesthat are then used to automatically create or augment a generation dictionary.
The automatically created generation dictionary consists of triples (U, R, S) representing a mapping between the original utterance U in the user review, its semantic representation R(U), and its syntactic structure S(U). Syntactic structures
are derived so that the dictionary can be used in grammar-based full-NLG systems[Stent et al. 2004]. The procedure is outlined brieﬂy in Fig. 1. It comprises the
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·

following steps:(1) Collect user reviews on the web to create a population of utterances U.
(2) To derive semantic representations R(U):—Identify distinguished attributes and construct a domain ontology;—Specify lexicalizations of attributes;—Scrape webpages’ structured data for named-entities;—Tag named-entities.
(3) Derive syntactic representations S(U).
(4) Filter inappropriate mappings.
(5) Add mappings (U, R, S) to dictionary.
In what follows, we describe each step in detail.
2.1 Collecting user reviews
Although there are many websites dealing with user reviews, we select only thosethat have individual ratings for various distinguished attributes of review entities.
We collect user reviews from such websites and store them as a corpus (U). Somewebsites may contain tabular data for review entities, such as names and addressesof restaurants. In such cases, we also store them as additional data.
2.2 Deriving semantic representations
We ﬁrst identify distinguished attributes for each review entity. They include attributes that the users are asked to rate, which have scalar values, and other attributes that can be extracted from the tabular data, which have categorical values.
For example, in the restaurant domain, food, service,atmosphere, value, and overall
are scalar-valued distinguished attributes, and foodtype and location are the distinguished attributes with categorical values. Given the distinguished attributes, a
simple domain ontology can be automatically derived by assuming that a meronymy
relation, represented by the predicate ‘has’, holds between the entity type (e.g.,
restaurant) and the distinguished attributes. Thus, in the restaurant domain,
the following ontology consisting of the seven relations can be derived:⎧
restaurant has foodquality⎪⎪⎪⎪
restaurant has servicequality⎪⎪⎪⎪⎨ restaurant has valuequality
restaurant has atmospherequality⎪⎪
restaurant has overallquality⎪⎪⎪⎪
restaurant has foodtype⎪⎪⎩
restaurant has location
We assume that, although users may discuss other attributes of the entity, atleast some of the utterances in the reviews realize the relations speciﬁed in the
ontology. Our task then is to identify these utterances. We test the hypothesis
that, if an utterance U contains named-entities corresponding to the distinguishedattributes, R for that utterance includes the relation concerning that attribute in
the domain ontology.
ACM Journal Name, Vol. V, No. N, Month 20YY.

5

6

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

We also hypothesize that the rating given for the distinguished attribute speciﬁes
the scalar value of the relation. For example, a sentence containing named-entities
for foodquality (e.g., food or meal. See Tab. I.) is assumed to realize the relation ‘restaurant has foodquality.’, and the value of the foodquality attribute is
assumed to be the value speciﬁed in the user rating for that attribute, e.g., ‘restaurant has foodquality = 5’ in Fig. 1. Similarly, the other relations in Fig. 1 areassumed to be realized by the utterance “The best Spanish food in New York”
because it contains one foodtype named-entity and one location named-entity.
Values of categorical attributes are replaced by variables representing their type
before the learned mappings are added to the dictionary, as shown in Fig. 1.
To detect our distinguished attributes, we prepare a named-entity tagger. Currently, lexicalizations of rating-related distinguished attributes are created byhand,1 and those for other distinguished attributes are automatically imported
from the tabular data. We also prepare lexicalizations of named-entities that are
not relevant to the domain from webpages in order to detect pieces of information irrelevant to the domain. Finally, we augment the named-entity tagger with
our list of lexicalizations, and apply it to the review sentences to derive semantic
representations.
2.3 Parsing and DSyntS conversion
We adopt DSyntSs as a format for syntactic structures because they can be realized
by the fast portable realizer RealPro [Lavoie and Rambow 1997]. Since DSyntSs
are a type of dependency structure, we ﬁrst process the sentences with Minipar [Lin1998], a general-purpose dependency parser, and then convert Minipar’s representations into DSyntSs with a converter we developed. We also apply a POS tagger[Brill 1992] to the sentences in parallel so that the converter can incorporate thePenn Tree POS tag information, which is not in the Minipar output.
The conversion process consists of three parts. The ﬁrst part reads in the dependencies shown in the Minipar output as a tree. Two hand-constructed mappingtables are used. One maps the Minipar dependency arc labels to those used by the
RealPro DSyntS, and the other maps the POS tags to feature structures for the
Realpro DSyntS. This results in the assignment of dependency labels and feature
structures to each node in the tree. The second part uses hand-crafted rules tomodify the derived tree structures for easily identiﬁable inaccuracies in dependencies and feature structures. Finally, the tree is converted to the representation thatis used by the Realpro surface realizer [Lavoie and Rambow 1997].
Since we are processing sentences from user reviews, which are diﬀerent from thenewspaper articles on which Minipar was trained, the output of Minipar for suchsentences can be inaccurate, leading to failure in conversion. We check whether theconversion is successful for each sentence in the ﬁltering stage.
2.4 Filtering
The goal of ﬁltering is to identify U that realizes the distinguished attributes andto guarantee high precision for the learned mappings. Recall is less importantsince systems need to convey requested information as accurately as possible. Our
1 In future, we will investigate other techniques for bootstrapping the lexicalizations.

ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·

procedure for deriving semantic representations is based on the hypothesis that, if
U contains named-entities that realize the distinguished attributes, R will include
the relevant relation in the domain ontology.
We also assume that, if U contains named-entities that are not covered by thedomain ontology or contains words indicating that the meaning of U depends on
the surrounding context, R will not completely characterize the meaning of U, andso U should be eliminated. We also require an accurate S for U. Therefore, theﬁlters described below eliminate U that (1) realizes semantic relations not in the
ontology; (2) contains words indicating that its meaning depends on the context;(3) contains unknown words; or (4) cannot be parsed accurately. The ﬁlters areapplied in a cascading manner.
No Relations Filter. The sentence does not contain any named-entities for the
distinguished attributes.
Other Relations Filter. The sentence contains named-entities that are notcovered by the domain ontology.
Contextual Filter. The sentence contains indexicals, such as I, you, that or cohesive markers of rhetorical relations that connect it to some part of the preceding
text, which means that the sentence cannot be interpreted out of context. These
include discourse markers, such as list item markers with LS as the POS tag, thatsignal the organization structure of the text [Hirschberg and Litman 1987; Prasad
et al. 2005], as well as discourse connectives that signal semantic and pragmatic
relations of the sentence with other parts of the text [Knott 1996], such as coordinating conjunctions at the beginning of the utterance like and and but etc., andconjunct adverbs such as however, also, then.
Unknown Words Filter. The sentence contains words not in WordNet [Fellbaum 1998] (which includes typographical errors), or POS tags contain NN (Noun),
which may indicate an unknown named-entity, or the sentence has more than a ﬁxed
length of words,2 suggesting that its meaning cannot be estimated using only theoccurrence of named entities.
Parsing Filter. The sentence fails the parsing to DSyntS conversion. Failures
are automatically detected by comparing the original sentence with the one realized
by RealPro taking the converted DSyntS as an input.
Duplicate Filter. The triple for the sentence has already been observed.
Finally, we add the triples that survive the ﬁltering process to the dictionary.
3. EXPERIMENT
We ﬁrst create generation dictionaries in the restaurant and hotel domains, andthen evaluate the dictionaries using both objective and subjective criteria.
3.1 Creating generation dictionaries
3.1.1 Restaurant Domain. We collected user reviews from we8there.com(http://www.we8there.com/) and identiﬁed seven distinguished attributes from the2 We experimentally derived 20 as a suitable threshold.

ACM Journal Name, Vol. V, No. N, Month 20YY.

7

8

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

Table I.

Lexicalizations for distinguished attributes in the restaurant domain.

Dist. Attr.

Lexicalizationfood
service
atmospherevalue

food, meal
service, staﬀ, waitstaﬀ, wait staﬀ, server, waiter, waitress
atmosphere, decor, ambience, decoration
value, price, overprice, pricey, expensive, inexpensive,
cheap, aﬀordable, aﬀord
recommend, place, experience, establishment

overall

Table II. Filtering statistics: the number of sentences ﬁltered and retained by each ﬁlter in therestaurant domain.
ﬁlter
ﬁltered retained
Initial # of sentences
No Relations Filter
Other Relations Filter
Contextual Filter
Unknown Words Filter
Parsing Filter
Duplicates Filter

−
7,947
5,351
2,973
1,467
216
61

18,466
10,519
5,168
2,195
728
512
451

ratings and the tabular data; namely, food, service, atmosphere, value, overall, location, and food type. Table I shows the lexicalizations of the distinguished attributes
for the domain ontology in Section 2.2.
Out of 18,466 review sentences, we obtained 451 mappings; 2.4% of all sentenceswere used to create the mappings. Table II shows the number of sentences ﬁlteredand retained by each ﬁlter. Named-entities for food subtypes (e.g., pizza, wine),
person names, country names, dates (e.g., today, tomorrow, Aug. 26th) or prices(e.g., 12 dollars), or POS tag CD for numerals were used by the Other Relations
Filter to detect relations not in the domain ontology.
3.1.2 Hotel Domain. We collected user reviews in the hotel domain from travelocity (http://dest.travelocity.com/Reviews/) and identiﬁed eight distinguished attributes: service, entertainment, sports (sports/activities), overall, facility (public
facilities), room, dining, and location, where location is the only categorical attribute. From the attributes, we created the domain ontology:⎧
hotel has servicequality⎪⎪⎪⎪
hotel has entertainmentquality⎪⎪⎪⎪
hotel has sportsquality⎪⎪⎨
hotel has overallquality
hotel has facilityquality⎪⎪⎪⎪
hotel has roomquality⎪⎪⎪⎪
hotel has diningquality⎪⎪⎩
hotel has location
Table III shows lexicalizations of the rating-related distinguished attributes. Exactly the same ﬁlters were used as those for the restaurant domain, except thatfood subtypes were not detected by the named-entity tagger. Out of 20,723 review
sentences, 536 mappings were obtained; 2.6% of all sentences were used to create
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems

Table III.

·

Lexicalizations for distinguished attributes in the hotel domain.

Dist. Attr.

Lexicalizations

serviceentertainment
sportsoverallfacilityroom
diningservice, staﬀ
entertainment, attraction, amusement, fun, show
sport, activity, workout, exercise, athletics
hotel, overall, recommend, place, experiencefacility, amenity, appliance, equipmentroom
dining, dinner, food, meal

Table IV. Filtering statistics: the number of sentences ﬁltered and retained by each ﬁlter in the
hotel domain.
ﬁlter
ﬁltered retained
Initial # of sentences
No Relations Filter
Other Relations Filter
Contextual Filter
Unknown Words Filter
Parsing Filter
Duplicates Filter−
12,413
1,293
3,881
2,541
257
72

20,723
8,580
7,287
3,406
865
608
536

the mappings (See Table IV for the ﬁltering statistics).
3.2 Objective Evaluation
We evaluated the learned mappings with respect to domain coverage, linguistic
variation and generativity.
3.2.1 Domain Coverage. For the mappings to be useful, they must have gooddomain coverage. Table V shows the distribution of the 327 mappings realizing
a single scalar-valued relation, categorized by the associated rating score in therestaurant domain.3 For example, there are 57 mappings with R of ‘restaurant
has foodquality=5,’ and a large number of mappings for both the foodquality andservicequality relations. Although we could not obtain mappings for some relations,such as price={1,2}, coverage for expressing a single relation is fairly complete;domain coverage is 88% (22/25). When we look into the corpus, we ﬁnd that there
are quite a few sentences associated with the ratings price={1,2}, with 2,449 and
1,762 sentences, respectively. Therefore, the lack of mappings for these ratingssuggests that reviewers rarely mention such relations, or that our lexicalizations forvaluequality are not suﬃcient.
There are also mappings that express several relations. Table VI shows the countsof mappings for multi-relation mappings, with those containing a food or service
relation occurring more frequently as in the single scalar-valued relation mappings.
We found only 21 combinations of relations, which is surprising given the large
potential number of combinations (There are 50 combinations if we treat relations
with diﬀerent scalar values diﬀerently). We also ﬁnd that most of the mappings
3 There are two other single-relation but not scalar-valued mappings that concern location in our

mappings.
ACM Journal Name, Vol. V, No. N, Month 20YY.

9

10

·

Table V.

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

Domain coverage of single scalar-valued relation mappings in the restaurant domain.
RatingDist.Attr.

Table VI.

1

2

3

4

5

Total

food
service
atmospherevalue
overall

5
15
0
0
3

8
3
3
0
2

6
6
3
1
5

18
17
8
8
15

57
56
31
12
45

94
97
45
21
70

Total

23

15

21

64

201

327

Counts for multi-relation mappings in the restaurant domain.
#

Combination of Dist. Attrs

Count

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

food-service
food-value
atmosphere-foodatmosphere-service
atmosphere-food-service
food-foodtype
atmosphere-food-value
location-overall
food-foodtype-value
food-service-value
food-foodtype-location
food-overall
atmosphere-foodtype
atmosphere-overall
service-value
overall-service
overall-value
foodtype-overall
food-foodtype-location-overall
atmosphere-food-service-value
atmosphere-food-overallservice-value

39
21
14
10
7
4
4
3
3
2
2
2
2
2
1
1
1
1
1
1
1

Total

122

have just two or three relations, perhaps suggesting that system utterances should
not express too many relations in a single sentence.
In the hotel domain, we found 58 distinct semantic representation patterns, including 24 single-relation patterns (23 rating-related relations and one for location)and 16 multi-relation patterns (34 combinations of relations when scalar values are
considered). Table VII shows the distribution of single-relation mappings categorized by the associated rating score and Table VIII shows the counts for multirelation mappings (There are six other mappings that concern location). It isnoticeable that very few mappings for the entertainment and sports distinguishedattributes were induced, making the domain coverage rather low at 63% (23/35),
which means some additional manual mappings would be necessary for systems tobe able to generate utterances in this domain. However, when we exclude entertainment/sports, the domain coverage is as good as that in the restaurant domain.
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems

Table VII.

·

Domain coverage of single scalar-valued relation mappings in the hotel domain.
RatingDist.Attr.

1

2

3

4

5

Total

serviceentertainment
sportsoverallfacilityroom
dining

7
1
0
22
1
18
1

4
0
0
14
1
12
0

4
0
0
20
0
12
0

26
0
0
72
2
25
4

70
0
0
84
1
55
3

111
1
0
212
5
122
8

Total

50

31

36

129

213

459

Table VIII.

Counts for multi-relation mappings in the hotel domain.
#

Combination of Dist. Attrs

Count

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

room-service
overall-service
location-overall
overall-room
overall-room-service
dining-service
facility-room
dining-room-service
service-sportsroom-sports
location-service
facility-sports
facility-overall
dining-room
dining-overall
dining-facility-room-service

24
20
5
4
3
3
2
2
1
1
1
1
1
1
1
1

Total

71

An analysis of the corpus revealed that this sparseness comes from reviewers’ tendency to mention sub-concepts of entertainment/sports, such as casinos, pools, andbeaches; rarely did they mention entertainment or sports quality as a whole.
3.2.2 Linguistic Variation. We also wish to assess whether the linguistic variation of the learned mappings is greater than what we could easily have generated
with a hand-crafted dictionary or with a hand-crafted dictionary augmented withaggregation operators, as in [Stent et al. 2004]. Thus, we ﬁrst categorized themappings by the patterns of the DSyntSs. Table IX shows the ﬁve most common
syntactic patterns in both domains, indicating that 25–30% of the learned patterns
consist of the simple form “x is adj,” where adj is an adjective, or “x is rb adj,”
where rb is a degree modiﬁer. In fact, approximately half of the learned mappings (50.3% and 50.0% in the restaurant and hotel domains, respectively) could
be generated from these basic patterns by applying a combination operator thatcoordinates multiple adjectives or coordinates predications over distinct attributes.
However, there are 137 syntactic patterns in the restaurant domain (97 withunique syntactic structures and 21 with two occurrences), and there are 142 synACM Journal Name, Vol. V, No. N, Month 20YY.

11

12

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

Table IX. Common syntactic patterns of DSyntSs in the restaurant and hotel domains, ﬂattened
to a POS sequence for readability. NN, VB, JJ, RB, CC stand for noun, verb, adjective, adverb,
and conjunction, respectively.
rank
1
2
3
4
5

Restaurant Domain
syntactic patternratio
NN VB JJNN VB RB JJ
JJ NN
NN VB JJ CC JJ
RB JJ NN

20.4%
11.5%
8.0%
5.5%
4.9%

Hotel Domain
syntactic patternratio

NN VB JJNN VB RB JJ
JJ NN
NN VB JJ CC JJ
NN VB RB JJ CC JJ

15.9%
9.5%
7.6%
5.9%
5.4%

tactic patterns in the hotel domain (96 with unique syntactic structures and 9with two occurrences). Figure 2 shows examples of learned mappings with distinct syntactic structures. It would be surprising to see this type of variety in a
hand-crafted generation dictionary. In addition, the learned mappings contain 275
distinct lexemes, with a minimum of 2, maximum of 15, and mean of 4.63 lexemes
per DSyntS in the restaurant domain, and 254 distinct lexemes, with a minimumof 2, maximum of 15, and mean of 4.56 lexemes per DSyntS in the hotel domain,indicating that the method extracts a wide variety of expressions of varying lengths.
When we merge the mappings of the two domains, we ﬁnd 229 distinct syntactic
patterns and 406 distinct lexemes, which shows the relative syntactical and lexical
diﬀerences between the two domains.
Another interesting aspect of the learned mappings is the wide variety of adjectival phrases (APs). Tables X and XI show the APs in single scalar-valued
relation mappings for food and room categorized by the associated ratings in therestaurant domain. Tables for other scalar-valued attributes can be found in the
Appendix. The bold/italicized fonts indicate whether the adjectives are registeredin the General Inquirer lexicon (http://www.wjh.harvard.edu/˜inquirer/) as positive (bold) or negative (italicized) words. Since the General Inquirer lexicon covers
domain-independent positive/negative adjectives, adjectives not in bold/italicized
font can be considered domain-speciﬁc ones. Since we observe many such adjectives, this indicates that our approach can make systems speak with a vocabulary
suited to the domain.
Moreover, the meanings for some of the learned APs are very speciﬁc to theparticular attribute, e.g., cold and burnt associated with foodquality of 1, attentive
and prompt for servicequality of 5, silly and inattentive for servicequality of 1. andmellow for atmosphere of 5. In addition, our method places the APs on a more
ﬁne-grained scale of 1 to 5, similar to the strength classiﬁcations in [Wilson et al.
2004], in contrast to other automatic methods that classify expressions into a binary
positive or negative polarity (e.g., [Turney 2002]).
3.2.3 Generativity. Our motivation for deriving syntactic representations forthe learned expressions was to explore the possibility of using an oﬀ-the-shelf sentence planner to derive new combinations of relations, and apply aggregation andother syntactic transformations. We examined how many of the learned DSyntSs
can be combined with each other, by taking every pair of DSyntSs in the mappings
and applying the built-in merge operation in the SPaRKy generator [Stent et al.
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·• Restaurant Domain—Very disappointing experience for the money charged. (overall=1, value=2)—The food is excellent and plentiful at a reasonable price. (food=5, value=5)—The food is exquisite as well as the service and setting. (food=5, service=5)—The food was spectacular and so was the service. (food=5, foodtype, value=5)—Best foodtype food with a great value for money. (food=5, foodtype, value=5)—This is the best place to eat foodtype food in location. (food=5, foodtype)—Simply amazing foodtype food. (food=5, foodtype)—restaurantname is the best of the best for foodtype food. (food=5)—The food is to die for. (food=5)—What incredible food. (food=5)—Very pleasantly surprised by the food. (food=4)—The food has gone downhill. (food=1)—This is a quiet little place with great atmosphere. (atmosphere=5, overall=5)• Hotel Domain—Great service by a friendly staﬀ. (service=5)—The staﬀ was unfriendly, the rooms were poorly kept and worn. (room=1, service=1)—The rooms were dirty old and smelled. (room=1)—The hotel is fairly new and so the rooms are nice and clean. (overall=5, room=5)—Great hotel to stay in. (overall=5)—Great place for an aﬀordable place to sleep. (overall=4)—The rooms are just average hotel rooms. (overall=2, room=3)—Just a place to sleep. (overall=2)—This hotel looks old and is old. (overall=2)—This is the best hotel to stay in location. (location, overall=5)—The entire facility was extremely dated. (facility=2)—There was NO entertainment. (entertainment=1)—Food was great along with service. (dining=5, service=4)Fig. 2. Acquired generation patterns (with shorthand for relations in brackets) whose syntactic
patterns occurred only once.
Table X.

Adjectival phrases (APs) in single scalar-valued relation mappings for foodquality.

food=1
food=2
food=3burnt, very ordinary, awful, bad, cold
ﬂavored, not enough, very good, very bland, acceptable, bad
ﬂavorful but cold, rather bland, very good, adequate, pretty good, bland andmediocreabsolutely wonderful, awesome, rather good, really good, very very good, very
fresh and tasty, very good, decent, excellent, good, good and generous,
great, outstanding, traditional
absolutely delicious, absolutely fantastic, absolutely great, absolutely terriﬁc,awesome, delectable and plentiful, delicious, delicious but simple, large and satisfying, quick and hot, simply great, so delicious, so very tasty, superb, very good,
ample, well seasoned and hot, best, excellent, exquisite, fabulous, fancy but
tasty, fantastic, fresh, good, great, just fantastic, outstanding, plentiful
and tasty, plentiful and outstanding, terriﬁc, tremendous, wonderful, hot,incredible

food=4
food=52004]. We found that only 306 combinations out of a potential 81,318 (0.37%),
and 544 combinations out of a potential 138,422 (0.39%) were successful in therestaurant and hotel domains, respectively. This is because the merge operation
in SPaRKy requires that the subjects and the verbs of the two DSyntSs be identical, e.g., the subject is restaurant (or hotel) and the verb is has, whereas thelearned DSyntSs often place the attribute in the subject position as a deﬁnite nounphrase.
However, the learned DSyntS can be incorporated into SPaRKy. The SPaRKy
ACM Journal Name, Vol. V, No. N, Month 20YY.

13

14

·

Table XI.
room=1room=2
room=3
room=4

room=5

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

Adjectival phrases (APs) in single scalar-valued relation mappings for roomquality.
disgusting, gross and nasty, old, old and dirty, particularly clean or comfortable,
really gross, small, very dirty, very nasty, whole, awful, dirty, horrible, nasty, terrible
average, old, old and dirty, outdated and dirty, really moldy and stale, really old
and ragged, very clean, staleaverage, very quiet, very basic, very clean, comfortable, decent, nice, blandhuge, large, new, public, so clean, spacious, understated and warm, very clean,very clean and pleasant, very comfortable, very nice, bright, clean, comfortable and spacious, clean, clean and comfortable, clean and neat, clean andnice, great, perfect, spotless
beautiful, beautiful, clean and spacious, beautiful, spotless and comfortable,
extremely large, huge, impeccable, large, large and very comfortable, large, clean
and quiet, new, newly renovated, quite spacious, renovated, so spacious and luxurious, spacious, spacious, comfortable and well appointed, very large, very
nicely decorated, very spacious, very spacious and beautiful, very clean, brightand attractive, very clean, very clean and comfortable, very clean and nice,very comfortable, very cozy, very nice, cheerful and unique, clean, clean andvery nice, clean and very reasonable, clean and comfortable, clean and nice,comfortable, elegant, extraordinary, fabulous, fantastic, great, nice, niceand spacious, spotless, wonderful• Content Plan
—RST relations: justify(p1, p2), justify(p1, p3), justify(p1, p4)—propositions:p1. assert-best(Babbo) ⇔ <no corresponding semantic representation>
p2. assert-food quality(Babbo, superb) ⇔ food=5
p3. assert-service(Babbo, excellent) ⇔ service=5
p4. assert-decor(Babbo, superb) ⇔ atmosphere=5
Original SPaRKy utterance: Babbo has excellent decor and superb food quality withexcellent service. It has the best overall quality among the selected restaurants.
With learned mappings:
u1. Babbo has superb food quality, the service is exceptional and the atmosphere isvery creative. It has the best overall quality among the selected restaurants.
u2. The food is phenomenal and the atmosphere is very unique. Babbo has excellent
service. It has the best overall quality among the selected restaurants.
u3. Great food and small cozy atmosphere. Babbo has excellent service. It has thebest overall quality among the selected restaurants.
u4. Since the service is fast and friendly, the food is really good and Babbo has excellent
decor, it has the best overall quality among the selected restaurants.
Fig. 3. Example utterances for the restaurant domain incorporating learned DSyntSs in SPaRKy.
The symbol ’⇔’ indicates how a SPaRKy proposition is mapped to a semantic representation (inshorthand). The infer relation is equal to the joint relation in RST.

generator takes as input a set of propositions and the rhetorical relations among
them. Each proposition has a corresponding DSyntS (assigned by hand), and theDSyntSs for propositions are aggregated by Rhetorical Structure Theory (RST)transformation rules to create a sentence plan tree satisfying the relations. We cansubstitute the DSyntSs for the propositions with the ones in the learned mappings in
the aggregation process. We prepared several rules to map SPaRKy propositions to
our semantic representations so as to enable the substitution. Figure 3 and Figure 4
show how an utterance would change by incorporating the learned mappings in eachdomain. The hand-crafted mappings used for comparison as the original SPaRKyutterance are also used in the subjective evaluation (See Section 3.3). The resulting
utterances seem more natural and colloquial; the subjective evaluation examines
whether this is true.
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·• Content Plan
—RST relations: infer(p1, p2, p3, p4, p5, p6, p7, p8)—propositions:p1. assert-service(Bellagio, excellent) ⇔ service=5
p2. assert-entertainment(Bellagio, excellent) ⇔ entertainment=5
p3. assert-sports(Bellagio, excellent) ⇔ sports=5
p4. assert-overall(Bellagio, excellent) ⇔ overall=5
p5. assert-facility(Bellagio, excellent) ⇔ facility=5
p6. assert-room(Bellagio, excellent) ⇔ room=5
p7. assert-dining(Bellagio, excellent) ⇔ dining=5
p8. assert-nbhd(Bellagio, Las Vegas) ⇔ location
Original SPaRKy utterance: Bellagio is located in Las Vegas and has excellent dining
quality. It is excellent for sports and activities. Bellagio, which has excellent rooms andexcellent overall quality, has excellent facilities. It has excellent entertainment quality.
It has excellent service.
With learned mappings:
u1. Really all fairly good for Las Vegas, Bellagio is excellent for sports and activities andthe staﬀ is super friendly and extremely attentive. Bellagio has excellent entertainment
quality and the facility is clean and comfortable. Great food and the room is very large.
The hotel is magical.
u2. Very nicely decorated rooms. Not bad for Las Vegas. Bellagio is excellent for sports
and activities and the facility is clean and comfortable. Regardless, great service, thefood is excellent and the hotel is great. Bellagio has excellent entertainment quality.
u3. The room is great. The staﬀ is helpful and courteous and Bellagio has excellent
entertainment quality. Las Vegas is just beautiful and the facility is clean and comfortable. The hotel is wonderful, Bellagio is excellent for sports and activities and the foodis excellent.
u4. Las Vegas is just beautiful and the whole experience is awesome. Very nicely
decorated rooms and Bellagio has excellent facilities. The food is delicious. Bellagio
is excellent for sports and activities, the staﬀ is very pleasant and accommodating andhas excellent entertainment quality.
Fig. 4. Example utterances incorporating learned DSyntSs in SPaRKy. The symbol ’⇔’ indicates
how a SPaRKy proposition is mapped to our semantic representation (in shorthand). The infer
relation is equal to the joint relation in RST.

3.3 Subjective Evaluation
We also wish to evaluate the learned dictionary in the context of our application:
generating evaluative utterances (recommendations and comparisons) in the restaurant and hotel domains. A primary concern is whether the precision of our procedure for deriving semantic representations is high enough. A secondary concern
is whether the mappings are natural and appropriate for spoken language generation. Thus, we evaluate the obtained mappings in two respects: (i) the consistency
between the semantic representation and the realization and (ii) the naturalness of
the realization. In the evaluation, we compare hand-crafted mappings (a baseline)with the learned mappings. Figure 3 and Figure 4 show sample recommendations
in both the restaurant and hotel domains. Each ﬁgure shows the content plan withthe propositions realized, an original SPaRKy utterance (one of several) and sample
utterances incorporating the learned mappings for the same content plan.
In the restaurant domain, the baseline mappings were those from [Stent et al.
2004] except that we changed the word decor to atmosphere and added ﬁve mappingsfor overall. For scalar relations, this consists of the realization “restaurant has
ACM Journal Name, Vol. V, No. N, Month 20YY.

15

16

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad
serviceentertainment
sportsoverallfacilityroom
dininglocation

⇔
⇔
⇔
⇔
⇔
⇔
⇔
⇔

hotelname has adj service.
hotelname has adj entertainment quality.
hotelname is adj for sports and activities.
hotelname has adj overall quality.
hotelname has adj facilities.
hotelname has adj rooms.
hotelname has adj dining quality.
hotelname is located in location.

Fig. 5. Hand-crafted mappings in the hotel domain. adj is mediocre, decent, good, very good, or
excellent depending on the rating values 1-5.

adj lex,” where adj is mediocre, decent, good, very good, or excellent for rating values 1-5 and lex is food quality, service, atmosphere, value, or overall depending on
the relation. restaurant is ﬁlled with the name of a restaurant at runtime. For example, ‘restaurant has foodquality=1’ is realized as “restaurant has mediocre
food quality.” The location and food type relations are mapped to “restaurant islocated in location” and “restaurant is a foodtype restaurant.” In the hotel
domain, we created new hand-crafted mappings (Fig. 5).
In the restaurant domain, the learned mappings include 23 distinct semantic representations for a single-relation (22 for scalar-valued relations and one for location)and 50 for multi-relations. Therefore, using the hand-crafted mappings, we ﬁrst created 23 utterances for the single-relations. We then created three utterances foreach of 50 multi-relations using diﬀerent clause-combining operations from [Stentet al. 2004]. This gave a total of 173 baseline utterances, which together with 451
learned mappings, yielded 624 utterances for evaluation. In the same way, we created 662 [536 (# of learned mappings) + 24 (# of single-relation mappings) + 34(# of multi-relation mappings) × 3] utterances in the hotel domain.
Ten subjects and ﬁve subjects, all native English speakers, evaluated the mappings in the restaurant and hotel domains, respectively. They were presented withpairs comprising a semantic representation (in shorthand) and a realization (utterance) for all learned and baseline mappings, which were randomly ordered. The
subjects were asked to express their degree of agreement, on a scale of 1 (lowest)
to 5 (highest), with the statement (a) The meaning of the utterance is consistent
with the ratings expressing their semantics and with the statement (b) The style ofthe utterance is very natural and colloquial. They were asked not to correct theirdecisions and also to rate each utterance on its own merit.
3.3.1 Results. Tables XII and XIII show the means and standard deviations of
the scores for baseline vs. learned utterances for consistency and naturalness in therestaurant and hotel domains, respectively. In both domains, a t-test shows thatthe consistency of the learned mappings is signiﬁcantly lower than the baseline andthat their naturalness is signiﬁcantly higher than the baseline.
However, consistency is still high. Table XIV shows the distribution of meanconsistency/naturalness scores of the learned mappings, which indicates that
only a very small proportion of the learned utterances have a mean consistency/naturalness score lower than 3: Although there is some ﬂuctuation in thequality of mappings across domains, by and large, the human judges felt that theinferred semantic representations were consistent with the meaning of the learned
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·

Table XII. Consistency and naturalness scores averaged over 10 subjects in the restaurant domain.

Consistency
Naturalness

baselinemean
sd.

learnedmean
sd.

statistical signiﬁcanceby Welch’s t-test

4.714
4.227

4.459
4.613

+ (df=4712, p < .001)+ (df=3107, p < .001)

0.588
0.852

0.890
0.844

Table XIII. Consistency and naturalness scores averaged over ﬁve subjects in the hotel domain.

Consistency
Naturalness

baselinemean
sd.

learnedmean
sd.

statistical signiﬁcanceby Welch’s t-test.

4.533
3.603

4.176
3.948

+ (df=975, p < .001)+ (df=977, p < .001)

0.906
1.062

0.942
1.108

expressions and that they were natural. The correlation coeﬃcient between consistency and naturalness scores is 0.42 and 0.33 in the restaurant and hotel domain,
respectively, indicating that consistency is not strongly related to naturalness.
We also performed an ANOVA (ANalysis Of VAriance) of the eﬀect of eachrelation in R on naturalness and consistency. In the restaurant domain, there wereno signiﬁcant eﬀects except that mappings combining food, service, and atmosphere
were signiﬁcantly worse for naturalness (df=1, F=7.79, p=0.005). However, there
is a tendency for mappings to be rated higher for the food attribute (df=1, F=3.14,
p=0.08) and the value attribute (df=1, F=3.55, p=0.06) for consistency.
In the hotel domain, mappings expressing overall were signiﬁcantly worse fornaturalness (df=1, F=7.32, p=0.007). For consistency, attributes sports (df=1,
F=13.15, p=0.0003), room (df=1, F=4.31, p=0.04), and combinations of service and room (df=1, F=4.19, p=0.04), and overall and dining (df=1, F=6.24,
p=0.004) had negative eﬀects. On the other hand, mappings expressing service(df=1, F=10.85, p=0.001), and those combining service, overall and room (df=1,
F=8.23, p=0.004) were rated signiﬁcantly higher. By and large, some attributes
seem to be more diﬃcult to learn than others, perhaps because of the diﬀerence inthe quality of lexicalizations of distinguished attributes and the choice of ratings ofthe review websites.
4. CONCLUSION AND FUTURE WORK
We proposed automatically obtaining mappings between semantic representations
and realizations from reviews with individual ratings. Experimental results showthat (1) the learned mappings provide good coverage of the domain ontology andexhibit good linguistic variation; (2) the consistency between the semantic representations and realizations is high; (3) the naturalness of the realizations is signiﬁcantly
higher than the baseline; and (4) although there are some diﬀerences in the quality of mappings across domains, the method is likely to be applicable to multiple
domains.
One limitation of the work is with the evaluation method. The evaluation in
the restaurant domain was based on a comparison with a previously developed
generation dictionary, which was previously evaluated independently and shown to
produce high quality output [Stent et al. 2004]. However, we had no such preACM Journal Name, Vol. V, No. N, Month 20YY.

17

18

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

Table XIV.

Distribution of mean consistency/naturalness scores of the learned mappings.
mean score (s)range
1.0 ≤ s < 1.5
1.5 ≤ s < 2.0
2.0 ≤ s < 2.5
2.5 ≤ s < 3.0
3.0 ≤ s < 3.5
3.5 ≤ s < 4.0
4.0 ≤ s < 4.5
4.5 ≤ s ≤ 5.0

Restaurant domain
cons.
nat.
2
3
2
7
21
41
92
283

2
1
3
1
15
24
62
343

Hotel domain
cons.
nat.
2
5
11
16
53
59
174
216

3
8
15
28
73
77
197
135

existing dictionary in the hotel domain, and therefore created it ourselves for the
purposes of evaluation. Thus one limitation of the subjective evaluation is that
there could be better, more natural realizations in the hotel domain, which we did
not explore.
There are also limitations of our method. Even though consistency is rated
highly by human subjects, this is a judgment of whether the polarity of the learned
mapping is correctly placed on the 1 to 5 rating scale. Thus, alternate ways of
expressing, for example foodquality=5, shown in Table X, cannot be guaranteed tobe synonymous, which is a requirement for use in spoken language generation. An
examination of the adjectival phrases in Table X shows that diﬀerent aspects ofthe food are discussed. For example, ample and plentiful refer to the portion size,
fancy may refer to the presentation, and delicious describes ﬂavors. One solution
would be to automatically extend the ontology to represent these sub-attributesof the food attribute, and sub-attributes in general. Another solution would be
to use frequency information or other information to ﬁnd the most general terms
which are likely to refer to overall food quality rather than a sub-attribute [Lapata
and Keller 2005]. We believe this could be done automatically in future work. We
have made no use of frequency information as yet, but much related work bases
judgments of syntax/semantics mapping on frequency information [Turney 2002;
Soderland 2007; Etzioni et al. 2005].
Another problem with consistency is that the same AP, e.g., very good in Table
X, may appear with multiple ratings. For example, very good is used for every
foodquality rating from 2 to 5. This may reﬂect a diﬀerence in the use of language
by individuals, related to ideolect or personality [Reiter and Sripada 2002; Mairesse
and Walker 2007]. One solution to this would be to use frequency information todecide which rating was the most accurate, or to eliminate adjectives that appear
with more than one rating. However, even without further extensions, the method
presented here could reduce the amount of time a system designer spends developing the spoken language generator and could increase the naturalness of spoken
language generation.
Another issue is that the recall appears to be quite low given that all of thesentences concern the same domain: only around 2.5% of the sentences could be
used to create the mappings. One way to increase recall might be to automaticallyaugment the list of distinguished attribute lexicalizations, using WordNet or workon automatic identiﬁcation of synonyms, such as [Lin and Pantel 2001]. However,
ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·

the method here has high precision, and automatic techniques may introduce noise.
In addition, recall is greatly reduced by the parsing/generation ﬁlter which veriﬁes
that the automatically derived syntactic representation (DSyntS) can be used toregenerate the exact string. Thus the utility of syntactic structures in the mappings
should be further examined, especially given the failures in DSyntS conversion. An
alternative would be to leave some sentences unparsed and use them as templateswith hybrid generation techniques [White and Caldwell 1998].
A related recall issue is that the ﬁlters are in some cases too strict. For example,
the contextual ﬁlter is based on POS-tags, so that sentences that do not require theprior context for their interpretation are eliminated, such as those containing subordinating conjunctions like because, when, if, whose arguments are both given in
the same sentence [Prasad et al. 2005]. In addition, recall is aﬀected by the domain
ontology, and the automatically constructed domain ontology from the review webpages may not completely cover the domain. In some review domains, the attributes
that get individual ratings are a limited subset of the domain ontology. Techniques
for automatic feature identiﬁcation [Hu and Liu 2005; Popescu and Etzioni 2005]
could help here, although these techniques currently do not automatically identify
diﬀerent lexicalizations of the same feature.
A diﬀerent type of limitation is that dialogue systems need to generate utterances
for information gathering, whereas the mappings we obtained can only be used
for information presentation. Thus, mappings for this purpose would have to beconstructed by hand, as in current practice, or perhaps other types of corpora or
resources could be utilized.
Finally, the comparison of evaluation results in the two domains suggested thatsome distinguished attributes are not necessarily expressed by their straightforward lexicalizations: sports and entertainment are usually expressed by their subconcepts, making the domain coverage low. This provides another motivation for
considering how to automatically extend the domain ontology to better suit thedomain. The diﬀerence in the quality of the mappings between the domains may
also indicate need for further reﬁnement of the method. Although we did not ﬁndoutstanding faults with the mappings in the hotel domain, we need to investigate
why they received relatively low scores.
Despite the limitations, the results of our experiments suggest that our approachis promising. We would like to pursue ways to overcome the limitations to further
facilitate the development of a generation module of spoken dialogue systems.
ACKNOWLEDGMENTS

This work was supported by a Royal Society Wolfson award to Marilyn Walker anda research collaboration grant from NTT to the Cognitive Systems Group at the
University of Sheﬃeld.
REFERENCES
Baptist, L. and Seneff, S. 2000. Genesis-ii: A versatile system for language generation inconversational system applications. In Proc. ICSLP. Vol. 3. 271–274.
Barzilay, R. and Lapata, M. 2005. Collective content selection for concept-to-text generation.
In Proc. HLT/EMNLP. 331–338.

ACM Journal Name, Vol. V, No. N, Month 20YY.

19

20

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

Barzilay, R. and Lapata, M. 2006. Aggregation via set partitioning for natural language generation. In Proc. HLT/NAACL. 359–366.
Barzilay, R. and Lee, L. 2002. Bootstrapping lexical choice via multiple-sequence alignment.
In Proc. EMNLP. 164–171.
Barzilay, R. and Lee, L. 2003. Learning to paraphrase: An unsupervised approach using
multiple-sequence alignment. In Proc. HLT/NAACL. 16–23.
Barzilay, R. and McKeown, K. 2001. Extracting paraphrases from a parallel corpus. In Proc.
39th ACL. 50–57.
Bobrow, D. G., Kaplan, R. M., Kay, M., Norman, D. A., Thompson, H., and Winograd, T.
1977. GUS, a frame driven dialog system. Artif. Intel. 8, 155–173.
Brill, E. 1992. A simple rule-based part-ofspeech tagger. In Proc. ANLP. 152–155.
Chambers, N. and Allen, J. 2004. Stochastic language generation in a dialogue system: Toward
a domain independent generator. In Proc. SIGDIAL. 9–18.
Chen, J., Bangalore, S., Rambow, O., and Walker, M. 2002. Towards automatic generationof natural language generation systems. In Proc. 19th COLING. 1–7.
Etzioni, O., Cafarella, M., Downey, D., Popescu, A.-M., Shaked, T., Soderland, S., Weld,
D. S., , and Yates, A. 2005. Unsupervised named-entity extraction from the web: An experimental study. Artif. Intel. 165, 1, 91–134.
Fellbaum, C. 1998. WordNet: An Electronic Lexical Database (Language, Speech, and Communication). The MIT Press.
Feng, J., Reddy, S., and Saraçlar, M. 2005. Webtalk: mining websites for interactively answering questions. In Proc. Eurospeech. 2485–2488.
Gildea, D. and Jurafsky, D. 2002. Automatic labeling of semantic roles. Comp. Ling. 28, 3,
245–288.
Goddeau, D., Meng, H., Polifroni, J., Seneff, S., and Busayapongchai, S. 1996. A formbased dialogue manager for spoken language applications. In Proc. ICSLP. Vol. 2. 701–704.
Heisterkamp, P. 2001. Linguatronic: Product-level speech system for mercedes-benz car. In
Proc. HLT. 1–2.
Higashinaka, R., Prasad, R., and Walker, M. 2005. Augmenting variation of system utterances
using corpora in spoken dialogue systems. In Proc. 2005 IEEE Automatic Speech Recognition
and Understanding Workshop. 262–267.
Higashinaka, R., Prasad, R., and Walker, M. 2006. Learning to generate naturalistic utterances using reviews in spoken dialogue systems. In Proc. COLING/ACL. 265–272.
Higashinaka, R., Sudoh, K., and Nakano, M. 2006. Incorporating discourse features into conﬁdence scoring of intention recognition results in spoken dialogue systems. Speech Comm. 48, 3–4,
417–436.
Hirschberg, J. and Litman, D. J. 1987. Now let’s talk about NOW: Identifying cue phrases
intonationally. In Proc. 25th ACL. 163–171.
Hu, M. and Liu, B. 2005. Mining and summarizing customer reviews. In Proc. KDD. 168–177.
Kaji, N. and Kitsuregawa, M. 2006. Automatic construction of polarity-tagged corpus fromhtml documents. In Proc. COLING/ACL. 452–459.
Kim, S.-M., Pantel, P., Chklovski, T., and Pennacchiotti, M. 2006. Automatically assessing
review helpfulness. In Proc. EMNLP. 423–430.
Knott, A. 1996. A Data-Driven Methodology for Motivating a Set of Coherence Relations. Ph.D.
thesis, University of Edinburgh, Edinburgh.
Langkilde-Geary, I. 2002. An empirical veriﬁcation of coverage and correctness for a generalpurpose sentence generator. In International Natural Language Generation Conference(INLG). 17–24.
Lapata, M. and Keller, F. 2005. Web-based models for natural language processing. ACM
Transactions on Speech and Language Processing 2, 1, 1–31.
Lavoie, B. and Rambow, O. 1997. A fast and portable realizer for text generation systems. In
Proc. 5th Applied NLP. 265–268.

ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·

Lin, D. 1998. Dependency-based evaluation of MINIPAR. In Workshop on the Evaluation of
Parsing Systems.
Lin, D. and Pantel, P. 2001. Discovery of inference rules for question answering. Natural
Language Engineering 7, 4, 343–360.
Mairesse, F. and Walker, M. 2007. Personage: Personality generation for dialogue. In ACL
Annual Meeting. to appear.
Melčuk, I. A. 1988. Dependency Syntax: Theory and Practice. SUNY, Albany, New York.
Moore, J. D., Foster, M. E., Lemon, O., and White, M. 2004. Generating tailored, comparativedescriptions in spoken dialogue. In Proc. 7th FLAIR.
Oh, A. and Rudnicky, A. 2000. Stochastic language generation for spoken dialogue systems. In
Proceedings of the NAACL-ANLP 2000 Workshop on Conversational Systems. 27–32.
Pantel, P. and Ravichandran, D. 2004. Automatically labeling semantic classes. In Proc.
HLT/NAACL. 321–328.
Pieraccini, R. and Lubensky, D. 2005. Spoken language communication with machines: Thelong and winding road from research to business. In Proc. IEA/AIE. 6–15.
Popescu, A.-M. and Etzioni, O. 2005. Extracting product features and opinions from reviews.
In Proc. HLT/EMNLP. 339–346.
Prasad, R., Joshi, A., Dinesh, N., Lee, A., Miltsakaki, E., and Webber, B. 2005. The Penn
Discourse TreeBank as a resource for natural language generation. In Proc. Corpus Linguistics
Workshop on Using Corpora for NLG.
Rambow, O., Rogati, M., and Walker, M. 2001. Evaluating a trainable sentence planner for a
spoken dialogue travel system. In Proc. 39th ACL. 426–433.
Reiter, E. and Dale, R. 2000. Building Natural Language Generation Systems. Cambridge
University Press.
Reiter, E. and Sripada, S. 2002. Human variation and lexical choice. Comp. Ling. 28, 545–553.
Reiter, E., Sripada, S., and Robertson, R. 2003. Acquiring correct knowledge for natural
language generation. Journal of Artificial Intelligence Research 18, 491–516.
Seneff, S. and Polifroni, J. 2000. Formal and natural language generation in the mercury
conversational system. In Proc. ICSLP. Vol. 2. 767–770.
Snyder, B. and Barzilay, R. 2007. Database-text alignment via structured multilabel classiﬁcation. In Proc. 20th IJCAI. 1713–1718.
Soderland, S. 2007. Moving from textual relations to ontologized relations. In AAAI Spring
Symposium workshop on Machine Reading.
Stent, A., Prasad, R., and Walker, M. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proc. 42nd ACL. 79–86.
Theune, M. 2003. From monologue to dialogue: natural language generation in OVIS. In AAAI
2003 Spring Symposium on Natural Language Generation in Written and Spoken Dialogue.
141–150.
Turney, P. D. 2002. Thumbs up or thumbs down? semantic orientation applied to unsupervisedclassiﬁcation of reviews. In Proc. 40th ACL. 417–424.
Walker, M., Rambow, O., and Rogati, M. 2002. Training a sentence planner for spoken dialogue
using boosting. Computer Speech and Language 16, 3-4.
White, M. and Caldwell, T. 1998. EXEMPLARS: A practical, extensible framework for dynamic text generation. In Proc. INLG. 266–275.
Wilson, T., Wiebe, J., and Hwa, R. 2004. Just how mad are you? ﬁnding strong and weak
opinion clauses. In Proc. AAAI. 761–769.

ACM Journal Name, Vol. V, No. N, Month 20YY.

21

22

·

Ryuichiro Higashinaka, Marilyn A. Walker and Rashmi Prasad

APPENDIX
A. ADJECTIVAL PHRASES (APS) IN THE RESTAURANT DOMAIN
service=1
service=2
service=3
service=4

service=5

atmosphere=2
atmosphere=3
atmosphere=4
atmosphere=5

horrendous, inattentive, forgetful and slow, really slow, still marginal,
young, great, awful, bad, horrible, marginal, silly and inattentive, terrible
overly slow, very slow and inattentive
very friendly, friendly and knowledgeable, good, pleasant, prompt,
bad, bland and mediocre
extremely friendly and good, extremely pleasant, really friendly, so
nice, swift and friendly, very friendly, very friendly and accommodating, attentive, fantastic, friendly, friendly and helpful, good, great,
great and courteous, prompt and friendly, all very warm and welcoming
extremely friendly, impeccable, intrusive, legendary, quick and cheerful, superb, the most attentive, very timely, very attentive, very congenial, very courteous, very friendly, very friendly and totally personal, very friendly and welcoming, very friendly and helpful, very
friendly and pleasant, very good, very helpful, excellent, excellent
and friendly, fabulous, fantastic, friendly, friendly and very attentive, friendly and helpful, good, great, prompt and courteous,
great, happy and friendly, outstanding, pleasant, polite, attentive
and prompt, prompt and courteous, prompt and pleasant, stupendous, warm and friendly, wonderful, all courteous, unbelievableeclectic, unique and pleasantbusy, pleasant but extremely hotquite nice and simple, typical, very trendy, very casual, fantastic, great,
wonderful
beautiful, interior, phenomenal, quite pleasant, unbelievably beautiful,very relaxing, very comfortable, very cozy, very friendly, very intimate, very nice, very nice and relaxing, very pleasant, comfortable,
excellent, great, lovely, mellow, nice, nice and comfortable, pleasant, warm and contemporary, warm and very comfortable, wonderfulvalue=3
value=4
value=5very reasonable
very good, great, pretty good, reasonable
extremely reasonable, totally reasonable, very good, very reasonable,
best, good, great, reasonable

overall=1
overall=2
overall=3
overall=4

thoroughly humiliating, just bad, nicereally bad, greatinteresting, really fancy, decent, great, badnever busy, not very busy, recommended, excellent, good, great, justgreat, outstanding, wonderful
awesome, capacious, extremely pleasant, local, new, overall, overwhelmingly pleasant, pampering, really great, really neat, really nice, really cool, tasty, truly great, ultimate, very enjoyable, very excellent,very good, very nice, very wonderful, amazing, delightful, fantastic,
good, great, marvelous, neat, peaceful but idyllic, special, uniqueand enjoyable, warm and friendly, wonderful

overall=5

ACM Journal Name, Vol. V, No. N, Month 20YY.

Learning to Generate Naturalistic Utterances Using Reviews in Spoken Dialogue Systems·

B. ADJECTIVAL PHRASES (APS) IN THE HOTEL DOMAIN
service=1
service=2
service=3
service=4

service=5

overall=1
overall=2
overall=3
overall=4

overall=5

clueless and not helpful, very poor, very unpleasant and rude, friendly,
awful, rude, rude and very uncooperative
uncooperative and unprofessional, very nice and apologetic, pretty poor,
all other, poor
very friendly and helpful, very helpful and friendly, friendly, niceextremely courteous and eﬃcient, extremely friendly and helpful, so
friendly, very friendly, very friendly and attentive, very helpful, very
helpful and friendly, very helpful and wonderful, very polite andhelpful, excellent, friendly, friendly and knowledgeable, friendly andvery helpful, friendly and helpful, good, great, helpful and friendly,
pleasant, super friendly
awesome, entire, exceptional and very helpful, exceptionally polite andcourteous, extremely attentive, extremely friendly, extremely friendly
and helpful, impeccable, phenomenal, very attentive and exceptionallyhelpful, professional and very friendly, really friendly and helpful, really helpful, so professional, so courteous, so nice, top, unbeatable,
unbelievably courteous and helpful, very eﬃcient and helpful, very
friendly, helpful and professional, very friendly, very friendly andvery helpful, very friendly and courteous, very friendly and helpful, very helpful, very helpful and friendly, very helpful and prompt,
very pleasant and accommodating, amazing, excellent, free, friendly,
personable and accommodating, friendly, friendly and accommodating,friendly and very wonderful, friendly and helpful, great, great andhelpful, helpful and courteous, helpful and friendly, marvelous, niceand helpful, outstanding, super friendly and extremely attentive, super nice and helpful, terriﬁc, wonderful, wonderful and friendly,
wonderful and helpful, all very friendly and helpful, all very niceabsolutely disgusting, disgusting, old, old and outdated, old and dirty,
really bad, so bad, very old, very old and dirty, very unprofessional, very
marginal, whole, pleasant, awful, bad, horrible, nasty, terrible
old, really old and rickety, very seedy, very clean, very comfortable, bad,cheap, terrible
otherwise clean, very comfortable, very nice, very satisfactory, best,clean, decent, fair, great, nice, nice and quiet, pretty good and inexpensive, pretty nice, mediocre
aﬀordable, beautiful, convenient, extremely aﬀordable, extremely clean,fairly new, new, really beautiful, really nice, very aﬀordable, very spacious, very clean, comfortable and beautiful, very clean, comfortableand quiet, very clean, very clean and quiet, very clean and nice, very
good, very nice, very pleasant, clean and aﬀordable, clean and very
nice, clean and luxurious, comfortable, decent, excellent, fabulous,
good, great, impressive, lovely, nice, clean and aﬀordable, nice, niceand friendly, perfect, pleasant, pretty nice, spotless, wonderful,
bad, cool
absolutely beautiful, absolutely brilliant, absolutely remarkable, awesome, beautiful, fairly new, large and nice, little, new, really great, simply the most beautiful, so great, somewhat pricey, truly enjoyable, very
very nice, very clean, very clean and comfortable, very good, very
nice, very positive, whole, clean, quiet and aﬀordable, clean and accessible, excellent, friendly, good, great, lovely, magical, nice, niceand quiet, perfect, real nice, wonderful

facility=1
facility=2
facility=4
facility=5

brokenentirequite nice, very niceclean and comfortabledining=1
dining=4
dining=5

expensivevery good, good, great
delicious, excellent, great

ACM Journal Name, Vol. V, No. N, Month 20YY.

23