1

Learning to Generate Naturalistic
Utterances Using Reviews in
Spoken Dialogue Systems
Ryuichiro Higashinaka (NTT)Rashmi Prasad (Univ. of Pennsylvania)
Marilyn Walker (Univ. of Sheffield)

2Overview• A new method for automaticallyacquiring a generation dictionary inspoken dialogue systems• Uses user reviews on the web• Uses ratings to map a sentence to its
semantic representation with highprecision

3

Objective• Reduce the cost involved withhand-crafting a spoken language
generation module• Benefits:- Facilitate development of a spoken dialoguesystem- Achieve more natural system utterances
using attested language examples fromthe web4

NLG in Spoken Dialogue
Systems
Semantic Representations
Assert-food_quality(Babbo,superb)

Assert-décor(Babbo, superb)

Mappings between semantic representations and realizations
Assert-food_quality(X, superb) Ù X has superb food.
Assert-décor(X, superb) Ù X has superb décor.
Utterances
Babbo has superb food.

Babbo has superb décor.

5Problem• Mappings are created by handÎ It is very costly• Quality may be left uncheckedÎ Utterances can be unnatural
We need an automatic method foracquiring mappings that generate
naturalistic utterances

6

Related Work
• Create mappings from tagged corpora(Barzilay et al., 2002)
Î Requires an hand-annotated corpus• Find expressions from reviews- Adjectives for products (Hu and Liu, 2005)- Product features and adjectives withpolarity (Popescu and Etzioni, 2005)
Î Do not focus on creating mappings

7Approach• Automatically acquire mappings from user
reviews on the webÎ user reviews are widely available• Uses ratings to derive accurate semanticrepresentations for sentences• A mapping is a triple consisting of­ U (Utterance), R (Semantic representation),and S (Syntactic structure)- Having syntactic structures is useful for full-NLG
systems to perform syntactic transformation

8Procedures
1. Collect user reviews on the webÎ create a population of U
2. For each U,
Derive semantic representation R
Derive syntactic structure S
3. Filter inappropriate mappings
4. Add remaining mappings to dictionary

9Collecting user reviews on theweb• Select review websites with individual
ratings for review entities• Collect review comments and ratings• Collect tabular data
Ratings
Food, Service, Value,
Atmosphere, Overall

Tabular Data
Name, Food Type, Location

Ratings

Deriving semantic representation R
Domain Ontology(relations)

Food, Service, Value,
Atmosphere, Overall

Tabular Data
Name, Food Type,
Location

Distinguished
Attributes Meronymy

relationwith thereview entity

Named-entities for
distinguished attributes

Food=food, meal
Service=service, wait staff,…
Value=value, price, expensive,…
Atmosphere=atmosphere, décor,…
Overall=place, experience,…

10RESTAURANT has foodqualityRESTAURANT has servicequality
RESTAURANT has valuequality
RESTAURANT has atmospherequality
RESTAURANT has overallquality
RESTAURANT has FOODTYPERESTAURANT has LOCATION
Utterance U
Named-entity Tagger
Distinguished attributes in U
Corresponding relations in thedomain ontology Î R

Ratings

Review Comment

11

Food=5, Service=5, Value=5, The best Spanish food in New York. I am
Atmosphere=5, Overall=5
from Spain and I had my 28th birthday
there and we all had a great time. Salud!

Review Sentence (U)The best Spanish food in New York.

DSyntS converter

NE-tagged Review Sentence
The best {NE=Foodtype, string=Spanish}{NE=Food, string=food, rating=5}in {NE=Location, string=New York}.

DSyntS (S)Semantic Representation (R)RESTAURANT has FOODTYPERESTAURANT has foodquality=5
RESTAURANT has LOCATION(Rating-related relation has the same scalar value of that rating)12Filtering inappropriate mappings• 6 Filters to guarantee the quality of mappings• No Relations Filter, Other Relations FilterÎ Check whether a mapping has just the relations
expressed in the ontology• Contextual FilterÎ Checks whether U can be utteredindependently of the context• Unknown Words Filter (typos, etc.), Parsing
Filter, Duplicates Filter

Filtering ExampleRatings

13

Review Comment

Food=5, Service=5, Value=5, The best Spanish food in New York. I am
Atmosphere=5, Overall=5
from Spain and I had my 28th birthday
there and we all had a great time. Salud!

Review Sentence (U)•The best Spanish food in New York.
Successfully created R and SOK
Added to thedictionary•I am from Spain and I had my 28th birthday thereand we all had a great time.
•Salud!
NGNo distinguished attributes are foundÎ cannot create R

Filtered by NoRelations Filter

14Experiment• Obtaining mappings in the restaurant
domain• Reviews collected from we8there.com- 3,004 user reviews on 1,810 restaurants- 18,466 sentences- Obtained 451 mappings after filtering• Objective and subjective evaluations

15

Objective Evaluation• Domain coverage- Check how many of the relations in thedomain ontology can be expressed by theobtained mappings• Linguistic variation- Check the expressiveness of the mappings• Generativity- Check whether the mappings can beincorporated in conventional generationengines16

Domain Coverage
Distribution of mappings containing a single relation
RatingRelation

12345Total

food586185794

service

1536175697

atmosphere3383145

value

000

181221overall

325154570

Total

23152164201327

Domain Coverage is almost complete.

17Linguistic Variation• 137 syntactic patterns, 275 distinct
lexemes, 2-15 lexemes per DSyntS(mean 4.63)• Adjectival phrases- food=1 Î awful, bad, cold, burnt- service=1 Î silly and inattentive- atmosphere=5 Î comfortable, mellow

The mappings show good variation of language

Example mappings

18[RESTAURANT has foodquality=1][RESTAURANT has foodquality=5][1] Food was cold or burnt.
[2] The food was awful.
[3] The food was very ordinary.
[4] The food has gone downhill.
[5] Bad food.

[1] The food was excellent.
[2] The food is delicious.
[3] Keep up the great food.
[4] The food is delicious but simple.
[5] Great food.
[6] The food was very good.
[7] The food was exquisite.
[8] Food is wonderful.
[9] The food was simply inspired.
[10] The food was absolutely great.
[11] Food was excellent.
[12] The food was great.
[13] The food was superb.
[14] All the food is terrific.
[15] The food was fantastic. etc.

[RESTAURANT has foodquality=2][1] Bad food.
[2] The food was very bland.
[3] Not enough food.
[4] The food is very bland.
[5] Food was not very good.
[6] Very bland flavored food. etc.

19

Generativity• Incorporating the learned mappings into
SPaRKy generator (Walker et al., 2003)
SPaRKy: Babbo has the best overall quality among theselected restaurants with excellent decor, excellent
service and superb food quality.
With learned mappings: Babbo has the best overallquality among the selected restaurants because
atmosphere is exceptionally nice, food is excellent andthe service is superb.

Successfully incorporated into
a conventional generation engine

20

Subjective Evaluation• 10 native English speakers evaluatedbaseline vs. learned mappings- Baseline: 27 hand-crafted mappings(taken from SPaRKy generator)- Learned: 451 learned mappings• Evaluation criteria:- Consistency between semantic
representations and realizations- Naturalness of realizations- 1-5 Likert scale

21Results
5

Consistency

5

Naturalness

4.71
4.46

4.5

4.61
4.5

44

3.5

3.5

baselinelearned4.23

baselinelearned•Consistency is significantly lower, but still high•Naturalness is significantly higherÎThe method creates good quality mappings

22Conclusion• A new method for automaticallyacquiring a generation dictionary inspoken dialogue systems- Uses user reviews on the web- Uses ratings to obtain accurate semantic
representations for sentences• Experimental results showed theeffectiveness of our approach