E2E Refined Dataset
Keisuke Toyama, Katsuhito Sudoh and Satoshi Nakamura
Nara Institute of Science and Technology

O-COCOSDA 2023

Data-to-Text
Structured Data

Weathercondition1

sunnydate_time1

this weekend

avg_high1

60s

low2

43

date_time2

Sunday evening

chance3

likelywind_summary3strongdate_time3

Saturday morning

It’ll be sunny throughout this weekend. The highwill be in the 60s, but expect temperatures todrop as low as 43 degrees by Sunday evening.
There’s also a chance of strong winds on
Saturday morning.
2Data-to-Text(Natural Language Generation)

Text(sentences)

WikiBio

RotoWire

Frederick Parker-Rhodes (21 March 1914–21
November 1987) was an English linguist, plant
pathologist, computer scientist, mathematician,mystic, and mycologist.

The Atlanta Hawks defeated the Miami Heat,
103-95, at Philips Arena on Wednesday. Atlanta
was in desperate need of a win and they wereable to take care of a shorthanded Miami team
here. Defense was key for the Hawks, as theyheld the Heat to 42 percent shooting ...(*snip*)

E2E Refined Dataset

O-COCOSDA 2023

E2E dataset⚫ MR(Meaning Representation)-to-Text dataset• MR: 8 attribute-value pairs• Text: British English sentences in the restaurant recommendation domainMRText

attributevaluename

The Olive Grove

eatTypepubfood(empty)

priceRange

moderatecustomer rating(empty)

areariverside

familyFriendly

yesnear(empty)

Moderately priced The Olive Grove pub is located on the riverside. It welcomes kids.

• used in the E2E NLG Challenge (2017)

3

E2E Refined Dataset

O-COCOSDA 2023

Deletion/Insertion/Substitution Errors in E2E dataset⚫E2E dataset includes many errorsMR

Text

attributevalue

nameThe PuntereatTypecoffee shop

foodEnglish

priceRange

moderatecustomer rating

1 out of 5area(empty)

familyFriendly

yesnearCafe Sicilia

The Punter is a cheap family friendly coffee shop located in City Centre near Cafe Sicilia. 1 out of 5
customer rating.
moderate is replaced with cheap↓
substitution error

4

English is missing↓
deletion error

E2E Refined Dataset

City Centre is added↓
insertion error
O-COCOSDA 2023

Updated dataset for E2E⚫updated versions• cleaned (Dušek et al. 2019)• enriched (Ferreira et al. 2021)→ still contain a certain number of errors
Error Typeoriginal E2E dataset

Cleaned dataset

Enriched dataset

training

validation

test

training

validation

test

training

validation

test

deletion

10,931

1,096

1,315

23

1

1

1,262

145

89

insertion

10,028

263

16

4,475

471

745

25,570

2,724

3,082

substitution

9,290

794

945

5,795

616

666

4,172

413

395

E2E refined dataset

developed Python programs to convert the original E2E dataset to the E2E refined dataset
5E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Error Correction⚫indefinite articles• For an child friendly, average coffee shop serving fast food try The Eagle, riverside nearBurger King.

⚫irregular MR values• Clear Hall is known for Fast food and coffee shop style bakeries although, customers only
rate them average the Clowns are quite amusing.

⚫overlaps• The Golden Curry served English food, is adult only, is in the city centre, is adult only, hasa customer rating of 5 out of 5 and is near the Café Rouge.

⚫typos• Moderately priced fast found can be found at Blue Spice in city centre.
6E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Error Correction⚫indefinite articles• For an child friendly, average coffee shop serving fast food try The Eagle, riverside nearaBurger King.

⚫irregular MR values• Clear Hall is known for Fast food and coffee shop style bakeries although, customers only
rate them average the Clowns are quite amusing.

⚫overlaps• The Golden Curry served English food, is adult only, is in the city centre, is adult only, hasa customer rating of 5 out of 5 and is near the Café Rouge.

⚫typos• Moderately priced fast found can be found at Blue Spice in city centre.
7E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Error Correction⚫indefinite articles• For an child friendly, average coffee shop serving fast food try The Eagle, riverside nearBurger King.

⚫irregular MR values(removed)• Clear Hall is known for Fast food and coffee shop style bakeries although, customers only
rate them average the Clowns are quite amusing.

⚫overlaps• The Golden Curry served English food, is adult only, is in the city centre, is adult only, hasa customer rating of 5 out of 5 and is near the Café Rouge.

⚫typos• Moderately priced fast found can be found at Blue Spice in city centre.
8E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Error Correction⚫indefinite articles• For an child friendly, average coffee shop serving fast food try The Eagle, riverside nearBurger King.

⚫irregular MR values• Clear Hall is known for Fast food and coffee shop style bakeries although, customers only
rate them average the Clowns are quite amusing.

⚫overlaps• The Golden Curry served English food, is adult only, is in the city centre, is adult only, hasa customer rating of 5 out of 5 and is near the Café Rouge.
(removed)⚫typos• Moderately priced fast found can be found at Blue Spice in city centre.
9E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Error Correction⚫indefinite articles• For an child friendly, average coffee shop serving fast food try The Eagle, riverside nearBurger King.

⚫irregular MR values• Clear Hall is known for Fast food and coffee shop style bakeries although, customers only
rate them average the Clowns are quite amusing.

⚫overlaps• The Golden Curry served English food, is adult only, is in the city centre, is adult only, hasa customer rating of 5 out of 5 and is near the Café Rouge.

⚫typosfood• Moderately priced fast found can be found at Blue Spice in city centre.
10

E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Normalization⚫British English• Cheap family favorite, The Twenty Two, near The Rice Boat in riverside, got a 5 out of 5.

⚫capital letters• a coffee shop Zizzi located by the riverside has a high price range with an averagecustomer rating. they are children friendly.

⚫currency expression / prices• The Punter is a Japanese restaurant under 20 pounds.

⚫quotation marks• A highly rated coffee shop ``The Punter’’ serving English food priced between ₤20 - ₤25
and is child friendly.
11E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Normalization⚫British Englishfavourite• Cheap family favorite, The Twenty Two, near The Rice Boat in riverside, got a 5 out of 5.

⚫capital letters• a coffee shop Zizzi located by the riverside has a high price range with an averagecustomer rating. they are children friendly.

⚫currency expression / prices• The Punter is a Japanese restaurant under 20 pounds.

⚫quotation marks• A highly rated coffee shop ``The Punter’’ serving English food priced between ₤20 - ₤25
and is child friendly.
12

E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Normalization⚫British English• Cheap family favorite, The Twenty Two, near The Rice Boat in riverside, got a 5 out of 5.

⚫capital letters• a coffee shop Zizzi located by the riverside has a high price range with an average
Acustomer rating. they are children friendly.
They⚫currency expression / prices• The Punter is a Japanese restaurant under 20 pounds.

⚫quotation marks• A highly rated coffee shop ``The Punter’’ serving English food priced between ₤20 - ₤25
and is child friendly.
13

E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Normalization⚫British English• Cheap family favorite, The Twenty Two, near The Rice Boat in riverside, got a 5 out of 5.

⚫capital letters• a coffee shop Zizzi located by the riverside has a high price range with an averagecustomer rating. they are children friendly.

⚫currency expression / prices₤20• The Punter is a Japanese restaurant under 20 pounds.

⚫quotation marks• A highly rated coffee shop ``The Punter’’ serving English food priced between ₤20 - ₤25
and is child friendly.
14

E2E Refined Dataset

O-COCOSDA 2023

Text Refinement: Normalization⚫British English• Cheap family favorite, The Twenty Two, near The Rice Boat in riverside, got a 5 out of 5.

⚫capital letters• a coffee shop Zizzi located by the riverside has a high price range with an averagecustomer rating. they are children friendly.

⚫currency expression / prices• The Punter is a Japanese restaurant under 20 pounds.

⚫quotation marks`‘• A highly rated coffee shop ``The Punter’’ serving English food priced between ₤20 - ₤25
and is child friendly.
15

E2E Refined Dataset

O-COCOSDA 2023

MR Refinement⚫re-annotate all MR data manually• developed Python programs for annotation⚫re-define all variations of MR values

16

attribute# variation

valuesname1NAME

eatType4(empty), coffee shop, pub, restaurantfood

11(empty), American, Canadian, Chinese, English, French, Indian, Italian, Japanese, Thai, fast foodpriceRange7(empty), less than £20, £20-25, more than £30, cheap, expensive, moderate

customer rating

7(empty), 1 out of 5, 3 out of 5, 5 out of 5, average, high, lowarea

3(empty), city centre, riverside

familyFriendly

3(empty), no, yes

near2(empty), NEARE2E Refined Dataset

newly added

replaced from “high”

O-COCOSDA 2023MR Refinement (cont.)⚫additional annotations• MR order• number of sentences• sentence indexes

17MR

attributevalueordersentence index

nameTHE OLIVE GROVE

21eatTypepub

3

1

food(empty)

00

priceRange

moderate

1

1

customer rating(empty)

00

area

riverside

4

1

familyFriendly

yes

52

near(empty)

00

Text

Moderately priced THE OLIVE GROVE pub is located on the riverside. It
welcomes kids.

number of sentences

2

E2E Refined Dataset

O-COCOSDA 2023

MR-Text Pair Refinement⚫ delexicalization• The value for “name” and “near” attributes are always directly reflected in the sentences→ substitute the value in MR and sentence with special strings, “NAME” and “NEAR”
MR

18

attributevalue

ordersentence index

nameNAME (THE OLIVE GROVE)

2

1

eatTypepub

3

1

food(empty)

00

priceRange

moderate

1

1

customer rating(empty)

00

areariverside

4

1

familyFriendly

yes

5

2near(empty)

00

Text

Moderately priced THE OLIVE GROVE pub is located on the riverside. It welcomes kids.

Text (delexicalized)

Moderately priced NAME pub is located on the riverside. It welcomes kids.

number of sentences

2E2E Refined Dataset

O-COCOSDA 2023

Experiments⚫ method• TGen (Dušek et al. 2016)- LSTM-based sequence-to-sequence model with an attention mechanism- used as the baseline method for the E2E NLG Challenge• trained 2 models- (A) training dataset in the E2E dataset- (B) training dataset in the E2E refined dataset⚫ metrics• BLEU, NIST, METEOR, ROUGE_L, CIDEr- used for the E2E NLG Challenge⚫ results• the performance of the model (B) outperformed that of the model (A)modeltraining data

BLEU(↑)NIST(↑)METEOR(↑)

ROUGE_L(↑)

CIDEr(↑)(A)

E2E dataset

0.54627.62090.41030.65612.2448(B)

E2E Refined Dataset

0.5581

7.83780.42520.64882.3865accurate label information led to improved performance
19

E2E Refined Dataset

O-COCOSDA 2023

Summary⚫E2E Refined Dataset• refined version of E2E dataset (MR-to-Text)• reduced deletion / insertion / substitution errors in the original E2E dataset- text refinement- error correction- normalization- MR refinement- labelling annotations manually- MR-text pair refinement- delexicalization• additional annotation- MR order
Python programs are available- number of sentences- sentence indexes
20

E2E Refined Dataset

O-COCOSDA 2023