E2E Refined Dataset
Keisuke Toyama, Katsuhito Sudoh, and Satoshi Nakamura
Nara Institute of Science and TechnologyIkoma, Japan{toyama.keisuke.tb5, sudoh, s-nakamura}@is.naist.jp

Abstract—As a well-known meaning representation (MR)-totext dataset, the E2E dataset has been used by many studies innatural language generation. However, the dataset suffers frommany deletion, insertion, and substitution errors in its MR-text
pairs that affect the quality of MR-to-text system trained using
the dataset. In this paper, we develop a refined dataset by fixing
text and MR errors, applying text normalization, and giving extra
annotations on the MR part. We release Python codes to convert
the original E2E dataset to the refined one on GitHub.
Index Terms—data-to-text, meaning representation, mr-to-text,
natural language generation

TABLE IE XAMPLE OF THE E2E DATASETMR

Textname
The Olive Grove
eatTypepubfood(empty)
priceRange
moderatecustomer rating (empty)
areariverside
familyFriendly
yesnear(empty)
Moderately priced The Olive Grove pub is located on the riverside.
It welcomes kids.

I. I NTRODUCTION
Natural language generation (NLG) [1] is a generative
process that produces natural written or spoken language frominput data not limited to text. For example, machine translationor question answering is an NLG task that generates text
from an unstructured textual input. Data-to-text is another
NLG task that generates text from structured inputs such asconcepts, tables, knowledge graphs, and resource description
frameworks (RDFs). Meaning representation (MR)-to-text isone of the data-to-text tasks where MR is composed of acollection of pairs of a brief text passage and a corresponding
MR with several attribute-value pairs, as shown in Table I.
There are several well-known corpora of MR-to-text, Weather(generating weather reports from meteorological data) [2],RotoWire (generating summaries of sports matches fromgame statistics) [3], WikiBio (generating biographies fromWikipedia infobox) [4] and so on. The E2E dataset [5] ina restaurant recommendation domain, used in the E2E NLG
Challenge [6], is one of the most popular datasets for MR-totext. However, this dataset was developed by crowdsourcing
and suffers from errors in MR-text pairs that affect theperformance of MR-to-text models. In this paper, we aim
to refine the E2E dataset by resolving errors and giving
extra annotations. We fix errors in MR-text correspondences
and remove irrelevant data samples from the dataset. We
also provide additional annotations: Number of sentences, MR
order, and Sentence indexes to control the generated text more
precisely. We demonstrate that the refined dataset, called E2E
Refined Dataset1 , improves MR-to-text performance.
II. E2E DATASETThe E2E dataset is made up of pairs consisting of sentencesof restaurant recommendations in British English and an MR,
1 The dataset and Python programs are available at https://github.com/
KSKTYM/E2E-refined-dataset/

TABLE IIE XAMPLE OF MR ERRORS IN THE E2E DATASET: B OLD INDICATES
DELETION ERROR , U NDERLINE INDICATES INSERTION ERROR , AND Italic
INDICATES SUBSTITUTION ERROR .

MRTextnameWildwoodeatTypepubfoodEnglish
priceRange
more than £30customer rating higharea(empty)
familyFriendly(empty)near(empty)
Wildwood is a restaurant providing take-away deliveries in the
low price range. It is located in the city centre.

which corresponds to the sentences, with eight attributes asshown in Table I. However, some MR-text pairs containdeletion, insertion, and substitution errors. For example, in thetext part of Table II, the value “English” for the food attributeand the value “high” for the customer rating attributeare missing, the value “city centre” for the area attribute iswrongly added, and the value “pub” for the eatType attributeand the value “more than £30” for the priceRange attributeare wrongly replaced with “restaurant” and “low”, respectively.
The dataset should not contain such wrong data for effective
control over the sentence content in MR-to-text. Despite theexistence of updated versions of the E2E dataset that address
errors, including the cleaned [7] and enriched versions [8],these revised datasets still contain deletion, insertion, andsubstitution errors. We found a certain number of errors, shownin Table III. To address this issue, we fixed the inaccuracies in
the correspondence between MR-text and discarded unsuitabledata samples. Moreover, we refined the E2E dataset by manually annotating the MR values to provide further constraints
from the text part. As a result, the E2E refined dataset isTABLE III
N UMBER OF MR LABELLING ERRORS IN EACH DATASET
Error type
Deletion
Insertion
Substitution

E2E dataset [5]Training Validation
Test
10,931
1,096
1,315
10,028
263
16
9,290
794
945

Cleaned dataset [7]Training Validation Test23114,475
471
745
5,795616
666

obtained with 40,560, 4,489, and 4,555 samples in the train,validation, and test splits, respectively. Table VII shows anexample from it.
III. T EXT R EFINEMENTThe E2E dataset contains errors and discrepancies in thelanguage used. We improved the quality of the text parts ofthe dataset by correcting errors and standardizing expressions
as follows.
A. Error Correction
We refined various kinds of errors presented in the originalE2E dataset. Table IV shows their examples. We focused on
the following four error categories.
1) Indefinite Articles: We fixed any errors in the usage ofthe indefinite articles “a” and “an.”2) Irregular MR Values: Every MR is designed to containjust one value per attribute or none. Nonetheless, there are
instances where some MR data might contain two values fora single attribute. To maintain data integrity, we deleted any
inapposite data from the dataset.
3) Overlaps: We deleted duplicated phrases within a sentence.
4) Typos: We identified and fixed more than 3,700 typographical errors.
B. Normalization
We normalized the text in the following six types of aspects.
1) British English: Given that the E2E dataset adheres toBritish English, we substituted words such as “neighbor”,“favor”, and “specialize” spelt in American style with their
British counterparts: “neighbour”, “favour”, and “specialise”.
2) Capital Letters: We refined how to capitalize letters in
MR values and text as follows:• all values of the name and near attributes,• the first letter of all values in the food attribute except“fast food,”• the first letter of each sentence.
3) Currency Expressions: For ease of use, we standardized
the currency unit as “£20” instead of using variations such as“20 quid”, “20lb”, “20gbp”, “20 pounds”, and so on.
4) Prices: The category of priceRange is defined as“lower than £20”, “£20-25”, “more than £30”, “cheap”, “moderate”, and “expensive”, as shown in Table VI. In this context,“£22” should be annotated as “£20-25.” However, to eliminate
confusion, we used the label “£20-25” for all prices fallingwithin that range, including values like “£22”, “£23”, “£24”,and “from £20 to £25.” This approach was also taken for thelabels “lower than £20” and “more than £30.”

Enriched dataset [8]Training Validation
Test
1,262145
89
25,5702,724
3,082
4,1724133955) Quotation Marks: We replaced double quotation marks
with single quotation marks.
6) Symbols: We standardized symbols such as commas,periods, and white spaces.
IV. MR R EFINEMENTThe MRs in the original E2E dataset include labelling
errors. We refined the MR labels as described in Section IV-A.
We provided additional annotations for further controllable
generation study regarding flexible content planning, as described in Section IV-B.
A. Labelling
Throughout the E2E dataset, we corrected MR labelling
errors manually. Additionally, we replaced the value “high”
with “expensive” for the priceRange attribute. Moreover,
we added new labels for the food attribute, including “American”, “Canadian”, and “Thai.” Table VI lists all the refinedlabels.
B. Additional Annotations
1) MR Order: As shown in Table VII, we marked the orderof the MR values mentioned in the corresponding text. In caseof an empty MR value, we represented the order with a “0.”2) Number of Sentences: In Table VII, we marked the total
number of sentences in the text. The count of sentences wasestablished by looking for periods (“.”) and question marks(“?”). As per the sample in the table, the section of the text
contains two periods. Therefore, we set the count of sentences
as “2.”
3) Sentence Indexes: We also provided annotations for theappearance of each MR value in the corresponding sentences
as shown in Table VII. For instance, the phrases related to thevalue “riverside” for the area attribute and the value “yes”
for the familyFriendly attribute are found in the first andsecond sentences, respectively. In cases where an MR value isempty, we set the index to “0.”V. MR-T EXT PAIR R EFINEMENT
To further enhance the dataset, we refined the MR-text pairs
by removing repetitions and utilizing a strategy to convey
certain values effectively in both the MR and text.
A. Deduplication
We excluded approximately 1,500 MR-text pairs from thedataset as a result of the deduplication process.

TABLE IV
E XAMPLES OF ERROR CORRECTION
Refinement typeIndefinite article
Irregular MR values

OverlapTypos

Original text
Cocom is a average family friendly restaurant.
Clare Hall is known for Fast food and coffee shop style bakeries although, customers only rate them average the Clowns
are quite amusing.
The Golden Curry served English food, is adult only, is in thecity centre, is adult only, has a customer rating of 5 out of 5and is near the Café Rouge.
Moderately priced fast found can be found at Blue Spice incity centre.

Refined text
COCOM is an average family friendly restaurant.
(removed)
THE GOLDEN CURRY served English food, is adult only, isin the city centre, has a customer rating of 5 out of 5 and isnear the CAFÉ ROUGE.
Moderately priced fast food can be found at BLUE SPICE incity centre.

TABLE VE XAMPLES OF NORMALIZATION
Refinement type
British English
Capital letters
Currency expression
PricesQuotation marks
Symbols

Original text
Cheap family favorite, The Twenty Two, near The Rice Boatin riverside, got a 5 out of 5.
Giraffe is kid friendly. it is located near riverside
The Punter is a Japanese restaurant under 20 pounds.
If you’re looking for pub grub or Indian food, you could try
The Plough. No you can’t take your kids there but the prices
are reasonable about £24 for a meal. You’ll find it near to CaféRouge.
A highly rated coffee shop “The Punter” serving English foodpriced between £20 - £25 and is child friendly.
Wildwood, located near the city center., is a low price pub.

Refined text
Cheap family favourite, THE TWENTY TWO, near THE
RICE BOAT in riverside, got a 5 out of 5.
GIRAFFE is kid friendly. It is located near riverside.
THE PUNTER is a Japanese restaurant under £20.
If you’re looking for pub grub or Indian food, you could try
THE PLOUGH. No you can’t take your kids there but the prices
are reasonable about £20-25 for a meal. You’ll find it near toCAFÉ ROUGE.
A highly rated coffee shop ‘THE PUNTER’ serving English
food priced between £20-25 and is child friendly.
WILDWOOD, located near the city centre, is a low price pub.

TABLE VI
A LL VARIATIONS OF MR VALUES IN THE E2E REFINED DATASET
AttributeName
eatTypefood
priceRange
customer ratingarea
familyFriendly
near

Number of variations
141177332MR values (delexicalized)
NAME(empty), coffee shop, pub, restaurant(empty), fast food, American, Canadian, Chinese, English, French, Indian, Italian, Japanese, Thai
(empty), less than £20, £20-25, more than £30, cheap, expensive, moderate(empty), average, high, low, 1 out of 5, 3 out of 5, 5 out of 5(empty), city centre, riverside(empty), no, yes(empty), NEAR

B. Delexicalization

the value “expensive” for the priceRange attribute isreverted to “high” (see Section IV-A).
We did not use additional annotations (described in SectionIV-B) either.
•As the value for the name attribute and that for near
attribute are always directly reflected in the sentences, we standardized the data by substituting these values in the MR values
and sentences with special letters, “NAME” and “NEAR.”
We retained the original values for both attributes, which
are required to generate proper sentences, even though thedelexicalized data are beneficial to train MR-to-text models.
VI. E XPERIMENTS
We investigated the effect of the refinement by the following
experiments.
A. Dataset
We used the original E2E dataset and the E2E refined
dataset. For a fair comparison, we editted some values in theE2E refined dataset as follows:• only the first letter of each word of name and nearvalues are capitalized,

B. Method
We used TGen2 [9], an LSTM-based sequence-to-sequencemodel that utilizes an attention mechanism. It was the baseline method of the E2E NLG Challenge. We developed twomodels. The first model was trained using the original E2E
dataset, while the second was trained using the E2E refined
dataset.
C. Metrics
We used BLEU [10], NIST [11], METEOR [12],ROUGE L [13], and CIDEr [14], which were used for theE2E NLG challenge3 , as evaluation metrics for both models.
2 https://github.com/UFAL-DSG/tgen
3 https://github.com/tuetschek/e2e-metrics

TABLE VII
E XAMPLE OF THE E2E REFINED DATASET ( ORIGINAL SAMPLE IS SHOWN IN TABLE I).
AttributeValue
NAME(THE OLIVE GROVE)pub(empty)
moderate(empty)
riverside
yes(empty)

nameMR

Number of sentences
Text
Text (delexicalized)

Order

Sentence index

21

eatType3
1
food00
priceRange
11customer rating
00area4
1
familyFriendly
52
near002Moderately priced THE OLIVE GROVE pub is located on the riverside. It welcomes kids.
Moderately priced NAME pub is located on the riverside. It welcomes kids.

TABLE VIIIR ESULTS OF THE EXPERIMENTS
DatasetE2E dataset
E2E refined dataset

BLEU(↑)
0.5462
0.5581

NIST(↑)
7.62097.8378The evaluation was conducted on the test set of the E2E refined
dataset.
D. Results
As listed in Table VIII, the scores reveal that the modeltrained on the E2E refined dataset surpassed the performanceof the model trained on the original dataset, except for
ROUGE L. These results suggest that the refined dataset
contains more accurate label information, which ultimately led
to improved performance.
VII. L IMITATIONS
Despite our efforts to adapt the E2E dataset for the progression of MR-to-text models, several constraints persist:• As discussed in Section III-A, we deleted data withirregular MR values. Nonetheless, different formulationsof MR-to-text problems might permit multiple values inmore intricate scenarios.
• We currently disregard referring expressions, even though
they are generally acceptable.
• We treat all attributes except name as modifiers of aname. However, there are instances where an attributemodifies near, which our current formulation overlooks.
VIII. C ONCLUSION
In this study, we presented our E2E refined dataset. We
reduced the number of errors in the original dataset byfixing inaccuracies and standardizing phrases. Furthermore, weadded new annotations, number of sentences, MR order, andsentence indexes, enabling us to control the generated text
more precisely. Our tests indicated that this refined dataset
contributed to NLG’s better performance. We expect thisdataset to inform future research in various areas, including
data-to-text.

METEOR(↑)
0.41030.4252

ROUGE L(↑)
0.65610.6488CIDEr(↑)
2.24482.3865ACKNOWLEDGMENT
Part of this work was supported by JSPS KAKENHI Grant
Number JP21H05054.
R EFERENCES[1] C. Dong, Y. Li, H. Gong, M. Chen, J. Li, Y. Shen, and M. Yang, “Asurvey of natural language generation,” ACM Computing Surveys, vol.
55, issue 8, pp. 1–38, 2022.
[2] A. Balakrishnan, J. Rao, K. Upasani, M. White, and R. Subba, “Constrained decoding for neural NLG from compositional representations
in task-oriented dialogue,” in Proceedings of the 57th Annual Meetingof the Association for Computational Linguistics, pp. 831–844, 2019.
[3] S. Wiseman, S. M. Shieber, and A. M. Rush, “Challenges in datato-document generation,” in Proceedings of the 2017 Conference onEmpirical Methods in Natural Language Processing, pp. 2253–2263,
2017.
[4] R. Lebret, D. Grangier, and M. Auli, “Neural text generation from structured data with application to the biography domain,”, in Proceedingsof the 2016 Conference on Empirical Methods in Natural Language
Processing, pp. 1203–1213, 2016.
[5] J. Novikova, O. Dušek, and V. Rieser, “The e2e dataset: new challenges
for end-to-end generation,” in Proceedings of the 18th Annual SIGdial
Meeting on Discourse and Dialogue, pp. 201–206, 2017.
[6] O. Dušek, J. Novikova, and V. Rieser, “Evaluating the state-of-the-art
of end-to-end natural language generation: the e2e NLG challenge,”
Computer Speech and Language, vol. 59, pp.123–156, 2020.
[7] O. Dušek, D. M. Howcroft, and V. Rieser, “Semantic noise mattersfor neural natural language generation,” in Proceedings of the 12thInternational Conference on Natural Language Generation, pp. 421–426,
2019.
[8] T. C. Ferreira, H. Vaz, B. Davis, and A. Pagano, “Enriching the e2edataset,” in Proceedings of the 14th International Conference on Natural
Language Generation, pp. 177–183, 2021.
[9] O. Dušek and F. Jurčı́ček, “Sequence-to-sequence generation for spoken
dialogue via deep syntax trees and strings,” in Proceedings of the 54th
Annual Meeting of the Association for Computational Linguistics, pp.
45–51, 2016.
[10] K. Papineni, S. Roukos, T. Ward, and W. Zhu, “BLEU: a method forautomatic evaluation of machine translation,” in Proceedings of the 40th
Annual Meeting of the Association for Computational Linguistics, pp.
311–318, 2002.
[11] G. Doddington, “Automatic evaluation of machine translation quality
using n-gram co-occurrence statistics,” in Proceedings of the 2nd International Conference on Human Language Technology Research, pp.
138–145, 2002.

[12] M. Denkowski and A. Lavie, “Meteor universal: language specifictranslation evaluation for any target language,” in Proceedings of the9th Workshop on Statistical Machine Translation, pp. 376–380, 2014.
[13] C. Lin, “ROUGE: a package for automatic evaluation of summaries,” inText Summarization Branches Out, pp. 74–81, 2004.
[14] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: consensus-based
image description evaluation,” in Proceedings of 2015 IEEE conference
on Computer Vision and Pattern Recognition, pp. 4566–4575, 2015.