SlowGuess/NAACL_2021_OCR huggingface dataset:
BLEU ↑	METEOR ↑	TER ↓	MoverScore ↑	BERTScore(F1) ↑	BLEURT ↑	PARENT ↑
LSTM with Attention	29.66	0.27	0.63	0.31	0.90	-0.13	0.35
End-to-End Transformer	27.24	0.25	0.65	0.25	0.89	-0.29	0.28
BART-base	47.11	0.38	0.46	0.51	0.95	0.37	0.55
BART-large	48.56	0.39	0.45	0.52	0.95	0.41	0.57
T5-small	47.69	0.39	0.46	0.52	0.95	0.40	0.56
T5-base	49.21	0.40	0.44	0.53	0.95	0.43	0.57
T5-large	50.66	0.40	0.43	0.54	0.95	0.44	0.58
Table 3: Model results on the test set of DART  $\uparrow$  : Higher is better.  $\downarrow$  : Lower is better.
BLEU ↑	METEOR ↑	TER ↓
SEEN	UNSEEN	ALL	SEEN	UNSEEN	ALL	SEEN	UNSEEN	ALL
Pipeline Transformer† (Castro Ferreira et al., 2019)	56.28	23.04	42.41	0.42	0.21	0.32	0.39	0.63	0.50
Pipeline GRU† (Castro Ferreira et al., 2019)	56.09	25.12	42.73	0.42	0.22	0.33	0.39	0.64	0.51
MELBOURNE (Gardent et al., 2017)	54.52	33.27	45.13	0.41	0.33	0.37	0.40	0.55	0.47
BestPlan† (Moryossef et al., 2019)	53.30	34.41	47.24	0.44	0.34	0.39	0.47	0.56	0.51
DualEnc (Zhao et al., 2020)	63.45	36.73	51.42	0.46	0.37	0.41	0.34	0.55	0.44
PlanEnc (Zhao et al., 2020)	64.42	38.23	52.78	0.45	0.37	0.41	0.33	0.53	0.42
Ribeiro et al. (2020)
BART-base‡	63.02	41.74	53.36	0.45	0.35	0.40	0.33	0.52	0.42
BART-large‡	63.71	44.17	54.95	0.46	0.39	0.42	0.33	0.51	0.41
T5-small‡	65.30	45.58	56.57	0.46	0.39	0.43	0.32	0.49	0.40
T5-base‡	64.89	52.86	59.44	0.46	0.42	0.44	0.33	0.42	0.37
T5-large‡	64.89	54.01	59.95	0.46	0.43	0.44	0.34	0.41	0.37
+ DART
BART-base	62.36	46.21	55.14	0.44	0.37	0.41	0.34	0.45	0.39
BART-large	64.51	50.20	58.06	0.46	0.40	0.43	0.32	0.44	0.38
T5-small	65.05	47.81	57.32	0.46	0.40	0.43	0.33	0.46	0.39
T5-base	65.42	50.71	58.80	0.46	0.41	0.44	0.32	0.43	0.37
T5-large	65.82	56.01	61.44	0.46	0.43	0.45	0.32	0.38	0.35
Table 4: The WebNLG 2017 results on the test set. †: We report results from Zhao et al. (2020) who use the evaluation scripts that are strictly the same as the official challenge.‡: We report results calculated with the model outputs on the WebNLG 2017 testset released by Ribeiro et al. (2020).
Tripleset source	Sentence source	% fluent	% faithful	% (fluent+ mostly fluent)	% (faithful+ mostly faithful)
WikiTableQuestions (§ 2.1)	human-written reference	75%	81%	96%	99%
BART-base	74%	57%	93%	84%
T5-base	72%	54%	94%	76%
WikiSQL (§ 2.2)	auto-generated reference	59%	56%	87%	88%
BART-base	66%	51%	92%	83%
T5-base	75%	65%	97%	90%
Table 5: Human evaluation over references and model outputs.
dataset, and WebNLG partition keeps the original data format. In general, we observe that adding DART instances that contain human written sentences brings most improvement, especially on unseen split. While adding E2E partition boosts the scores on seen test split and deteriorates the performance on unseen test split. This trend is consistent across all models. Comparing results of declarative sentence partition and human written sentence partition, we see that for most of the models, DART instances with human written sentences have better quality as it brings more improvement to the task.
In Table 5, we perform human evaluation on DART based on two criteria: (1) fluency if a sentence is natural and grammatical, and (2) semantic faithfulness if a sentence is supported by the input triples. We defined three levels of fluency: fluent, mostly fluent, and not fluent, and the same for semantic faithfulness. We ask 5 internal annotators to evaluate on 100 triplesets sampled from declarative sentence partition and another 100 triplesets sampled from human written sentence partition. Each
tripleset is paired with 3 sentences, one of them is the reference sentence, and the other two are outputs of BART-base and T5-base models.
The results in Table 5 attest to the high quality of our annotations since the human written references achieve highest fluency and faithfulness comparing to outputs of two strong baseline models. The evaluation on faithfulness also demonstrates that there is a considerable gap between the DART reference and the outputs of the state-of-the-art pretrained model, showing that there is a large room for improvement. We also noticed that the auto-generated declarative sentences are not as fluent or faithful as the model outputs because they are generated with a rule-based system. However, we decided to release this partition, along with other partitions of DART because it demonstrates an economic way to obtain large amounts of DART instances and it also shows benefits for generalization due to the diverse topics it contains.
Data-to-Text Data-to-Text generation aims to produce natural language output from structured input. Applications include generating sports commentaries (Chen and Mooney, 2008; Wiseman et al., 2017), weather forecasts (Liang et al., 2009; Konstas and Lapata, 2012), biographical texts (Lebret et al., 2016; Liu et al., 2018), knowledge-base descriptions (Gardent et al., 2017), dialogue response generation (Wen et al., 2015, 2016), and commonsense reasoning (Lin et al., 2020). Yet, most existing datasets are restricted to specific domains and applications. In contrast, a major source of DART is from Wikipedia tables covering various domains and topics.
Representation of Data The input of the Datato-Text datasets take different formats, including slot-value pairs, Abstract Meaning Representation (AMR) (Song et al., 2017; Ribeiro et al., 2019), Minimal Recursion Semantics (MRS) (Hajduk et al., 2019), Resource Description Framework (RDF triples) (Gardent et al., 2017), and logic forms (Chen et al., 2020b). There are also studies of converting tabular data to RDF triples in the Semantic Web community (Kellogg et al., 2015). Recently, some open-domain table-to-text datasets have been proposed including WikiTableText (Bao et al., 2018), LogicNLP (Chen et al., 2020a), and ToTTo (Parikh et al., 2020), whose inputs are rows or entire tables. In ToTTo, highlighted cells are
also provided as input, and the authors found using only highlighted cells with flat row and column headers led to higher performance than using the entire table.
In contrast, DART is constructed by first annotating the tree-structured table ontology that encodes the semantic dependencies among table headers, and we could flexibly incorporate additional contexts such as the table title to the ontology tree. We then use an automatic procedure to extract connected components from the tree to form the input of a DART instance. Our annotation framework not only provides a flexible way of incorporating any contexts to the representation of tables, but also encodes hierarchical relationships among table headers and contexts, ensuring the extracted triples are logically consistent and can be described in text without loss of information.
Model Traditional Data-to-Text models break the generation progress into different stages such as signal analysis, data interpretation, document planning, microplanning, and realization (Reiter and Dale, 2000; Reiter, 2007). Recently, neural encoder-decoder models based on attention and copy mechanisms have shown promising results (Gehrmann et al., 2018; Puduppully et al., 2018, 2019; Castro Ferreira et al., 2019). Furthermore, recent progress on pretrained models such as GPT-2 (Radford et al., 2018), BART (Lewis et al., 2020) and T5 (Raffel et al., 2020) has shown effective results for text generation tasks on machine translation, summarization, and conversation response generation. Chen et al. (2020c); Peng et al. (2020); Kale (2020) also finetune pretrained models on Data-to-Text tasks.
In this paper, we introduce DART, an open-domain corpus for structured data record to text generation. DART's ontology-preserving representation of data inputs differentiates itself from other open-domain Data-to-Text corpora. We found that DART introduces new challenges to several state-of-the-art Data-to-Text models due to its open-domain nature and its ontology structure of the semantic triple input. Furthermore, we found that using it for data augmentation improves other Data-to-Text tasks. For future work, we will explore more controlled, high-fidelity generation that better incorporates the ontology hierarchy of data.
Our dataset is constructed by accumulating and processing resources from various existing datasets that are open to the public. In addition, we collect annotations on structure of tabular data and human written sentences that describe data records.
The existing resources that we utilize mainly consist of (1) tabular data from Wikipedia, (2) information of restaurants presented with dialogue-act meaning representation and its textual description (E2E), and (3) information of various entities and their relationship that are in 15 different categories of DBPedia, which is a knowledge base built on contents created in various Wikimedia projects (WebNLG). It is possible that there are biases in these resources, either in the tabular data or the textual description written by humans.
For additional annotations we collected, we have two groups of annotators participating: internal annotators who are the authors of this work, and external annotators recruited from the Amazon Mechanical Turk platform. On MTurk, we use a pay rate of $15 per hour approximately based on our estimation of the time it takes to complete our annotation tasks. In total, it took 125 hours to complete all tasks on the Amazon Mechanical Turk platform. There are three annotation tasks: (1) Annotators are asked to specify ontological structure of the table by indicating relationship between table column headers, (2) Annotators are asked to write descriptions that are fluent and semantically faithful to the data records presented to them, and (3) Annotators are asked to evaluate sentences that are either references or model generated outputs. We acknowledge that it is also possible to have biases in the sentences written by the annotators, or in the data records that are presented to them.
We conducted experiments on our own dataset and the WebNLG dataset using BART and T5, two large-scale pretrained models. Both models are trained on large amounts of textual data such as news, books, and web text, which may contain any kinds of biases. As a result, it is possible to insert those biases into the models.
In total, we conducted 43 experiments: 7 on DART and 36 for our ablation study on the WebNLG dataset. We use a single NVIDIA V100 GPU for all experiments and each experiment took from 5 to 40 hours depending on the model size.
Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuanhua Lv, Ming Zhou, and Tiejun Zhao. 2018. Table-to-text: Describing table region with natural language. In AAAI.
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. In EMNLP.
Alison J Cawsey, Bonnie L Webber, and Ray B Jones. 1997. Natural language generation in health care.
David L Chen and Raymond J Mooney. 2008. Learning to sportscast: a test of grounded language acquisition. In ICML.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In ACL.
Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020b. Logic2Text: High-fidelity natural language generation from logical forms. In Findings of EMNLP.
Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Yang Wang. 2020c. Few-shot nlg with pre-trained language model. In ACL.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02922.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In ACL.
Ondrej Dusek, David M. Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural language generation. In INLG.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In LREC.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In EMNLP-IJCNLP.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In INLG.
Sebastian Gehrmann, Falcon Dai, Henry Elder, and Alexander Rush. 2018. End-to-end content and plan selection for data-to-text generation. In INLG.
Valerie Hajdik, Jan Buys, Michael Wayne Goodman, and Emily M Bender. 2019. Neural text generation from rich semantic representations. In *NAACL*.
Hamza Harkous, Isabel Groves, and Amir Saffari. 2020. Have your text and use it too! end-to-end neural data-to-text generation with semantic fidelity. arXiv preprint arXiv:2004.06577.
Mihir Kale. 2020. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005.10433.
Gregg Kellogg, Ivan Herman, and Jeremy Tandy. 2015. Generating RDF from tabular data on the web. W3C recommendation, W3C. https://www.w3.org/TR/2015/REC-csv2pdf-20151217/.
Ioannis Konstas and Mirella Lapata. 2012. Unsupervised concept-to-text generation with hypergraphs. In NAACL.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In EMNLP.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In ACL.
Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In ACL.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of EMNLP.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In AAAI.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. In NAACL.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017a. Why we need new evaluation metrics for nlg. In EMNLP.
Jekaterina Novikova, Ondrej Dusek, and Verena Rieser. 2017b. The E2E dataset: New challenges for end-to-end generation. In SIGDIAL.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In EMNLP.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In ACL.
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task-oriented dialog. In arXiv.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2018. Data-to-text generation with content selection and planning. In AAAI.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with entity modeling. In ACL.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Ehud Reiter. 2007. An architecture for data-to-text systems. In Proceedings of the Eleventh European Workshop on Natural Language Generation (ENLG 07).
Ehud Reiter. 2020. Openai gpt system: What does it do? Technical report, Arria.
Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press.
Leonardo F. R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In EMNLP.
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2020. Investigating pretrained language models for graph-to-text generation. arXiv.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. In ACL.
Linfeng Song, Xiaochang Peng, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2017. Amr-to-text generation with synchronous node replacement grammar. In ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Pavlos Vougiouklis, Hady ElSahar, Lucie-Aimée Kaffee, Christophe Gravier, Frédérique Laforest, Jonathon S. Hare, and Elena Simperl. 2018. Neural wikipedian: Generating textual summaries from knowledge base triples. Journal of Web Semantics, 52-53:1 - 15.
Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. Conditional generation and snapshot learning in neural dialogue systems. In EMNLP.
Tsung-Hsien Wen, Milica Gašić, Nikola Mrksić, PeiHao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In EMNLP.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In EMNLP.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In ICLR.
Chao Zhao, Marilyn Walker, and Snigdha Chaturvedi. 2020. Bridging the structural gap between encoding and decoding for data-to-text generation. In ACL.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In EMNLP.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
The Appendix contains the following contents:
- Results of the ablation study on WebNLG 2017 testset.
- Statistics of the table ontology annotations.
- Examples of tables that help illustrate DART's annotation procedure.
Examples of model outputs.
Tables	Ontology depth (min, med, max)	Nodes in ontology (min, med, max)	Branching factor (mean)
WikiTableQuestions	2060	1, 1, 4	2, 6, 25	4.0
WikiSQL	3563	1, 1, 4	3, 7, 25	5.1
Table 7: Properties of the ontology in the WikiTableQuestions and WikiSQL samples in DART. Branching factor refers to the average number of children across all non-leaf nodes in a table's ontology.
Figure 3: Distribution of column ontology depths in the WikiTableQuestions and WikiSQL samples in DART v1.1.1.
Figure 4: Examples of DART instance
Figure 5: An example of the data cleaning. The top left table had a missing column name and the table title was not specific to the data; our internal annotators add the missing column name "Year" and linked the rest of the columns to the "Year" column. The bottom left table had repeat column names in the table; our internal annotators disambiguate the columns by making the column names more specific.
Figure 6: A WikiTableQuestions table that uses [TITLE] in the ontology.
Figure 7: A manually annotated table from WikiTableQuestions with a sentence that uses the table title.
Figure 8: A manually annotated table from WikiTableQuestions. Annotators created a table ontology, and they wrote sentences encapsulating the information in the orange cells for a given row. Whenever a sentence referenced the table title, that sentence was also highlighted green.
Table Title: Peter De Villiers
Competition	Event	Competition	[TITLE]	Position	[TITLE]
Year	Competition	Venue	Position	Event	Time
2000	World Junior Championships	Santiago, Chile	2nd	400 m hurdles	50.52
2000	World Junior Championships	Santiago, Chile	4th	4x400 m relay	3:07.66
2005	World Championships	Helsinki, Finland	17th (sf)	400 m hurdles	49.75
2005	World Championships	Helsinki, Finland	13th (h)	4x400 m relay	3:04.64
2006	Commonwealth Games	Melbourne, Australia	7th	400 m hurdles	50.51
2006	African Championships	Bambous, Mauritius	4th	400 m hurdles	50.96
2007	All-Africa Games	Algiers, Algeria	2nd	400 m hurdles	48.91
2007	All-Africa Games	Algiers, Algeria	8th	4x400 m relay	DNF
2007	World Championships	Osaka, Japan	15th (sf)	400 m hurdles	49.37
2008	Olympic Games	Beijing, China	12th (sf)	400 m hurdles	49.44
The competitor won a silver medal at the 2000 World Junior Championships in Santiago, Chile.
They finished the  $400\mathrm{m}$  hurdles 7th in 50.51.
In Bambous, Peter De Villiers came in 4th in the African Championships.
The 2007 All-Africa Games held a  $4 \times 400 \mathrm{~m}$  relay.
He competed in the 2008 Beijing Olympic Games.
Title: 1950 Indianapolis 500
Pos No Driver Constructor Lap Gap 7 69 Duke Dinsmore Kurtis Kraft-Offenhauser 4:34.67 + 6.70
Duke Dinsmore, number 69, finished in 7th position at the 1950 Indianapolis 500.
Title: Kurt Maschler Award
Year Author Illustratorator Title Publisher 1991 Colin McNaughton McNaughton Have You Seen who's just moved in next door to us? Walker
"Have You Seen who's just moved in next door to us?" was published by Walker in 1991.
Title: Maxi Priest
Year	Song	U.S.	U.S. R&B	U.S. AC	UK	Album
1990	"Just a Little Bit Longer"	62	30	None	62	Bonafide
Max Alfred "Maxi" Elliott known by his stage name Maxi Priest, is a British reggae vocalist of Jamaican descent.
Figure 9: An example of collected MTurk-generated sentences for WikiTableQuestions. Internal annotators went through the generated sentences and checked for both sentence coherence and title usage. Below the generated sentences, 'y' meant the sentence references the table title, 'n' meant the sentence did not use the table title, 'x' meant the sentence was nonsensical.
Title: Harrisburg City Islanders
Year	Division	League	Regular Season	Playoffs	U.S. Open Cup	Avg. Attendance
2012	3	usl pro	6th	quarterfinals	quarterfinals	1452
A team in the usl pro league has an average attendance of 1452.
Year	Division	League	Regular Season	Playoffs	U.S. Open Cup	Avg. Attendance
2009	3	usl second division	3rd	semifinals	quarterfinals	1857
In 2009, the Harrisburg City Islanders got semifinals in Playoffs in league usl second division.
Title: Cincinnati Kings
Year	Division	League	Regular Season	Playoffs	Open Cup
2008	4	usl pdl	4th, great lakes	did not qualify	did not qualify
When 4th, great lakes is the regular season usl pdl is the league.
Figure 10: Automatically generated declarative sentences from WikiSQL with human validation. Annotators went through the generated sentences and checked for both sentence coherence and title use. Below the generated sentences, 'y' meant the sentence references the table title, 'n' meant the sentence did not use the table title, 'x' meant the sentence was nonsensical.