# Quality Control for Distantly-Supervised Data-to-Text Generation via Meta Learning
## 5 Experiments
### Setup
#### 5.1.2 Evaluation Metrics  
Following [PERSON] et al. [6], we use the following evaluation metrics to assess models' performance with the evaluation script by [PERSON] et al. [41]:  
* BLEU [42]: It evaluates the model's generated text's quality based on the n-gram overlap between generated text and reference text. The output of BLEU ranges from 0% to 100%. If the generated text is identical to the reference text, the score will be 100%. If the generated text shares no n-gram overlap with the reference text, the score will be 0%. The calculation of BLEU is illustrated in Equation (11). It consists of the following three parts. \(P_{n}\) is the n-gram precision of the generated text, compared to the reference. \(w_{n}\) is a positive weight for each n-gram. \(BP\) is the abbreviation for brevity penalty, which will penalize those generated texts that are shorter than reference, since the shorter the text, the higher precision it can potentially obtain. In practice, we report the BLEU score with N = 4 and the uniform weights \(w_{n}=\frac{1}{4}\). \[BLEU=\exp(\sum_{n=1}^{N}w_{n}\log P_{n})\cdot BP\] (11)
* NIST [43]: It proposes information-weighted N-grams counts that weigh more heavily to the N-grams that are deemed more informative (i.e., occur less frequently than others). The details can be found in Doddington [43].
* METEOR [44]: In addition to exact string matching between words in generated text and reference, it proposed to use WordNet to match words that share the same stem or are synonyms of each other, since those words share the same meaning. Furthermore, it proposes to group words in the text into chunks and use this to measure how well-ordered the words in the generated text are with respect to the reference. Equation (12) shows how the METEOR score is calculated. The \(Fmean\) combines the precision and the recall of generated text. The \(Penalty\) is based on the number of chunks in the matched sequence in the generated text. The fewer the chunks are grouped, the better the words are ordered, compared to the reference. \[METEOR=Fmean\cdot(1-Penalty)\] (12)
* ROUGE\({}_{L}\)[45]: It proposes to use the Longest Common Subsequence (LCS) to match the generated text and reference. Equation (13) shows that \(ROUGE_{L}\) is the F-measure of the precision and recall of the length of matched LCS. Since the \(\beta\) will be set to a large number, this measure is actually recall-oriented. \[ROUGE_{L}=\frac{(1+\beta^{2})R_{lcs}P_{lcs}}{R_{lcs}+\beta^{2}P_{lcs}}\] (13)
* CIDEr [46]: It proposed to use Term Frequency Inverse Document Frequency (TF-IDF) as weights to characterize the similarity between the generated text and reference and use cosine similarity function to calculate the CIDEr score. The idea of using TF-IDF is that it will give higher weight to infrequently occurring words in the dataset and  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **WITA** & **DIST-ToTTo** \\ \hline Size & 55,400 & 128,533 \\ Text Length & (18.8, 17, 5, 59) & (24.0, 22, 3, 81) \\ KB Number & (3.0, 3, 1, 11) & (3.5, 3, 1, 130) \\ Vocabulary & 102,404 & 153,963 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of WITA (abbreviation for the data source Wikipedia and Wikidata) and DIST-ToTTo (abbreviation for Distantly-supervised Table-To-Text) datasets. KB number refers to the number of knowledge base triples, that served as the model’s input, for each data instance. For the text length and KB number, the data are mean, median, min and max, respectively. The statistics of WITA are from [PERSON] et al. [6].  
lower weight to those commonly occurring words, as the latter will be deemed less informative. The details can be found in [PERSON] et al. [46].