# Quality Control for Distantly-Supervised Data-to-Text Generation via Meta Learning
## 5 Experiments
### Analysis
#### 5.3.2 Noise Effect Analysis  
Since the dataset contains noise among its training instances, the straightforward way to avoid the noise is to drop the noisiest data from the dataset, avoiding the model from being misguided by them. Using the data quality confidence score for each training instance, we can filter those with the lowest score. Take the last instance in Figure 2 as the example, the input information is <[PERSON], data_of_birth, 1 January 1988> and \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Model** & **Grammaticality** & **Fidelity** \\ \hline \multirow{4}{*}{WITA} & S2 ST & 11.61 & 5.72 \\  & S2 ST + Full & 16.17 & 12.06 \\  & DSG & 11.33 & 7.44 \\  & D + Full & 12.67 & 11.33 \\ \hline \multirow{4}{*}{DIST-ToTTo} & S2 ST & 7.72 & \(-\)2.67 \\  & S2 ST + Full & 11.67 & 7.5 \\ \cline{1-1}  & DSG & 15.06 & 11.28 \\ \cline{1-1}  & D + Full & 20.00 & 22.78 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Human evaluation results on WITA (abbreviation for the data source Wikipedia and Wikidata) and DIST-ToTTo (abbreviation for Distantly-supervised Table-To-Text) datasets. The score in the table is calculated as the percentage of time the model is considered better minus the percentage of time it is considered worse. We use Kappa [54] to evaluate the agreement between evaluators. On the WITA dataset, the Kappa values on grammaticality and fidelity are 0.4 and 0.56, respectively. On the DIST-ToTTo dataset, the Kappa values on grammaticality and fidelity are 0.45 and 0.52, respectively. the corresponding text is [PERSON] (born 1 October 1988) is a Dutch professional racing cyclist. Since the text contains incorrect information and hallucination, which means it contains information that is not in the input, it has a low data quality confidence score. We rank the training instances based on the data quality confidence score and purge different percentages of those poorly-aligned training instances with the lowest data quality score. We compare the results with the S2 ST model trained on the full training set (0%) on the WITA dataset in Table 5. It shows that by purging 10% of them, the model is slightly improved on NIST (+2.62% from 8.631 to 8.857), ROUGE\({}_{L}\) (+0.18%, from 73.81 to 73.94) and CIDEr (+1.76%, from 5.045 to 5.134) without interference from those noises. However, after dropping more poorly-aligned training instances, the model's performance drops instead. With only 50% remaining data for training, the BLEU dropped significantly (-14.47%, from 54.52 to 46.63). We hypothesize that while dropping those noisy data can alleviate misguidance, they also hinder the model's ability to learn how to phrase the text with the limited dataset. Our model (S2 ST + Full in Table 2), which is applied to S2 ST, performed better than any settings of dropping noisy data for S2 ST on all five metrics. This indicates our model's ability to alleviate the negative impact of the noisy data. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Dataset** & **Model** & **1-Gram** & **2-Gram** & **3-Gram** & **4-Gram** & **5-Gram** \\ \hline \multirow{8}{*}{WITA} & S2 ST & 624 & 2232 & 2804 & 2770 & 2492 \\  & S2 ST + Meta & 588 & 2158 & 2733 & 2693 & 2406 \\  & S2 ST + Full & 569 & 2110 & 2671 & 2625 & 2343 \\  & DSG & 463 & 2041 & 2635 & 2594 & 2309 \\  & D + Meta & 431 & 1998 & 2606 & 2590 & 2317 \\  & D + Full & 434 & 1970 & 2584 & 2572 & 2294 \\ \hline \multirow{8}{*}{DIST-ToTTo} & S2 ST & 14234 & 27539 & 28954 & 26320 & 22769 \\  & S2 ST + Meta & 12096 & 25496 & 27249 & 24832 & 21343 \\ \cline{1-1}  & S2 ST + Full & 11646 & 24927 & 26880 & 24538 & 21081 \\ \cline{1-1}  & DSG & 10183 & 24437 & 26659 & 24473 & 21098 \\ \cline{1-1}  & D + Meta & 9336 & 22620 & 24629 & 22362 & 18954 \\ \cline{1-1}  & D + Full & 9253 & 22426 & 24479 & 22259 & 18863 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: N-gram statistics for over-generation error analysis.