# Quality Control for Distantly-Supervised Data-to-Text Generation via Meta Learning
## 5 Experiments
### Results
#### 5.2.3 Human Evaluation  
We compare the S2 ST, S2 ST + Full, DSG and D + Full model performance on the test set for both the WITA and DIST-ToTTo datasets in human evaluation. We sampled 50 generated texts on the test set for each model, arranged the results of those models into pairs and asked three human raters to determine which one in the pair performed better in terms of grammaticality (which one is more fluent and grammatical?) and fidelity (which one is more faithful to the data?).  
If the human raters deem those texts generated from two different models are of the same quality, they are allowed to assign a 0.5 score to each of the two models. Each example was evaluated by three different graduates, who are proficient in English, and we report the average of three human raters' results in Table 3. The reported result is the subtraction of the percentage of time a system is considered better and when considered worse. It can be  
\begin{table}
\begin{tabular}{c c c c c c c} \hline
**Dataset** & **Model** & **BLEU** & **NIST** & **METEOR** & **ROUGE\({}_{L}\)** & **CIDEr** \\ \hline \multirow{8}{*}{WITA} & S2 SL & 48.08 & 8.459 & 38.75 & 70.92 & 4.415 \\  & S2 SG & 47.78 & 8.267 & 38.31 & 72.21 & 4.543 \\  & S2 ST & 54.52 & 8.631 & 41.85 & 73.81 & 5.045 \\  & DSG & 56.69 & 9.241 & 43.09 & 76.26 & 5.380 \\  & S2 ST + Meta & 55.33 & 8.925 & 42.25 & 74.86 & 5.150 \\  & S2 ST + Full & 56.34 & 9.097 & 42.75 & 75.71 & 5.255 \\  & D + Meta & 58.00 & 9.304 & 43.85 & 76.72 & 5.486 \\  & D + Full & 58.80 & 9.341 & 44.10 & 77.14 & 5.573 \\ \hline \multirow{8}{*}{DIST-ToTTo} & S2 SL & 25.25 & 6.555 & 26.23 & 48.32 & 2.140 \\  & S2 SG & 24.41 & 6.727 & 25.73 & 47.72 & 2.050 \\ \cline{1-1}  & S2 ST & 35.91 & 8.343 & 31.64 & 54.89 & 2.697 \\ \cline{1-1}  & DSG & 38.22 & 8.323 & 33.09 & 57.60 & 2.950 \\ \cline{1-1}  & S2 ST + Meta & 38.52 & 8.869 & 33.11 & 57.14 & 2.898 \\ \cline{1-1}  & S2 ST + Full & 39.10 & 8.931 & 33.82 & 57.61 & 2.944 \\ \cline{1-1}  & D + Meta & 39.03 & 8.690 & 33.70 & 58.16 & 3.019 \\ \cline{1-1}  & D + Full & 39.68 & 8.785 & 34.02 & 58.68 & 3.084 \\ \hline \end{tabular}
\end{table}
Table 2: Automatic evaluation results on both datasets’ test set. We applied our approach to the base model S2 ST (abbreviation for Sequence-to-Sequence Transformer), which is represented as S2 ST + Full and the state-of-the-art DSG (abbreviation for Distant Supervision Generation) model, noted as D + Full. Furthermore, we include the ablation results in this paper by only applying the corpus reweight module to the corresponding model (S2 ST + Meta and D + Meta). The results show that both the proposed corpus reweight and corpus rewrite modules can improve the base model S2 ST and the state-of-the-art model DSG across all five metrics on both WITA (abbreviation for the data source Wikipedia and Wikidata) and DIST-ToTTo (abbreviation for Distantly-supervised Table-To-Text) datasets. Particularly, our model (D + Full) can improve the state-of-the-art model DSG by 3.72% on the WITA dataset and 3.82% on the DIST-ToTTo dataset in terms of the widely used metric BLEU (abbreviation for BiLingual Evaluation Understudy).  
seen that our model can help improve the S2 ST and DSG in terms of both grammaticality and fidelity on both the WITA and DIST-ToMTo datasets, respectively.  
For example, on the WITA dataset, our model (D + Full) obtained scores 12.67 and 11.33, in terms of grammaticality and fidelity, respectively. Both exceed the state-of-the-art model DSG's 11.33 and 7.44, respectively. On the DIST-ToTTo dataset, these conclusions still stand in terms of both grammaticality and fidelity. These results support our claim that reweighting training corpus based on each data-text pair's quality via meta learning and rewriting those poorly-aligned data-text pairs during training can help the model learn to generate more faithful text. Overall, our model can produce faithful text while keeping relatively good grammaticality.