Data Augmentation
Another direction we explore is improving generalization via data augmentation, which has recently shown promising results in other NLP tasks such as question answering Yue et al. ( 2021 ) and machine translation Cheng et al. ( 2020 ) .
We first train a claim generation model based on the BART Lewis et al. ( 2020 ) , using the (evidence, claim, label) triples in the source domain as training data. We use the format [LABEL] label [NER] NERs [EVIDENCE] evidence as the input, and use the claim as the target output for training, where NERs are the entities appearing in the claim (we add NERs to guide the model to generate more specific claims). We then apply the trained model to generate claims with different labels in the target domain by separately assigning supports , refutes , NEI as the label prefix of the evidence, and we randomly assign an entity from the evidence as the NERs 4 4 4 Since no ground-truth claim is available for the target domain, the entity cannot come from the claim. to guide the claim generation. We name this method as BART-gen .

We train BART-gen on the FEVER-sent dataset and generate synthetic training (evidence, claim, label) triples for other datasets. For each piece of evidence in the target domain, we generate six claims with different ( label , NERs ) combinations. Table 8 shows the zero- and few-shot generalization results for BART-gen and other baselines: 1) FEVER-full : the model is trained on the original FEVER-sent dataset; 2) FEVER-control : the model is trained on a random subset of FEVER-sent which has the same amount of data with the generated data; 3) BART-gen : the model is trained on the generated data. For the few-shot setting, the model is further fine-tuned with 150 in-domain samples.

For the zero-shot setting results in Table 8 , BART-gen consistently improves the generalization performance compared with FEVER-full (+24.9% in average) and FEVER-control (+47.4% in average). The results show that training with generated target data is in general more effective than directly generalizing a model trained on the source data. This is better reflected by comparing BART-gen with FEVER-control in which the data amount is the same. The improvement is especially noticeable for PubHealth , probably because it lacks the NEI claims in its original training set. Data augmentation addresses this by generating a sufficiently balanced number of claims for each label.

However, our human evaluation in Appendix E shows that around 30% of generated claims suffer the label inconsistency problem, i.e. , the BART-gen often generates a fluent claim that does not match our desired label (for example, we want to generate a refutes claim, but the generated claim is actually NEI ). Label inconsistency may introduce conflicting information between the pretraining and fine-tuning stages, which we hypothesize is the cause for the lower level of improvement in fine-tuning the model on the generated data, compared with fine-tuning the model on FEVER. Therefore, although data augmentation is a promising direction to improve generalization, it remains a challenging problem regarding how to generate high-quality claims with consistent labels.