WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning
Year: 2023

Abstract:
A crucial issue of current text generation models is that they often uncontrollably generate text that is factually inconsistent with inputs. Due to lack of annotated data, existing factual consistency metrics usually train evaluation models on synthetic texts or directly transfer from other related tasks, such as question answering (QA) and natural language inference (NLI). Bias in synthetic text or upstream tasks makes them perform poorly on text actually generated by language models, especially for general evaluation for various tasks. To alleviate this problem, we propose a weakly supervised framework named WeCheck that is directly trained on actual generated samples from language models with weakly annotated labels. WeCheck first utilizes a generative model to infer the factual labels of generated samples by aggregating weak labels from multiple resources. Next, we train a simple noise-aware classification model as the target metric using the inferred weakly supervised information. Comprehensive experiments on various tasks demonstrate the strong performance of WeCheck, achieving an average absolute improvement of 3.3% on the TRUE benchmark over 11B state-of-the-art methods using only 435M parameters. Furthermore, it is up to 30× faster than previous evaluation methods, greatly improving the accuracy and efficiency of factual consistency evaluation.1

heading: 1 Introduction
text: The research of text generation has achieved significant progress in recent years, but it still suffers the main issue of generating output which is factually inconsistent with the given inputs (Maynez et al., 2020). To tackle this issue, various metrics have been designed to check the consistency between
∗Work is done during an internship at Baidu Inc. †Corresponding author.
1Our metric can be easily accessed from https:// huggingface.co/nightdessert/WeCheck
generated text and the given inputs (Kryscinski et al., 2020; Scialom et al., 2021). As we know, how to construct such a metric has attracted increasing attention in a variety of fields (Wu et al., 2022b), including text summarization (Kryscinski et al., 2020; Wu et al., 2022a), dialogue generation (Welleck et al., 2019), and text simplification (Devaraj et al., 2022).
Existing factual metrics can be classified into two types: one based on synthetic data and the other based on task transfer. Synthetic-data based metrics (Kryscinski et al., 2020; Mishra et al., 2021) apply data augmentation techniques to construct factual and non-factual texts as positive and negative samples, respectively. Metrics trained from these synthetic samples often perform poorly due to the significant mismatch between features of actual generated and synthetic text (e.g. distribution of factual errors) (Goyal and Durrett, 2021). Task-transfer based metrics utilize the reasoning ability of models trained on relevant upstream tasks, such as natural language inference (NLI) (Falke et al., 2019; Laban et al., 2022) and question answering (QA) (Wang et al., 2020; Fabbri et al.,
307
2022) and directly apply them to evaluate factual consistency without any adaption.
As described above, previous metrics are learned indirectly from other related resources but without seeing the actual generated text. In such cases, they may overfit to their upstream tasks and fail to generalize to actual generated samples that have significantly different data features. Figure 1 illustrates the probability density of three metrics, where the horizontal axis is metric scores and the vertical axis is the score density. Though these metrics are comparable in performance, they vary significantly in probability distributions, especially in the XSUM dataset, where sample features are greatly different from upstream tasks of these metrics2, NLI-warmup is extremely confident in predicting both very high and low scores while SUMMAC and QAFact are only confident in predicting low scores3. Furthermore, during testing, ensembling different metric scores by simply averaging will further improve their performance (Honovich et al., 2022). This also implies that the evaluation metrics learned from different resources are also complementary.
To bridge the gap between training and testing and mitigate the scarcity of labeled data, in this paper, we propose WeCheck, a factual consistency Checking framework based on Weakly supervised learning. Specifically, WeCheck is based on a learning paradigm that provides weak supervision via modeling multiple label sources without access to ground truth. Different from previous metrics, WeCheck directly utilizes the abundant actual generated samples bootstrapped from models trained on target downstream tasks, e.g. BART on text summarization. Then, WeCheck follows a two-step pipeline consisting of weak annotation and noiseaware fine-tuning to get the target metric model.
In the weak annotation step, by aggregating multiple weak supervision resources, we infer the unknown ground truth label of a sample. To reach this goal, we first provide each sample with a set of weak supervision signals calculated from various other metrics. These metrics are learned from various resources or tasks such as QA-based metrics and NLI-based metrics. After unifying and filtering these signals, we train a generative labeling model that models agreements and disagreements
2In XSum, the summary of each document is abstractive, while existing NLI and QA datasets do not have this feature.
3For more details about these metrics please refer to § 2.3 and §3.2.
between them to infer the likelihood of their latent ground truth label. The inferred ground truth likelihood is then treated as a probabilistic label to provide weak supervision. In the second step, we apply noise-aware fine-tuning to train the target metric model. It is noted here, the weak annotation also brings noises to the supervision signal and brings new challenges to the model optimization process. As a solution, we first warmup our target metric model with NLI data for a better initialization before weakly supervised training. Then, after filtering out samples that are likely to be noisy, we finetune our target metric model with weak annotations. In summary, WeCheck could learn how to utilize multiple resources for weak annotation while recognizing and filtering the potential noises accompanied by weak supervision.
Experimental results show that WeCheck not only achieves state-of-the-art performance but also is computationally efficient. On the TRUE benchmark (Honovich et al., 2022), which is the current most comprehensive benchmark for factual consistency evaluation, WeCheck obtains an average ROC AUC of 84.8, 3.3% absolute improvement over previous 11B pre-trained task transferred metrics with only a size of 435M parameters. Moreover, it’s much more stable for various generation tasks, with much lower variance on different tasks. Thus, WeCheck is a simple but more effective and efficient metric for factual consistency evaluation.
We summarize our contributions as follows:
• We propose a novel factual consistency evaluation metric based on weakly supervised learning, namely WeCheck, which is directly trained on actual generated samples from language models with weakly annotated labels.
• WeCheck is both effective and efficient achieving 3.3% absolute improvement and up to 30 times faster comparing with previous state-ofart metrics.
• WeCheck is a general metric which is also more stable on various generation tasks and datasets than previous methods.

heading: 2 WeCheck Framework
text: Figure 2 illustrates the two-step pipeline of WeCheck framework. In the upper part of the figure, during the weak annotation step, we first calculate a set of weak supervision signals for each
sample bootstrapped from target generation tasks. Then, we use a mapping function to unify the weak supervision signals and infer the likelihood of the ground truth label of each sample. After annotation, we apply noise-aware fine-tuning to train our target metric model, shown in the lower part of the figure. Noise-aware fine-tuning first warmup target metric model with NLI data and training it with filtered probabilistic labels. In the following, we introduce our problem definition and detailed method.

heading: 2.1 Problem Definition
text: Factual Consistency Evaluation Given a textual sequence as a premise, and another textual sequence as a hypothesis, which may be a generated summary or dialogue, the goal of a factual consistency metric fθ is to predict whether the hypothesis is factual consistent given the premise. For simplicity, we follow the previous textual entailment based framework (Kryscinski et al., 2019), which takes x, the concatenation of hypothesis and premise, as the input format and unifies the evaluation as a binary classification problem: fθ(x) ∈ [0, 1], where the predicted logit indicates the probability of x being factually consistent. Another advantage of using the entailment-based framework is that it is effective in terms of time complexity compared with other methods (Laban et al., 2022). Taking fθ as the target metric model, the goal of WeCheck is to train fθ into an efficient factual consistency metric.
Weakly Supervised Training In our weakly supervised settings, we first bootstrap a set of sam-
ples from the generation tasks, e.g. text summarization, and dialogue generation. Using various factual metrics trained from multiple resources, we provide each sample x with a set of weak signals λ = (λ1, . . . , λk), where each λi is a logit separately calculated by a metric. We treat the ground truth label ỹ of x as a hidden variable that can be estimated by aggregating λ. To reach this goal, we train a labeling model pϕ to model agreements and disagreements relations between weak signals in λ and estimate the probability distribution of the truth label, pϕ(ỹ|λ). Then, we apply pϕ(ỹ|λ) to supervise the metric model fθ.

heading: 2.2 Weak Annotation
text: To provide weak supervision for training, we follow data programming (Ratner et al., 2017; Bach et al., 2017), a weakly supervised learning paradigm based on modeling multiple label sources. However, in data programming, weak supervision signals are often produced by various checking clauses, e.g. whether word “causes” appears in the sentence ? and produce a discrete weak signal λi ∈ {0, 1,−1}, where 0/1 stands for a vote for positive/negative label and −1 stands for a abstain vote. However, in our scenario, due to the diversity of metric frameworks, outputs of different metrics often do not share a unified output format and are usually continuous. For example, QA-based metrics often produce continuous logits in [0, 1], and NLI-based metrics often produce discrete labels of entailment or contradiction. Thus, the first thing before training the labeling model is to unify weak supervision signals by a mapping function, m (λi) → {0, 1,−1}. In this way, we can model the transformed λ by a data programming based labeling model.
Weak Signal Unification We first unify all the weak supervision signals from different metrics into the same format, a logit λi ∈ [0, 1]. For the metric with single logit output, we directly use its output as λi. For multi-label classification output, we select the probability of predicting entailment. Notice that all the signals predicted by imperfect metrics will introduce a portion of noises. For a more reliable signal, the core idea for designing a mapping function m is to map signals that the metric has high confidence into {0, 1} and abstain low-confidence signals by mapping them to −1. Generally, this can be achieved by setting thresholds on signals. But another important issue to be
noticed is that, as shown in Figure 1, signal distributions vary significantly across metrics and datasets, which makes threshold selection difficult. Thus, we instead dynamically determine thresholds by setting constant probability mass that contains the highest confidence. Specifically, we choose to map the lowest p− percent and the highest p+ percent of signal scores into label 0 and 1, separately, and map the rest interval of low-confident scores into -1. Given the inverse cumulative distribution function of the i-th signal Fi, we can calculate its positive and negative threshold γ+i and γ − i by: γ+i = Fi(1− p+), γ−i = Fi(p−). (1) The mapping function is then defined by:
m(λi) =    0 λi ≤ γ−i 1 λi ≥ γ+i
−1 γ−i < λi < γ+i . (2)
For simplicity, we share p− and p+ across different resources and datasets. By applying the mapping function, we unify each λi into a discrete label in {0, 1,−1}. Labeling model We treat the true label ỹ of x as a hidden variable and train the labeling model pϕ to estimate ỹ by aggregating λ4. The generative model pϕ models the generation process of λ and ỹ by their joint probability. Because all the weak supervision signals are inferred from different resources, we treat them as independent variables. Then, given the prior p(ỹ)5, the joint probability is formulated by
pϕ(λ, ỹ) = ∏
λi∈λ pϕ(λi, ỹ) =
∏
λi∈λ p (λi|ỹ) p (ỹ) ,
(3) following Bayesian rule. Next, we need to model the likelihood p (λi|ỹ) that labels the sample with λi based on the latent label ỹ. Following (Ratner et al., 2017), we define the labeling process of λi as a sequence of Bernoulli process. Concretely, the i-th metric has a probability of βi not to abstain the sample and a probability αi to label it correctly. Then, we calculate the likelihood by
pϕ(λi|ỹ) =    βiαi λi ̸= −1 ∧ λi = ỹ βi(1− αi) λi ̸= −1 ∧ λi ̸= ỹ
1− βi λi = −1, (4)
4All the weak supervision signals in λ have already been converted into discrete labels by the mapping function m.
5p(ỹ) usually depends on class distribution in a dataset. For simplicity, we set it as a uniform distribution.
where αi, βi are learnable hyper-parameters. Given all samples, we train the labeling model by optimizing:
Lϕ = min ϕ
∑
λ
∑
ỹ∈{0,1} log pϕ(λ, ỹ). (5)

heading: 2.3 Noise Aware Fine-tuning
text: NLI Warmup After we get the labeling model pϕ, the next step is to train our metric model fθ with the weak supervision inferred by it. But in practice, we find direct training with weak supervision will cause the model easily converges to the local minima. This may because reasoning over a long range of context is challenging and weak supervisions are also potential to be noisy. These problems cause great difficulties in optimization. Inspired by the idea of curriculum learning (Bengio et al., 2009), we first warmup our metric model on NLI, an easier and closely related task. We use the mixture of four NLI datasets, MultiNLI (Williams et al., 2018), Fever-NLI (Thorne et al., 2018), LingNLI (Parrish et al., 2021) and Adversarial-NLI (Nie et al., 2020). Based on the warmed-up checkpoint, our metric model achieves much better results under weak supervision, which we will later show in our experiments.
Noise Filtering and Training After warming up, we train our metric model with weak supervision. Because the estimated latent labels ỹ can still be noisy due to the imperfect labeling model and weak supervision signals, we apply the likelihood of ỹ that contains the certainty of the prediction as a soft probabilistic label instead of the discrete label for training. Based on the definition of joint probability in Eq. 3, we predict the likelihood of each sample by
pϕ(ỹ = 1|λ) = pϕ(λ, 1)
pϕ(λ, 1) + pϕ(λ, 0) . (6)
With convenience, we abbreviate pϕ(ỹ = 1|λ) as p(y+). Before training with p(y+), we first filter out estimated samples with low confidence, by applying the similar procedure in weak signal unification. By reusing mapping function m, we filter out the low confident probabilistic label and get the final training set by
X = {( x, p(y+) )∣∣m ( p(y+) ) ̸= −1 } , (7)
where p(y+) is the corresponding probabilistic label of x. Then, given fθ after warming up, we
finetune it by
Lf =min θ
∑
x∈X
[ p(y+) log (fθ(x))
+ (1− p(y+)) log(1− fθ(x)) ] ,
(8)
where p(y+) is kept fixed without gradient backpropagation to pϕ during training.
During inference, the model only needs to take the textual sequence x as input and output the logit prediction fθ(x).

heading: 3 Experimental Settings
text: In this section, we introduce the experimental settings of WeCheck including the evaluation benchmark, baseline models, and implementation details.

heading: 3.1 TRUE Benchmark
text: Recent works point out that the performance of a metric should be evaluated comprehensively across multiple tasks and datasets to reduce variance. Thus, we evaluate WeCheck on TRUE (Honovich et al., 2022), a benchmark consisting of 11 datasets of 4 tasks including text summarization, dialogue generation, paraphrasing, and fact checking, where each sample in datasets is annotated with a binary label manually. We only test on the first three tasks as fact checking is beyond our scope. Following TRUE, we normalize each metric score into a logit and report their Characteristic Area Under the Curve (ROC AUC) w.r.t binary logits. Evaluation with ROC AUC does not require metrics to set specific decision thresholds. Details of tasks and datasets of TRUE are introduce in the Appendix A.

heading: 3.2 Baseline
text: We evaluate WeCheck by comparing with recently proposed metrics. We categorize these baselines by types of their methods.
NLI-based Metrics FactCC (Kryscinski et al., 2020) is a BERT-based metric with synthetic training samples constructed from rule-based data augmentation. SUMMAC(SCZS) (Laban et al., 2022) aggregates sentence-level entailment scores for the final factual consistency score. We only report the zero-shot version SCZS instead of the supervised version SCCONV because it is more effective on the TRUE benchmark. ANLI (Honovich et al., 2022) directly apply a large 11B T5 trained on Adversarial-NLI (Nie et al., 2020) dataset for fact checking and achieve SOTA performance on TRUE.
QA-QG based Metrics QuestEval (Scialom et al., 2021) is a QA-QG based metric that jointly measures factual consistency and semantic relevance, where the importance of generated questions are weighted by a trained model. QAFactEval (QAFact) (Fabbri et al., 2022) is a metric designed by carefully optimizing each component of the QG-QA framework. Q2, from the version of Honovich et al. (2022), replace all the component of QA-QG framework into T5 11B large models.
Other Types BERTScore (BERTS) (Zhang et al., 2019a) measure the similarity of a generated text and its reference by aggregating tokenlevel similarities of their contextual representations. BARTScore (BARTS) (Yuan et al., 2021) evaluate the quality of generated text by its modeling perplexity of a fine-tuned BART (Lewis et al., 2020).

heading: 3.3 Implementation Details
text: All the baseline metrics are tested based on their open-sourced codes. The metric model of WeCheck is based on powerful pre-trained language model DeBERTaV3 (He et al., 2021). Following the description in § 2, we first warm up DeBERTaV3 on NLI datasets and apply it for weak supervised training. As regards to training data, we sample text summarization examples from BART fine-tuned on CNN/DM and XSum datasets. We sample dialogue generation examples from MemNet (Dinan et al., 2018) and dodecaDialogue (Shuster et al., 2020) trained on WoW dataset following Honovich et al. (2021). For paraphrase, we directly use samples in PAWS since it can be regard as a consistency checking dataset itself. For weak signals, we apply QAFact (Fabbri et al., 2022), SUMMAC (Laban et al., 2022), and the NLI warmed up DeBERTaV3 (NLI-warmup) as to provide weak signals for each sample as default. For weak signal unification, we set p+ and p− in mapping function m to 0.75 and 0.25 based on validation. For labeling model pϕ, we follow the implementation of Snorkel (Ratner et al., 2017) for efficiency and train it on CPUs with Adam optimizer. For noise-aware fine-tuning, we finetune the warmed up checkpoint with the learning rate of 1e−6, warmup steps of 500, and the total training steps of 3 epoch. We train on 4 NVIDIA Tesla V100 GPUs, and it takes around only 5000 steps to reach the best performance.

heading: 4 Results
text: The experimental results on TRUE are reported in Table 1, where we report the performance of our model after warmed up training with NLI as NLIwarmup, and further trained with weak supervision as WeCheck. Surprisingly, pre-trained language model trained with only NLI-warmup can achieve 80.3 ROC AUC score, which is a comparable performance with previous best metric. NLI-warmup achieves the second best performance in 5 out of 9 datasets. After further training with weak supervision, WeCheck improves the evaluation performance over NLI-warmup by 4.5 ROC AUC, which not only largely surpasses all the baselines but also outperforms previous SOTA metric SCZS by 3.4 ROC AUC. Separately on each dataset, WeCheck achieves either the best (6 out of 9) or the second best performance in each dataset. Specifically, WeCheck achieves 5.4%, 7.2%, and 1.6% of relative improvements over previous best performing methods on summarization, dialogue and paraphrase, respectively. Furthermore, WeCheck has the lowest variance of 13.2 across different tasks. This demonstrates that the performance of WeCheck is more comprehensive and general rather than biased towards a certain type of data. On the MNBM dataset where samples are very different from NLI or QA data (samples in MNBM are sam-
pled from XSUM, where hypothesis are extremely abstractive), WeCheck largely outperforms previous best metric QAFact by 14.3 point.
11B Baselines We also compare our models with large-scale 11B models based on task transfer. We compare with two models, Q2 and ANLI based on 11B T5 reported by Honovich et al. (2022). As shown in Table 1, they surpass the same type of method with smaller parameter size, and can be regarded as approaching the best performance of task transfer based methods can achieve. However, with only 435M parameters, WeCheck significantly outperforms them by 3-4 points. This further validates the superiority of our weak supervision learning framework.

heading: 5 Analysis
text: To analyse how each module and settings work, we conduct analysis experiments on each module and settings of WeCheck.
Training Mechanism We first study how the mechanisms proposed in §2 affect the overall framework by removing or replacing them. The results are reported in Table 2. Most important of all, by removing the NLI-warmup before weak supervision training, the performance drops significantly on each task and drops an average of 19.3% on each dataset. This proves that NLI, as an easier
and closely related task, provides a much better initialization for training with weak supervision. For noise-aware finetuning, we study how filtering potential noisy samples (Eq. 7) and the probabilistic label (Eq. 6) affect the overall performance. After removing noise filtering (w/o Noise Filter in Table 2), the performance drops around 1-2 points in each task and dataset in average. By replacing the probabilistic labels into hard labels (w/ Hard Label in Table 2), we observe around 0.1-0.2 drops in performance. This implies how to filter potential noisy samples is crucial in noise aware fine-tuning, and probabilistic labels also slightly help.
Effects of Task We also analyse how each bootstrapped task affect WeCheck. In Table 3, the left block rows indicate whether a type of task samples are used for training, and the right block rows are the corresponding performance. The first row is the results of NLI-warmup which does not use any task data for training. The second to forth rows separately train on summarization, dialogue, and paraphrase examples. The last row reports the default settings of WeCheck, which jointly train with all three task samples. From the results, we can conclude that, joint training on all tasks leads to a better performance on the comprehensive evaluation across tasks. For single task evaluation except dialogue, training using only the target task examples leads to better performance on this task than joint training. In horizontal comparisons of single task
performance, we observe that summarization examples contribute most to the overall performance, improving the performance of checking summarization and paraphrase by 6.2 and 3.8 points. Paraphrase examples benefit evaluating paraphrase and dialogue by 6.7 and 1.3 points. Dialogue samples worsen the performance of WeCheck. We suppose that is because these samples are boostrapped from relative weak dialogue models, MemNet and dodecaDialogue, which are not even pre-trained models. Thus, dialogue samples have no contributions to NLI-warmup. By contrast, the summarization samples, which are the most difficult type for checking, benefit most to the overall performance.
Computational Efficiency We analyze the computational efficiency of WeCheck by comparing with other metrics based on different architectures. As reported in Table 4, we select three other representative metrics: SCZS based on sentence-level NLI, FactCC based on document-level NLI, and QAFact based on QA-QG framework. All these methods are tested on the TRUE benchmark with a single NVIDIA 32G V100 GPU and we report the relative time cost of each method comparing with WeCheck6. Despite FactCC is the fastest method reported from the results, its fact checking performance (Table 1) is much worse than others. Among the rest two methods with comparable performance, WeCheck is 2.9 times faster than SCZS and 30 times faster than QAFact.
Abstractiveness As mentioned above, abstractive hypotheses are challenging for current metrics, e.g. XSUM summaries from MNBM. We give an in-depth analysis of the effect of hypothesis abstractiveness on the metrics performance. Following See et al. (2017), we use the percentage of unique unigrams in a hypothesis w.r.t its premise to measure abstractivenss. Then, we spilt all the examples in TRUE into 10 bins according to their abstractiveness. For each bin, we measure the ROC AUC of WeCheck and the other three representative baselines: QAFact, Summac, and NLI-warmup. From the results in Figure 3, we observe a significant drop in the performance for all baselines as the hypothesis becomes more abstractive, while, WeCheck keeps its performance (around 0.85). Moreover, WeCheck consistently outperforms baseline metrics in every bin of ab-
6The batch size of each metric is set to the maximum size that the GPU memory can hold.
stractiveness. This further verifies the superiority of directly training with real task data.

heading: 6 Labeling Model
text: We compare how different data programming based labeling models affect the final metric performance. In WeCheck, labeling model pϕ learns to aggregate multi-resource labels to infer the hidden true label. Comparing concretely, our method is similar to Snorkel (Ratner et al., 2017). Because, in our scenario, the number of weak supervision signals is small and their relationships are relatively simple as they are trained from different tasks, we prefer this method over other recent more advanced ones.
In Table 5, we demonstrate the effectiveness of our labeling model by replacing it with other methods. In these baselines, simpler methods include: Average Signals, which simply averages all the weak signals as the probabilistic label p(y+); Major Vote, which select the most frequently appeared label in a unified weak signal set as the true label. More advanced methods include: Flying Squid (Fu et al., 2020), which applies an Ising model (Parsons, 2011) to model more complex relations in a unified weak signal set; Weasel (Cachay et al., 2021) is the current SOTA data programming
method, which uses a neural network as the labeling method and trains it end-to-end with the target tasks model; DWS (Parker and Yu, 2021) treats the true label of a sample as the hidden variable and applies Estimation-Maximization (EM) for inference during training.
From the results in Table 5, our default labeling model outperforms all others. Furthermore, more complex methods (Flying Squid, Weasel, and EM) perform worse than simpler methods (Ours, Average Signal, and Major Vote). This further verifies that the relations between weak signals are simple, and complex modeling will not bring further improvements. From another perspective, overly simplistic approaches without any statistical modeling (Average Signal and Major Vote) also perform worse than our methods.

heading: 7 Related Work
text: Factual Consistency Evaluation Recently, automatically checking factual consistency has become an increasingly popular topic (Li et al., 2022). Reasoning over a long range of context for factual evaluation is a challenging task that even human annotators may frequently disagree with each other (Pagnoni et al., 2021). Thus, it is hard to collect a large-scale high-quality dataset for training a fully supervised model, and previous works search for indirect methods. One branch of them leverage the reasoning ability of NLI. Based on the model trained on NLI datasets, e.g. MNLI (Williams et al., 2018), ANLI (Nie et al., 2020), some works aggregate sentence-level entailment score for checking (Falke et al., 2019; Laban et al., 2022), while others adopt document-level NLI which directly reasoning over the full context (Maynez et al., 2020; Gehrmann et al., 2021). Another branch of methods apply QA-QG based pipeline for a more fine-grained checking. QAGS
(Wang et al., 2020) and FEQA (Durmus et al., 2020) are the earliest attempt on this method, and QuestEval (Scialom et al., 2021) and QAFactEval (Fabbri et al., 2022) further improve this type of methods by applying NLI for answer matching.
Data Programming In this paper, we mainly focus on data programming (Ratner et al., 2016) (DP), a weak supervision paradigm proposed to infer correct labels based on noisy labels from labeling functions (LFs), which are rule-based decision-making processes that generate discrete labels. Following the DP paradigm, Snorkel (Ratner et al., 2017) is proposed to for rapid training, more recent works study how to adapt label model in DP (Ratner et al., 2019; Awasthi et al., 2020) or modeling more complex structure between LFs (Fu et al., 2020). DP is also applied to several NLP tasks. DWS (Parker and Yu, 2021) combine DP and CRF for weakly supervised named entity recognition, Min et al. (2019) apply DP for QA. Different from all previous tasks, our weak supervision signals are logits from other models, rather than discrete labels generated from rules.

heading: 8 Conclusion
text: In this paper, we propose a weakly supervised framework, WeCheck, which aggregates weakly supervised signals from multiple resources and trains a target metric model in a noise-aware manner. Different from previous metrics that trains from synthetic data or transferred from other tasks, WeCheck directly trains with the real generated text. WeCheck first annotates each sample with a probabilistic label via a labeling function that aggregates multiple resources. Then, in the noise-aware finetuning stage, WeCheck applies probabilistic labels to train the target metric model. Experimental results show that, WeCheck not only surpass previous methods in performance but also time efficient. Moreover, WeCheck is potential to be compatible with future more stronger metrics, bring further improvements to the overall performance.
Limitations
Hyper-parameters Selection Some hyperparameters still acquire careful selection for WeCheck, e.g. p+, p−. Also, using different set of hyper-parameters for different tasks and datasets will further boost performance. Thus, we need to train the model several time and select the best performing parameters based on validation.
End-to-End Training WeCheck applies the weak annotation and noise-aware fine-tuning twostep pipeline, where the noises in the first step will greatly affect the performance of the second step. By modifying the overall framework into end-toend training will solve this problem.

heading: Acknowledgement
text: This work was partially supported by National Key R&D Program of China (No. 2022YFC3600402) and National Social Science Foundation Project of China (21&ZD287).

heading: A True Benchmark
text: The TRUE benchmark is composed of the following tasks and datasets.
Abstractive Summarization FRANK (Pagnoni et al., 2021) collect annotations for modelgenerated summaries on the CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018) datasets, resulting in 2250 annotated system outputs. SummEval (SumE) (Fabbri et al., 2021) collect human judgments for 16 model outputs on 100 articles taken from the CNN/DM dataset. MNBD (Maynez et al., 2020) sample 500 articles and annotate summaries generated by four different systems on XSum, as well as the gold summaries. QAGS (Wang et al., 2020) collect 474 generated summaries for CNN/DM and XSum, where each sample is annotated by three annotators.
Dialogue Generation BEGIN (Dziri et al., 2021) is a dataset for evaluating the factual consistency of knowledge-grounded dialogue systems. Dialogue responses are generated by fine-tuning two systems on Wizard of Wikipedia (WoW) (Dinan et al., 2018) dataset. Q2 (Honovich et al., 2021) annotate 1,088 generated dialogue responses from two dialogue models trained on WoW. DialFact (DialF) (Gupta et al., 2022) introduce a tasks of dialogue fact-verification and propose a conversation clams dataset grounded on Wikipedia. In TRUE benchmark, one only need to verify weather a conversation claim is correct given its grounding.
Paraphrase Detection PAWS (Zhang et al., 2019b) construct a paraphrase identification with paraphrase and non-paraphrase pairs from Wikipedia and the Quora Question Pairs (QQP). In True benchmark, only samples from Wikipedia are applied for verification.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 9

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 9

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In the introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4,5,6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.3
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? single run
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Moreover, WeCheck is potential to be compatible with future more stronger metrics, bring further improvements to the overall performance. Limitations
Hyper-parameters Selection Some hyperparameters still acquire careful selection for WeCheck, e. g. p+, p−. Also, using different set of hyper-parameters for different tasks and datasets will further boost performance. Thus, we need to train the model several time and select the best performing parameters based on validation. End-to-End Training WeCheck applies the weak annotation and noise-aware fine-tuning twostep pipeline, where the noises in the first step will greatly affect the performance of the second step. By modifying the overall framework into end-toend training will solve this problem