FRACTAL: Fine-Grained Scoring from Aggregate Text Labels
Authors: Yukti Makhija, Priyanka Agrawal, Rishi Saket, Aravindan Raghuveer
Accepted at: acl25

FRACTAL: Fine-Grained Scoring from Aggregate Text Labels

Yukti Makhija*  Priyanka Agrawal*  Rishi Saket*  Aravindan Raghuveer*
*Google DeepMind
{yuktimakhija, priyankagr, rishisaket, araghuveer}@google.com

Abstract

Fine-Tuning of LLMs using RLHF / RLAIF has been shown as a critical step to improve the performance of LLMs in complex generation tasks. These methods typically use response-level human or model feedback for alignment. Recent works indicate that finer sentence or span-level labels provide more accurate and interpretable feedback for LLM optimization. In this work, we propose FRACTAL, a suite of models to disaggregate response-level labels into sentence-level (pseudo-)labels through Multiple Instance Learning (MIL) and Learning from Label Proportions (LLP) formulations, novel usage of prior information, and maximum likelihood calibration. We perform close to 2000 experiments across 6 datasets and 4 tasks that show that FRACTAL can reach up to 93% of the performance of the fully supervised baseline while requiring only around 10% of the gold labels. Furthermore, in a downstream eval, employing step-level pseudo scores in RLHF for a math reasoning task leads to 5% absolute improvement in performance. Our work is the first to develop response-level feedback to sentence-level scoring techniques leveraging sentence-level prior information, along with comprehensive evaluations on multiple tasks as well as end-to-end finetuning evaluations.

1 Introduction

Large language models (LLMs) are being increasingly used for various generation tasks like generating text (Gero et al., 2022), seeking facts, answering complex queries (Adiwardana et al., 2020; Menick et al., 2022), and performing logical reasoning tasks (Kojima et al., 2022). The improvements to LLMs rely heavily on their evaluation using feedback from humans or automated model-based scoring (Ouyang et al., 2022; Touvron et al., 2023). However, such feedback has typically been taken at the response level, enabling efficient and cost-effective assessments of overall output quality.

Recent evidence (Amplayo et al., 2022) suggests that the sentence or step-level evaluation is more reliable over response-level evaluation. Moreover, Wu et al. (2023) have shown that collecting finer-grained human feedback results in considerably improved RLHF, as such feedback precisely localizes the strengths and weaknesses within a generated

<table>
  <tr>
    <th>Task</th>
    <th>Retrieval | MultiSpanQA Dataset</th>
  </tr>
  <tr>
    <td>Input Query</td>
    <td>Who were the last people to be hanged in england?</td>
  </tr>
  <tr>
    <td>Retrieved Context (Response)</td>
    <td>Nevertheless, there remained a working gallows at HMP Wandsworth, London, until 1994, which was tested every six months until 1992. This gallows is now housed in the Galleries of Justice in Nottingham. The last execution by hanging in England was on 13 August 1964, Peter Anthony Allen, at Walton Prison in Liverpool, and Gwynne Owen Evans, at Strangeways Prison in Manchester, were executed for the murder of John Alan West on 7 April that year. Scotland - Henry John Burnett, 21, on 15 August 1983 in Craigigiches Prison, Aberdeen, for the murder of seaman Thomas Guyan.</td>
  </tr>
  <tr>
    <td>Gold Response Label</td>
    <td>i.e. Retrieved Context is relevant to the Input</td>
  </tr>
  <tr>
    <td>Sentence Labels - FRAC-TAL</td>
    <td>[0, 0, 1, 0]<br>i.e. third sentence contains the answer</td>
  </tr>
</table>

<table>
  <tr>
    <th>Task</th>
    <th>Question Answering | QA-Feedback Dataset</th>
  </tr>
  <tr>
    <td>Input Query</td>
    <td>What is the name of the pirate in spongebob?</td>
  </tr>
  <tr>
    <td>Input Context</td>
    <td>SpongeBob SquarePants - Puff endures one of SpongeBob's crashes or is otherwise frightened, she puff's up into a ball. Special episodes of the show are hosted by a live action pirate named Patchy and his pet parrot Potty.</td>
  </tr>
  <tr>
    <td>Response 1</td>
    <td>The SpongeBob SquarePants is an American animated television series based on the film series SpongeBob SquarePants. A special pirate named Patchy and his pet parrot Potty, who host special episodes of the show, are presented in a dual narrative with the animated stories. Burger Beard is the name of the pirate in the SpongeBob SquarePants movies.</td>
  </tr>
  <tr>
    <td>Response 2</td>
    <td>Burger SquarePants Out of Water is a 2015 American live-action/animated absurdist comedy film based on the animated television series SpongeBob SquarePants. It follows a pirate named Burger Beard, who steals the Krabby Patty secret formula from the Krusty Krab, and makes any text written upon it come true. SpongeBob and his friends must stop the surface to confront Burger Beard and get the formula back.</td>
  </tr>
  <tr>
    <td>Gold Preference Label</td>
    <td>i.e. Response 1 is better than Response 2</td>
  </tr>
  <tr>
    <td>Sentence Labels - FRAC-TAL</td>
    <td>Response 1: [1, 1]; Response 2: [1, 0, 0]<br>i.e. Sentences 2 and 3 of Response 2 are inaccurate.</td>
  </tr>
</table>

Table 1: Our method, FRACTAL, disaggregates response-level gold label into fine-grained sentence-level labels. These examples are for the tasks of Retrieval and Question Answering from MultiSpanQA and QA-Feedback datasets respectively.
response. It further provides greater interpretability, allowing for more targeted LLM fine-tuning by highlighting the specific portions of a response that contribute to its overall quality. The work of Lightman et al. (2023) came to a similar conclusion for multi-step reasoning tasks, showing improved performance of models fine-tuned using process-supervised reward models (PRMs) trained on step-level labels, as compared to outcome-supervised reward models (ORMs) trained only on the correctness of the entire solution, on the MATH dataset (Hendrycks et al., 2021a). However, collecting fine-grained annotations adds significant cost due to the added quantity and precision of human labor needed. Even in situations where it is feasible to directly collect fine-grained feedback, doing so for the Side-by-Side (SxS) feedback (Ouyang et al., 2022) remains challenging.

In this paper, we argue that it is possible to convert coarse response / paragraph level labels provided by humans into fine-grained sentence level labels. We propose FRACTAL, a suite of modeling based techniques to dis-aggregate response-level labels into sentence-level pseudo-labels that accurately reflect the underlying quality distribution within a larger response. We show across 6 datasets and 4 tasks that FRACTAL can reach upto 93% of performance of a sentence-level model that uses 10X the number of labels as the FRACTAL model. Table 1 shows two examples of how FRACTAL converts a gold response label into precise sentence level labels. We also apply FRACTAL to a the fine-grained RLHF setting and show FRACTAL is able to better glean information from outcome labels for math reasoning and thereby provide a 5% boost to performance over vanilla response RLHF. To the best of our knowledge, FRACTAL is the first approach to comprehensively study the task of fine-grained scoring from aggregate text and demonstrate practical applicability on Fine-Grained RLHF.

As in supervised training, the first component of FRACTAL is a methodology to train a model on the response-labels to predict the scores (label probabilities) for sentences. For this we leverage and build upon techniques from multiple instance learning (MIL) and learning from label proportions (LLP) (see Section 2 for previous work on MIL and LLP). These have been used to train predictive models on datasets partitioned into bags or sets of instances. For the text-generation tasks, we model each response as a bag and its instances are the constituent sentences of the response. Each bag has an aggregated label i.e., bag-label which is assumed to be derived from the (unknown) instance-labels of the bag via an aggregation function. The instance-labels correspond to the sentence-labels and the aggregated bag-label is the aggregated response-label. In MIL, the aggregation is the MAX or MIN of binary or ordinal instance-labels – applicable to question-answering (relevance), summarization and math reasoning tasks – while LLP, which models the bag-label as the AVG of the instance-labels, is applicable to retrieval tasks. To estimate the instance-labels from bag-labels, a standard technique in MIL and LLP is bag-loss which minimizes a loss between the bag-labels and the aggregated instance-predictions, summed over all bags. While bag-loss is usually a strong baseline, the use of only bag-labels is seen to be insufficient supervision. To this extent, we propose use of prior distribution on the instance-label modeled from application-specific information.
We make the following contributions:
1. We propose a novel framework, FRACTAL, to disaggregate response labels into constituent sentence-labels. We formulate the fine-grained prediction as an MIL and LLP task. This abstraction allows us to leverage baselines proposed for MIL/ LLP . Through our experiments, we show that this formulation alone is not sufficient to demonstrate strong performance.
2. We introduce enhancements by adding priors over the instances. We propose to add two types of priors for every instance in a bag based on document-sentence similarity scores and correlations between sentences. The baseline MIL / LLP methods are augmented with prior information as new loss terms as show in Section 4.1. To the best of our knowledge, use of such priors to improve performance of MIL/LLP methods has not been studied before.
3. We develop pseudo-labeling strategies to calibrate instance-level model predictions into labels which are consistent with the response-level labels, allowing us to train the model on the derived pseudo-labels (refer Section 4.2). Our ablation experiments demonstrate that both the prior inclusion and pseudo-labeling steps significantly help improve performance.
4. In order to study the performance of FRACTAL, we formulate a wide variety of tasks shown in Section 5: retrieval, question answering, summarization, and math reasoning across
6 datasets. We define the required formulations for disaggregating response-level labels to sentence-level labels applicable to these tasks.
5. Through thorough experiments in Section 6, we show FRACTAL generalizes well across task objectives and aggregation functions. In intrinsic evals, FRACTAL achieves up to 93% of the performance of the fully supervised baseline while requiring only around 10% of the number of labels. In an extrinsic evals, FRACTAL also substantially improves over the performance of preference and response RLHF through fine grained labeling.

2 Related Work

Multiple Instance Learning (MIL). Here the bag label is modeled as MAX or MIN of the its (unknown) instance-labels (typically labels are {0, 1}-valued). Given a such a dataset of bags the goal is to train a model either to predict the bag-labels or in many cases the label of instances. Introduced to model drug activity detection by Dietterich et al. (1997), MIL has been applied to domains like drug discovery (Maron and Lozano-Pérez, 1997), time series prediction (Maron, 1998), retrieval (Lozano-Pérez and Yang, 2000), medical imaging (Wu et al., 2015a) and analysis of videos (Babenko et al., 2009; Sikka et al., 2013). On the theoretical side, Blum and Kalai (1998) showed that MIL PAC learning can be achieved using noise tolerant instance-level PAC learnability, while Sabato and Tishby (2012) gave generalization bounds for the classification error on bags. Ramon and De Raedt (2000) proposed a bag-loss method using log-sum exponential approximation to MIN, which was followed by adaptations of boosting and logistic regression (Zhang et al., 2005; Ray and Craven, 2005), while specialized methods such as diverse-density (DD) (Maron and Lozano-Pérez, 1997) and and its EM-based variant, EM-DD (Zhang and Goldman, 2001) have also been developed. More recent works have proposed deep learning methods for MIL based on convolutional and attention mechanisms (Wu et al., 2015b; Ilse et al., 2018).

Learning from Label Proportions (LLP). Here the bag-label is the average of the instance-labels, and the goal is to train a model to predict the labels of instances. LLP arises in the context of label privacy concerns (Rueping, 2010), costly supervision (Chen et al., 2004) or lack of labeling instrumentation (Dery et al., 2017). Early work developed Monte-Carlo (de Freitas and Kück, 2005; Hernández-González et al., 2013), clustering (Chen et al., 2009; Stolpe and Morik, 2011), and k-NN and SVM (Musicant et al., 2007) based adaptations of traditional supervised learning techniques. Subsequently, specialized techniques were developed by Quadrianto et al. (2009); Patrini et al. (2014) who estimated model parameters from bag-labels, and by Yu et al. (2013) who proposed a customized SVM for LLP. More recent works have developed methods based on the distribution and processing of bags (Scott and Zhang, 2020; Saket et al., 2022; Zhang et al., 2022; Chen et al., 2023; Busa-Fekete et al., 2023) and methods for training deep networks on bags (Kotzias et al., 2015; Liu et al., 2019; Dulac-Arnold et al., 2019; Nandy et al., 2022), specifically Ardehaly and Culotta (2017) proposed the bag-loss method which is commonly used as a baseline. Practical applications of LLP include image classification (Bortsova et al., 2018; Ørting et al., 2016), IVF predictions (Hernández-González et al., 2018), and privacy preserving modeling in online advertising (O’Brien et al., 2022).

MIL and LLP for NLP. Applications such as sentiment analysis (Pappas and Popescu-Belis, 2014; Angelidis and Lapata, 2018) and document modeling (Pappas and Popescu-Belis, 2017) have previously admitted MIL techniques, while more recently (Liu et al., 2022) modeled offensive language detection as an MIL problem and proposed a mutual attention based mechanism. On the other hand, the applications of LLP are relatively sparser: (Ardehaly and Culotta, 2016) applied it to domain adaptation for text data, while recent work (Chauhan et al., 2023) proposed a novel method improving on the baseline model training technique of (Ardehaly and Culotta, 2017) for text classification.

For both MIL and LLP, previous works have proposed pseudo-labeling based model training methods, in which the weak-supervision of bag-labels is used along with model predictions to derive pseudo-labels which can be used to train or fine-tune models. For e.g. pseudo-labels are computed via regularization (Wang et al., 2023; Liu et al., 2021) or expectation-maximization (Luo et al., 2020; Baruicic and Kybic, 2022) techniques.
Figure 1: Overview of our proposed method, FRACTAL. Input is a set of responses each with a response label. A response is a bag of sentences. The output is a model that can predict the score for each sentence in a response. The semantic meaning of a score depends on how the response label was defined. FRACTAL consists of two key components a) Bag-level loss functions and model training (Section 4.1), and b) Max-Likelihood Pseudolabeling (Section 4.2)

3 Preliminaries

As mentioned in Section 1, we consider text-generation tasks where each response can be modeled as a bag whose instances are the constituent sentences of the response. The bag-label is the response-label while the sentence-labels correspond to the respective instance-labels. In the following, we formally define the notion of instances, bags and their labels.

Let \( \mathcal{X} \) be the underlying set of instances and \( \mathcal{Y} \) be the label-set which is \( \{0, 1, \ldots, C\} \) for some \( C \in \mathbb{Z}^+ \) where \( C = 1 \) for binary and \( C > 1 \) for integer labels respectively. A dataset is a collection of labeled instances.

A bag \( B \) is a subset of \( \mathcal{X} \) and \( y_B \) denotes its label which is thought to depend on the labels of the instances in \( B \) via an aggregation function AGG which maps tuples with elements from \( \mathcal{Y} \) to \( [0, C] \). Specifically, if \( B = \{x_1, \ldots, x_k\} \) and \( y_i \) is the label of \( x_i \) (\( i \in [k] \)), then \( y_B = \mathrm{AGG}\,(y_1, \ldots, y_k) \). Typically, AGG is either MIN, MAX or AVG.

We consider prior information about the labels on individual instances, for e.g. through unsupervised modeling. For each \( x \) in the dataset, its point-prior \( p_x \in [0, 1] \) is prior for \( y_x / C \) where \( y_x \) is the underlying label of \( x \) and \( \{0, \ldots, C\} \) is the label set as defined above. The pair-prior for a pair \( (x, z) \) of instances is given by \( p_{xz} \in [0, 1] \) and measures the correlation between \( x \) and \( z \). See Appendix K for a discussion on these priors. Section 5 and Table 2 provide the specifics of the prior information for various datasets and tasks in our experiments.

Some applications provide preference bag-labels which encode comparisons between pairs of bags. Specifically, for a pair of bags \( (B_1, B_2) \) the preference bag-label \( y_{B_2 > B_1} \) is 0 if \( y_{B_1} > y_{B_2} \), and 1 if \( y_{B_1} < y_{B_2} \). There are no bag-labels, only preference bag-labels for some pairs of bags. In the description of our techniques we will for convenience use CE to denote the cross entropy loss.

Modeling Task. Given as input a collection \( \mathcal{B} \) of pairwise-disjoint (i.e., non-overlapping) bags along with their bag-labels (or preference bag-labels), possibly along with the priors \( \{p_x\} \) or \( \{p_{xz}\} \), the goal is to output a model predicting a score for each instance in \( \mathcal{X} \). For preference labels, the evaluation is in terms of the accuracy of the preference labels assigned by the model on a test set of bags.

4 Our Techniques

We present the components of the FRACTAL method along with the BagLoss baseline approach (see Figure 1). The two main components of FRACTAL are (i) model training using bag-level loss functions which incorporate the priors as defined in Section 3, and a (ii) pseudo-labeling technique to use the model predictions to provide instance-level pseudo-labels using which the final model training is trained.

4.1 Training with bag-loss and priors

We train a model \( \mathcal{M} \) on the collection \( \{B \in \mathcal{B}\} \) of training bags \( B \) with aggregate labels \( y_B \). The prediction \( \mathcal{M}(x) \) of the model on any instance \( x \) is a probability distribution over the label-set \( \{0, \ldots, C\} \), and we denote the probability of label \( \ell \) by \( \mathcal{M}(x)[\ell] \). Also, by \( \hat{y}(x) \) we denote the soft-label \( \sum_{\ell \in \{0, \ldots, C\}} \ell \cdot \mathcal{M}(x)[\ell] \), assigned by the model to \( x \). Let probAGG be an extension of AGG to sequences of model-predictions. In particular, probAGG maps sequences \( (\mathcal{M}(x_1), \ldots, \mathcal{M}(x_k)) \) to a probability
distribution over \( \{0, \ldots, C\} \), for \( k \in \mathbb{Z}^+ \). For MIN, probAGG \( (\mathcal{M}(x_1), \ldots, \mathcal{M}(x_k)) = \mathcal{M}(x_{i^*}) \) where \( i^* := \operatorname{argmin}\left(\tilde{y}(x_1), \ldots, \tilde{y}(x_k)\right) \) is the index of the instance with the minimum soft-label. We provide additional details about probAGG for the tasks used in our work in Appendix A.

The baseline BagLoss method optimizes the following loss, for a model \( \mathcal{M} \):

\[
L_{totbag}(\mathcal{B}, \mathcal{M})
:= |\mathcal{B}|^{-1} \sum_{B \in \mathcal{B}} \mathrm{CE}\left(y_B, \mathcal{M}(B)\right)
\] (1)

where \( \mathcal{M}(B) = \operatorname{probAGG}\left((\mathcal{M}(x))_{x \in B}\right) \) is the aggregate prediction of \( B \), and \( \mathcal{M}(B)[\ell] \) is probability of label \( \ell \).

**PriorsBagLoss.** In our bag loss with priors method, we have an additional loss which incorporates the priors. For point-priors \( \{p_x\} \) we have:

\[
L_{totptnprior}(\mathcal{X}, \mathcal{M})
:= |\mathcal{X}|^{-1} \sum_{x \in \mathcal{X}} \mathrm{CE}\left(p_x, \tilde{y}(x)/C\right)
\] (2)

while for pair-priors \( \{p_{xz}\} \) the loss is

\[
L_{totprprior}(\mathcal{X}, \mathcal{M})
:= |\mathcal{X}|^{-2} \sum_{(x,z) \in \mathcal{X}^2} L_{prprior}\left(p_{xz}, \tilde{y}(x), \tilde{y}(z)\right)
\] (3)

where \( L_{prprior} \) is a specialized loss.

\[
L_{prprior}\left(p_{xz}, \tilde{y}(x), \tilde{y}(z)\right) :=
\left| p_{xz} - \frac{\tilde{y}(x)}{C} \frac{\tilde{y}(z)}{C} \right|
\cdot \left| p_{xz} - \left(1 - \frac{\tilde{y}(x)}{C}\right) \left(1 - \frac{\tilde{y}(z)}{C}\right) \right|
\] (4)

We assume that \( p_{xz} \) represents the probability of instances x and z having the same label. A low \( p_{xz} \) value indicates that these instances likely belong to different classes. We design a loss function that is minimized when the predicted classes for both instances are consistent with the value of the prior.

The total loss in **PriorsBagLoss** is a convex combination of bag and prior losses:

\[
L_{totPB} = \lambda L_{totbag} + \lambda_1 L_{totptnprior} + \lambda_2 L_{totprprior}
\] (5)

for some \( \lambda, \lambda_1, \lambda_2 \in [0, 1] \) s.t. \( \lambda + \lambda_1 + \lambda_2 = 1 \). The values \( \lambda, \lambda_1 \) and \( \lambda_2 \) are selected through a hyperparameter search which is also described in the *Model Training Setup* part of Section 6.

<table>
  <tr>
    <th colspan="2">Input:</th>
    <th>Model \( \mathcal{M} \), bags \( \mathcal{B} \), batch size \( q \), learning rate \( lr \), optimizer opt, weights \( \lambda, \lambda_1, \lambda_2 \), epochs \( N \).</th>
  </tr>
  <tr>
    <td colspan="2">1.</td>
    <td>Order bags \( \mathcal{B} \) in a random order \( \pi \).</td>
  </tr>
  <tr>
    <td colspan="2">2.</td>
    <td>For epoch \( j = 1 \) to \( N \):<br>
      (a) For each step \( i = 1 \) to \( |\mathcal{B}|/q \):<br>
        i. Sample a minibatch \( S \) of next \( q \) bags \( \mathcal{B}_S \) from \( \pi \).<br>
        ii. Compute \( L_{totbag}, L_{totptnprior} \) and \( L_{totprprior} \) restricted only to the bags in \( S \), and compute \( L_{totPB} \).<br>
        iii. Using gradients from (1), (2) and (3) along with \( lr \) and opt, update the weights of \( \mathcal{M} \).
    </td>
  </tr>
</table>

Figure 2: Minibatch Model Training on Bags.

Using the above, Fig. 2 describes the minibatch based model training on bags.

**Preference based bag-loss with priors.** The approach is similar to that in the previous subsection, where instead of \( L_{totbag} \) we have a preference based loss for the pairs of bags \( S \) for which preference labels are available. Define \( \tilde{y}(B) := \sum_{\ell \in \{0, \ldots, C\}} \ell \cdot \mathcal{M}(B)[\ell] \) be the real-valued soft-label for a bag \( B \). For a pair of bags \( (B_1, B_2) \) with \( y_{B_2 > B_1} \) be the preference-label, we incorporate the Bradley-Terry model (Bradley and Terry, 1952) to measure the inconsistency of the predictions with the preference label. Specifically, we define the loss:

\[
L_{pref}(B_1, B_2, y_{B_2 > B_1})
:= \mathrm{CE}\left(y_{B_2 > B_1}, \frac{\tilde{y}(B_2)}{\tilde{y}(B_1) + \tilde{y}(B_2)}\right)
\] (6)

\( L_{pref}(B_1, B_2, y_{B_2 > B_1}) \) is averaged over all pairs in \( S \) to obtain \( L_{totpref} \) which we refer to as **PrefBagLoss**. The minibatch training now samples pairs of bags and computes \( L_{totpref} \) restricted to the sampled pairs. In the priors based augmentation, **PriorsPrefBagLoss**, \( L_{totptnprior} \) and \( L_{totprprior} \) losses remain the same, over all the instances in the minibatch. In this case, the total loss is

\[
L_{totPPB}
= \lambda L_{totpref} + \lambda_1 L_{totptnprior} + \lambda_2 L_{totprprior}.
\] (7)

**4.2 PsLab: Pseudo-labeling**

Our pseudo-labeling method, PsLab uses the predictions of the model \( \mathcal{M} \) trained as per the techniques described above, to output the max-likelihood instance-level labels for each bag, consistent with the bag-label, i.e., an instance-level pseudo-labeling independently for each bag, which well-defined since the bags are disjoint. We describe PsLab for the binary case of \( C = 1 \) and only for the MIN aggregation since MAX is equivalent
to MIN by flipping the labels.

Case {0,1}-labels and MIN aggregation. Given bag \( B = \{x_1, \ldots, x_k\} \) and \( y_B \), we output a pseudolabeling \( \Gamma_B : \{x_1, \ldots, x_k\} \rightarrow \{0, 1\} \) as follows: if \( y_B \) is 1 then \( \Gamma_B \) assigns 1 to all \( x \in B \).

If not, we first set \( \Gamma_B(x) = 1 \) for those \( x \in B \) on which the model prediction \( \mathcal{M}(x) \) is biased towards 1. If this results in all \( \Gamma_B \) assigning 1 to all \( x \in B \), then we let \( \Gamma_B(z) = 0 \) for some \( z \) on which the model predicts the smallest probability for label 1. We provide a detailed explanation of the PsLAB algorithm for \( C = 1 \) in Appendix I.

For \( C > 1 \) PsLAB is somewhat more involved and the algorithmic details along with a proof of correctness are included in Appendix J.

Model Training on Pseudo-labels. After computing the pseudo-labels on the training bags, we now have a train-set with labeled instances. The model retrained on this dataset is evaluated for comparative performance.

5 Tasks and Datasets

We consider following six datasets covering four tasks listed under Tables 2 and 3. More details on data processing in Appendix B.

Long-form Question Answering. We use QA-Feedback dataset, an SxS preference dataset collected and released by (Wu et al., 2023). These are human preferences on pairs of model generated responses for input questions and relevant passages from ASQA (Stelmakh et al., 2023). The data further contains segment-level annotations, and we make use of “irrelevance, repetition, or incoherence” category for evaluation. The responses are the bags and the preferences are the preference bag-labels. We use AVG as the aggregation function and point-prior is cosine similarity between knowledge passages and each sentence of the response. Cosine-similarity represents the semantic similarity or relevance between a sentence and the context and is an estimate of the sentence label. Specifically:

\[
p_x := \text{cospror}(x) \\
= 0.5 \left(1 + \langle x, U \rangle / (\|x\|_2 \|U\|_2)\right)
\] (8)

where \( x \) and \( U \) are (embeddings of) a sentence and the relevant passage. Additionally, we incorporate a pair-prior:

\[
p_{xz} := \text{corrprior}(x, z) = (1 + \rho_{xz}) / 2
\] (9)

where \( \rho_{xz} \) is the Pearson’s correlation between sentence embeddings \( x \) and \( z \) and represents the probability of two sentences have the same label.

Retrieval. We use two datasets for retrieval tasks: MultiSpanQA and FiRA. MultiSpanQA (Li et al., 2022) consists of question and retrieved context pairs, with annotated discontinuous answer spans for the train and validation splits (See Table 1 for example). Both the instance and bag labels are {0,1}-valued.

The FiRA dataset (Hofstätter et al., 2020) comprises word-level relevance annotations using {0, ..., 4}-valued labels. We derive the sentence-level scores by taking the word-level average across annotators and then the maximum across all words in a sentence. Similar to the previous setup, we treat the paragraph as a bag, its sentences as instances, and employ MAX as the aggregation function. The instance and bag-level belongs to the set {0, 1, 2, 3, 4}, with the goal of optimizing a cross entropy loss.

For both datasets, we integrate a correlation prior between sentence pairs and a cosine-similarity prior (see (8), (9)) between the query and each sentence of the context.

Summarization. We utilize two datasets: WikiCatSum (Perez-Beltrachini et al., 2019) and AquaMuSe (Kulkarni et al., 2020).

We adopt the binary entailment metric for this task. The reference summaries already provided in these two datasets serve as the entailed summaries\( ^1 \) with each sentence considered positively entailed. To generate non-entailed summaries, we synthesize negatives similar to (Yin et al., 2021). Examples of entailed and non-entailed summaries are provided in Appendix H, along with the method of generation.

As in previous tasks, we incorporate sentence-document cosine similarity and sentence correlation priors into our methods. Additionally, we experiment with NLI entailment scores (Honovich et al., 2022) as priors for this task.

Math Reasoning. We utilize PRM800K dataset (Lightman et al., 2023) releasing step-level annotations for model-generated solutions to MATH problems (Hendrycks et al., 2021b). The task at hand is to identify all the incorrect steps in the solution. Similar to previous tasks, we experiment with question-step cosine similarity prior and a cor-

\( ^1 \)In this work, we do not filter any noise present in the existing data splits.
<table>
  <tr>
    <th>Dataset (Input for training)</th>
    <th>Bags</th>
    <th>Instances</th>
    <th>Task</th>
    <th>Prior</th>
  </tr>
  <tr>
    <td>QA-Feedback (Question, Knowledge Passage, Pair of Responses, Preference Label)</td>
    <td>Both responses are treated as separate bags.</td>
    <td>Sentences in a response + question + Knowledge Passages</td>
    <td>Learn sentence-level scores for relevance using only the bag-label (indicates which response is better). The aggregation function used is AVG.</td>
    <td>knowledge passage-sentence cosine sim., corr. b/w sentences</td>
  </tr>
  <tr>
    <td>FirA (Paragraph, Query, Relevance Score)</td>
    <td>Paragraph</td>
    <td>sentence of the paragraph + query</td>
    <td>Learn {0, 1, 2, 3, 4}-valued relevance score for each sentence w/ query. The aggregation function used is MAX</td>
    <td>query-sentence cosine similarity</td>
  </tr>
  <tr>
    <td>MultiSpanQA (Context, Question, Label(answer present in context)</td>
    <td>Context</td>
    <td>Sentences of the context + Questions</td>
    <td>Identify sentences of the context which contain the answer to the question. We use MAX for this binary classification setup.</td>
    <td>query-sentence cosine sim., corr. b/w sentences</td>
  </tr>
  <tr>
    <td>AquaMuSe (Documents, Query, Summary, Entailment Label)</td>
    <td>Summary</td>
    <td>Sentence of the summary + documents + query</td>
    <td>Given a query, document and bag-level binary entailment label, determine the non-entailed sentences in a summary. MIN is the aggregate function used in this setup.</td>
    <td>doc-sentence cosine sim., corr. b/w sentences</td>
  </tr>
  <tr>
    <td>WikiCatSum (Documents, Summary, Entailment Label)</td>
    <td>Summary</td>
    <td>Sentence of a summary + documents</td>
    <td>Given a document and bag-level binary entailment label, determine the non-entailed sentences in a summary MIN is the aggregate function used in this setup.</td>
    <td>doc-sentence cosine sim., corr. b/w sentences</td>
  </tr>
  <tr>
    <td>PRM800K (MATH Problem, step-wise solution, Label(correctness))</td>
    <td>Solution to the MATH problem</td>
    <td>Step of the solution + question</td>
    <td>Using the binary aggregate label indicating the correctness of the solution, identify all incorrect steps in the solution.</td>
    <td>question-step cosine sim., corr. b/w steps</td>
  </tr>
</table>

Table 2: Summary of the bags, labels, instances, annotations and priors for each dataset.

<table>
  <tr>
    <th>Task</th>
    <th>Dataset</th>
    <th>Objective</th>
    <th>Train / Test Instances</th>
    <th>Train Bags</th>
  </tr>
  <tr>
    <td rowspan="2">Long-form QA</td>
    <td>QA-Feedback</td>
    <td>Preference</td>
    <td>95.8k / 9k</td>
    <td>14k</td>
  </tr>
  <tr>
    <td>FirA</td>
    <td>Classification</td>
    <td>102k / 20.4k</td>
    <td>18k</td>
  </tr>
  <tr>
    <td rowspan="2">Retrieval</td>
    <td>MultiSpanQA</td>
    <td>Classification</td>
    <td>47k / 5.7k</td>
    <td>5k</td>
  </tr>
  <tr>
    <td>AquaMuSe</td>
    <td>Classification</td>
    <td>13.5k / 2.5k</td>
    <td>3k</td>
  </tr>
  <tr>
    <td rowspan="2">Summarization</td>
    <td>WikiCatSum</td>
    <td>Classification</td>
    <td>20k / 7.7k</td>
    <td>4.8k</td>
  </tr>
  <tr>
    <td>PRM800K</td>
    <td>Classification</td>
    <td>663k / 18.5k</td>
    <td>98k</td>
  </tr>
  <tr>
    <td colspan="5">Math Reasoning</td>
  </tr>
</table>

Table 3: Summary of the setup used for each dataset

relation prior between steps of the solution.

A discussion on handling noisy priors and designing task-specific priors is provided in Appendix K.

6 Experiments

We evaluate FRACTAL along with baseline methods (listed below) on the tasks and datasets described in Section 5.

Priors as Baselines. The following methods based on the priors described in Section 5 are directly used as baselines to score the sentences as shown in Table 4:

Cosine Similarity: In this baseline, the semantic similarity of individual sentences of the response with the input context is used to estimate their relevance score for the task. For this, we compute the cosine similarity (see Eq. (8)) between the corresponding embeddings.

NLI - Entailment Scorer: For summarization and relevance tasks, we also compute entailment using the NLI scorer from TRUE paper (Honovich et al., 2022). This is a T5x-11B model (Raffel et al., 2023) finetuned on several NLI datasets.

Trainable Baselines. These baselines use the bag or instance labels to train models.

BagLoss: This uses BagLoss, \( L_{totbag} \), on bag-labels (or PrefBagLoss \( L_{totpref} \) in case of preference bag-labels) described in Section 4.

Response-level: For training, this uses entire response as a singleton bag i.e. \( |B| = 1 \) with \( x_1 = \) response. Inference is done on sentences.

Supervised: Trains directly on sentence-labels i.e. Train Instances of Table 3 to provide an upper baseline for comparison.

FRACTAL: As described in Section 4, this involves PriorsBagLoss (or PriorsPrefBagLoss) based model training using bag-labels (or preference bag-labels) as well priors. In the tables, PriorsBagLoss(\( \lambda_1, \lambda_2 \)) and PriorsPrefBagLoss(\( \lambda_1, \lambda_2 \)) denote the instantiation of these methods with weights \( \lambda_1 \) and \( \lambda_2 \) for the losses corresponding to the point and pair priors respectively (see (8), (9) and Section 4.1). For the WikiCatSum dataset, we incorporate NLI entailment scores as a prior, assigning a weight of \( \lambda_3 \) for the corresponding loss term. The weight for bag-loss (or preference bag-loss) is adjusted so that the sum of all the loss weights is 1. PSLAB denotes the performance of the model trained after pseudo-labeling the train-set using the best performing prior augmented bag loss. Note that when we only have preference bag-labels i.e., in the QA Preference Feedback dataset PSLAB is not applicable, and FRACTAL provides the model trained using PriorsPrefBagLoss.

Model Training Setup. We use the same model architecture across all tasks: a Sentence-T5 Large encoder to generate embeddings for text components, followed by a 2-hidden layer MLP with 73728 parameters for predicting sentence-level scores. To handle lengthy documents exceeding 2000 tokens in MultiSpanQA, WikiCatSum, and AquaMuSe datasets, we partition documents into 1000-token paragraphs which are encoded separately to improve embedding quality. Subsequently, attention weights representing importance are learnt for each document split, and the document embedding is ob-
<table>
  <tr>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td colspan="4"><i>MultiSpanQA (47k | 5.7k)</i></td>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.734 ± 0.013</td>
    <td>0.358 ± 0.009</td>
    <td>0.872 ± 0.042</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.455</td>
    <td>0.125</td>
    <td>0.851</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>0.631</td>
    <td>0.366</td>
    <td><b>0.859</b></td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.582 ± 0.113</td>
    <td>0.221 ± 0.089</td>
    <td>0.851 ± 0.007</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.662 ± 0.069</td>
    <td>0.307 ± 0.075</td>
    <td>0.849 ± 0.091</td>
  </tr>
  <tr>
    <td>FRACTAL*</td>
    <td><b>0.687 ± 0.062</b></td>
    <td><b>0.329 ± 0.053</b></td>
    <td>0.843 ± 0.049</td>
  </tr>
  <tr>
    <td>FRACTAL/Supervised%</td>
    <td>93.59%</td>
    <td>91.89%</td>
    <td>96.67%</td>
  </tr>
  <tr>
    <td colspan="4"><i>QA Preference Feedback (93.8k | 9k)</i></td>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.652 ± 0.018</td>
    <td>0.611 ± 0.007</td>
    <td>0.691 ± 0.015</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.335</td>
    <td>0.525</td>
    <td>0.483</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.491 ± 0.008</td>
    <td>0.4613 ± 0.007</td>
    <td>0.453 ± 0.015</td>
  </tr>
  <tr>
    <td>PriorBagLoss</td>
    <td>0.511 ± 0.004</td>
    <td>0.53 ± 0.002</td>
    <td><b>0.524 ± 0.007</b></td>
  </tr>
  <tr>
    <td>FRACTAL**</td>
    <td><b>0.537 ± 0.003</b></td>
    <td><b>0.531 ± 0.003</b></td>
    <td>0.519 ± 0.006</td>
  </tr>
  <tr>
    <td>FRACTAL/Supervised%</td>
    <td>82.36%</td>
    <td>86.91%</td>
    <td>75.11%</td>
  </tr>
  <tr>
    <td colspan="4"><i>WikiCutSum (209k | 10.7k)</i></td>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.831 ± 0.051</td>
    <td>0.889 ± 0.062</td>
    <td>0.714 ± 0.048</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.408</td>
    <td>0.829</td>
    <td>0.362</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>0.639</td>
    <td>0.817</td>
    <td>0.648</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.478 ± 0.048</td>
    <td>0.829 ± 0.051</td>
    <td>0.569 ± 0.062</td>
  </tr>
  <tr>
    <td>FRACTAL*</td>
    <td><b>0.643 ± 0.031</b></td>
    <td><b>0.875 ± 0.035</b></td>
    <td><b>0.665 ± 0.062</b></td>
  </tr>
  <tr>
    <td>FRACTAL/Supervised%</td>
    <td>77.37%</td>
    <td>98.42%</td>
    <td>93.14%</td>
  </tr>
  <tr>
    <td colspan="4"><i>AguaMeSe (13.5k | 2.5k)</i></td>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.878 ± 0.007</td>
    <td>0.925 ± 0.002</td>
    <td>0.867 ± 0.008</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.632</td>
    <td>0.763</td>
    <td>0.649</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>0.793</td>
    <td>0.889</td>
    <td>0.824</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.695 ± 0.016</td>
    <td>0.775 ± 0.005</td>
    <td>0.673 ± 0.010</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.747 ± 0.007</td>
    <td>0.825 ± 0.005</td>
    <td>0.730 ± 0.011</td>
  </tr>
  <tr>
    <td>FRACTAL*</td>
    <td><b>0.815 ± 0.005</b></td>
    <td><b>0.839 ± 0.006</b></td>
    <td><b>0.833 ± 0.011</b></td>
  </tr>
  <tr>
    <td>FRACTAL/Supervised%</td>
    <td>92.71%</td>
    <td>97.19%</td>
    <td>96.08%</td>
  </tr>
  <tr>
    <td colspan="4"><i>PRM800K (663k | 18.5k)</i></td>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.652 ± 0.015</td>
    <td>0.933 ± 0.013</td>
    <td>0.727 ± 0.022</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.51</td>
    <td>0.876</td>
    <td>0.516</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.537 ± 0.016</td>
    <td>0.879 ± 0.045</td>
    <td>0.535 ± 0.076</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.562 ± 0.024</td>
    <td>0.883 ± 0.006</td>
    <td><b>0.671 ± 0.033</b></td>
  </tr>
  <tr>
    <td>FRACTAL*</td>
    <td><b>0.593 ± 0.014</b></td>
    <td><b>0.901 ± 0.006</b></td>
    <td>0.618 ± 0.025</td>
  </tr>
  <tr>
    <td>FRACTAL/Supervised increase%</td>
    <td>90.05%</td>
    <td>96.35%</td>
    <td>85.01%</td>
  </tr>
</table>

Table 4: Evaluations on Test-set (instance-level). Prefix * indicates PsLAB method and ** indicates PriorsPrefBagLoss. Last row for each dataset has % of supervised achieved by FRACTAL. Note: PsLAB is not applicable QA Preference Feedback and FRACTAL is the model trained using PriorBagLoss.

<table>
  <tr>
    <th>Method</th>
    <th>MAE</th>
    <th>MSE</th>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.283 ± 0.072</td>
    <td>0.141 ± 0.088</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.319 ± 0.047</td>
    <td>0.186 ± 0.098</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.304 ± 0.007</td>
    <td>0.163 ± 0.002</td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.2, 0.2)</td>
    <td>0.294 ± 0.003</td>
    <td>0.155 ± 0.001</td>
  </tr>
  <tr>
    <td>FRACTAL</td>
    <td><b>0.293 ± 0.001</b></td>
    <td><b>0.152 ± 0.002</b></td>
  </tr>
  <tr>
    <td>FRACTAL/Supervised increase%</td>
    <td>3.5%</td>
    <td>7.8%</td>
  </tr>
</table>

Table 5: Test-test (instance-level) evaluation on FiRA.

Feedback, since PsLAB is not applicable, FRACTAL provides the model trained using PriorsPrefBagLoss. Table 6 provides the results for fine-grained RLHF on the QA-Feedback dataset using the framework provided by Wu et al. (2023) by replacing the human annotations (supervised) with relevance label predictions from our FRACTAL model trained only on preference labels. We evaluated the performance of the generated summaries by calculating the ROUGE score between them and the reference summaries across the entire test set. The Adv column provides the % of questions on which the generated response is rated comparatively more relevant by the Claude 3 Opus model (Cla). We repeat the FRACTAL-RLHF experiment on PRM800K (Table 6 column Acc) and evaluate using (Cla) the final answer accuracy on the test set for both Response and FRACTAL RLHF, noting a 5% absolute improvement. In Appendix M, we compare FRACTAL-RLHF with Implicit PRM (Yuan et al., 2024), a more recent and highly performant method for training PRMs using only response labels.

Ablations. In Table 7 we provide an ablation study on the effectiveness of priors and show how much addition of each prior contributes to the performance of BagLoss shown in Table 4. We also adapt our loss functions to ingest instance-level data by adding the instance-level loss term. In Appendix L, we demonstrate improvements by adding even

<table>
  <tr>
    <th>Method</th>
    <th colspan="2">QA-Feedback</th>
    <th>PRM800K</th>
  </tr>
  <tr>
    <th></th>
    <th>ROUGE</th>
    <th>Adv</th>
    <th>Acc</th>
  </tr>
  <tr>
    <td>SFT + Pref. or Response RLHF</td>
    <td>42.847</td>
    <td>62.731</td>
    <td>18.93</td>
  </tr>
  <tr>
    <td>SFT + FG RLHF (FRACTAL)</td>
    <td>43.221</td>
    <td>37.269</td>
    <td>24.17</td>
  </tr>
</table>

Table 6: FRACTAL RLHF on QA-Feedback and PRM800K.

<table>
  <tr>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td colspan="4"><i>MultiSpanQA</i></td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.2, 0)</td>
    <td><b>0.668 ± 0.054</b></td>
    <td><b>0.313 ± 0.04</b></td>
    <td>0.836 ± 0.052</td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.2)</td>
    <td>0.629 ± 0.055</td>
    <td>0.279 ± 0.028</td>
    <td><b>0.850 ± 0.033</b></td>
  </tr>
  <tr>
    <td colspan="4"><i>QA Preference Feedback</i></td>
  </tr>
  <tr>
    <td>PriorsPrefBagLoss(0, 2.0)</td>
    <td>0.517 ± 0.004</td>
    <td>0.495 ± 0.003</td>
    <td>0.512 ± 0.006</td>
  </tr>
  <tr>
    <td>PriorsPrefBagLoss(0, 0.4)</td>
    <td>0.528 ± 0.003</td>
    <td>0.521 ± 0.002</td>
    <td><b>0.533 ± 0.004</b></td>
  </tr>
  <tr>
    <td>PriorsPrefBagLoss(0.2, 0.5)</td>
    <td><b>0.537 ± 0.003</b></td>
    <td><b>0.531 ± 0.003</b></td>
    <td>0.519 ± 0.006</td>
  </tr>
  <tr>
    <td colspan="4"><i>WikiCutSum</i></td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.2, 0.0)</td>
    <td>0.636 ± 0.019</td>
    <td>0.877 ± 0.003</td>
    <td>0.639 ± 0.01</td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.3, 0.0)</td>
    <td>0.518 ± 0.005</td>
    <td>0.719 ± 0.009</td>
    <td>0.391 ± 0.006</td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.2, 0.1)</td>
    <td>0.639 ± 0.021</td>
    <td><b>0.885 ± 0.009</b></td>
    <td><b>0.653 ± 0.013</b></td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0, 0.4)</td>
    <td><b>0.643 ± 0.024</b></td>
    <td>0.881 ± 0.012</td>
    <td>0.652 ± 0.017</td>
  </tr>
  <tr>
    <td colspan="4"><i>PRM800K</i></td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.6, 0)</td>
    <td>0.573 ± 0.014</td>
    <td>0.889 ± 0.008</td>
    <td><b>0.624 ± 0.017</b></td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0, 0.1)</td>
    <td>0.577 ± 0.023</td>
    <td>0.925 ± 0.017</td>
    <td>0.603 ± 0.038</td>
  </tr>
  <tr>
    <td>PriorsBagLoss(0.5, 0.1)</td>
    <td><b>0.589 ± 0.017</b></td>
    <td><b>0.891 ± 0.006</b></td>
    <td>0.622 ± 0.015</td>
  </tr>
</table>

Table 7: Prior Ablation: Across all tasks, we see that using priors improves performance over all baselines given in Table 4 and that PointPair priors have additive benefits.
20% instance level labels.
The code for the experiments included in this paper is available at https://github.com/google-deepmind/fractal_acl25.

6.2 Discussion

FRACTAL is highly label-efficient while performing close to full supervision. As we can see from Table 3, the number of training bags is only a small fraction of (10 to 25 %) of the training instances. Thus, FRACTAL consumes only 10 to 25% of the labels required to train a fully supervised model. Nevertheless, the classification performance of the FRACTAL method (Table 4) shows that it achieves > 90% of the performance of the fully supervised model for most of the datasets/metrics except for QA Preference Feedback where the it recovers \( \approx 80\% \) of the supervised baseline.

FRACTAL renders more precise sentence-level scores. From Table 4, we observe a consistent improvement in the sentence scoring over the BagLoss as well as the Response-level baseline across all these datasets in terms of AUC-ROC, bridging the performance gap between them and the best approach supervised (instance-level trained) model. The use of priors along with pseudo-labeling based model training allows for an improved estimation of task specific score for sentences. It is interesting to note that BagLoss outperforms the Response-level baseline, suggesting that introduction of aggregate loss based methods to estimate sentence-level scores is itself useful. Response-level model trained with large response doesn’t generalize well to individual sentences.

Leveraging Prior improves Feedback Disaggregation. Among the key ideas of FRACTAL is to augment BagLoss with cosine-similarity and correlation priors at the sentence-level (see (8) and (9)). As observed from our ablations in Table 7 along with the results in Table 4, as well as Table 5 for FiRA, introducing the prior loss terms, provides substantially improved performance over either using just the BagLoss, or the cosine-similarity/NLI baselines. In effect, our proposed combination performs better than either of its constituents. We hypothesize that the priors provide sentence-level insight which complements the aggregate label based optimization of BagLoss.

Calibrated Pseudo-Labels are more helpful. From Tables 4, 5, 7 we can see that for all tasks applying PsLAB produces outperforms the models trained using PriorsBagLoss. This indicates the effectiveness of calibrating the disaggregated scores such that their aggregate matches the response scores for training the final model.

Downstream Tasks benefit from sentence-level scoring. Wu et al. (2023) showed that using fine grained human labels per segment level can help improve performance. In their setup, we replace the fine grained human labels of relevance with the FRACTAL predictions (Refer Long-Form Question Answering task in Section 5). We observe in Table 6 that FRACTAL (Row 2) helps improve performance over using only preference labels (Row 1) by around 0.873% in ROUGE score. Running the same fine-tuning experiment on PRM800k resulted in a 5% absolute increase in the accuracy of the generated solutions over Response RLHF.

7 Conclusions

We propose a novel method FRACTAL, which augments bag-loss using instance level priors to train predictor models, along with a pseudo-labeling technique for improved model training. Extensive evaluations of FRACTAL along with vanilla bag-loss and response-level model training baselines, as well as off-the-shelf scorers demonstrate substantial performance gains from FRACTAL models on six datasets spanning four tasks: retrieval, question answering, summarization, and math reasoning.

8 Limitations

Our method for label calibration and pseudo-labeling works well in classification tasks, leading to better performance. However, applying this technique becomes difficult when dealing with preference feedback. Also, if the bag size becomes very large, there is a risk of the bag-loss functions becoming intractable. FRACTAL treats each step or sentence independently. This results in a loss of valuable sequential context, which can impact the quality of the reward model. As a result, when performing fine-tuning (FRACTAL-RLHF) using predicted pseudo-labels, our method may underperform compared to recent approaches like Implicit PRM (see Appendix M).

9 Ethical Considerations

We propose techniques to transform via model training methods, a response-level score into sentence-level scores. While such response-level
scores are commonly obtained by human annotators, we have not conducted any human annotations. Our evaluations are on publicly available datasets, and our method produces artificial labels for downstream tasks and does not modify in anyway the original response scores or attempt to associate them to any individual(s).

References

Claude 3 opus model. claude-3-opus-20240229, https://www.anthropic.com/.

Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. CoRR, abs/2001.09977.

Reinald Kim Amplayo, Peter J. Liu, Yao Zhao, and Shashi Narayan. 2022. Smart: Sentences as basic units for text evaluation. Preprint, arXiv:2208.01030.

Stefanos Angelidis and Mirella Lapata. 2018. Multiple instance learning networks for fine-grained sentiment analysis. Transactions of the Association for Computational Linguistics, 6:17–31.

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, and 109 others. 2023. Palm 2 technical report. Preprint, arXiv:2305.10403.

Ehsan Mohammady Ardehaly and Aron Culotta. 2016. Domain adaptation for learning from label proportions using self-training. In IJCAI, pages 3670–3676.

Ehsan Mohammady Ardehaly and Aron Culotta. 2017. Co-training for demographic classification using deep learning from label proportions. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 1017–1024. IEEE.

Boris Babenko. 2008. Multiple instance learning: algorithms and applications. View Article PubMed/NCBI Google Scholar, 19.

Boris Babenko, Ming-Hsuan Yang, and Serge Belongie. 2009. Visual tracking with online multiple instance learning. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 983–990.

Denis Barucic and Jan Kybic. 2022. Fast learning from label proportions with small bags. In Proc. IEEE ICIP, pages 3156–3160.

Avrim Blum and Adam Kalai. 1998. A note on learning from multiple-instance examples. Machine learning, 30:23–29.

G. Bortsova, F. Dubost, S. N. Ørting, I. Katramados, L. Hogeweg, L. H. Thomsen, M. M. W. Wille, and M. de Bruijne. 2018. Deep learning from label proportions for emphysema quantification. In MICCAI, volume 11071 of Lecture Notes in Computer Science, pages 768–776. Springer.

Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.

Robert Istvan Busa-Fekete, Heejin Choi, Travis Dick, Claudio Gentile, and Andres Munoz medina. 2023. Easy learning from label proportions. arXiv.

Jatin Chauhan, Xiaoxuan Wang, and Wei Wang. 2023. Learning under label proportions for text classification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12210–12223, Singapore. Association for Computational Linguistics.

L. Chen, Z. Huang, and R. Ramakrishnan. 2004. Cost-based labeling of groups of mass spectra. In Proc. ACM SIGMOD International Conference on Management of Data, pages 167–178.

Lin Chen, Thomas Fu, Amin Karbasi, and Vahab Mirrokni. 2023. Learning from aggregated data: Curated bags versus random bags. arXiv.

S. Chen, B. Liu, M. Qian, and C. Zhang. 2009. Kernel k-means based framework for aggregate outputs classification. In Proc. ICDM, pages 356–361.

N. de Freitas and H. Kück. 2005. Learning about individuals from group statistics. In Proc. UAI, pages 332–339.

L. M. Dery, B. Nachman, F. Rubbo, and A. Schwartzman. 2017. Weakly supervised classification in high energy physics. Journal of High Energy Physics, 2017(5):1–11.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.

Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez. 1997. Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell., 89(1-2):31–71.

G. Dulac-Arnold, N. Zeghidour, M. Cuturi, L. Beyer, and J. P. Vert. 2019. Deep multi-class learning from label proportions. CoRR, abs/1905.12909.

Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Proceedings of the 2022 ACM
Designing Interactive Systems Conference. DIS ’22, page 1002–1019, New York, NY, USA. Association for Computing Machinery.

Google, Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Paschos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, and 110 others. 2023. Palm 2 technical report. Preprint, arXiv:2305.10403.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021a. Measuring mathematical problem solving with the MATH dataset. In Proc. NeurIPS Datasets and Benchmarks.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.

J. Hernández-González, I. Inza, and J. A. Lozano. 2013. Learning bayesian network classifiers from label proportions. Pattern Recognit., 46(12):3425–3440.

J. Hernández-González, I. Inza, L. Crisol-Ortiz, M. A. Guembe, M. J. Inarra, and J. A. Lozano. 2018. Fitting the data from embryo implantation prediction: Learning from label proportions. Statistical methods in medical research, 27(4):1056–1066.

Sebastian Hofstätter, Markus Zlabinger, Mete Sertkan, Michael Schröder, and Allan Hanbury. 2020. Fine-grained relevance annotations for multi-task document ranking and question answering. In Proc. of CIKM.

Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. True: Re-evaluating factual consistency evaluation. In Workshop on Document-grounded Dialogue and Conversational Question Answering.

Maximilian Ilse, Jakub Tomeczak, and Max Welling. 2018. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR.

Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.

D. Kotzias, M. Denil, N. de Freitas, and P. Smyth. 2015. From group to individual labels using deep features. In Proc. SIGKDD, pages 597–606.

Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization. Preprint, arXiv:2010.12694.

Haonan Li, Martin Tomko, Maria Vasardani, and Timothy Baldwin. 2022. MultiSpanQA: A dataset for multi-span question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1250–1260, Seattle, United States. Association for Computational Linguistics.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. Preprint, arXiv:2305.20050.

J. Liu, B. Wang, Z. Qi, Y. Tian, and Y. Shi. 2019. Learning from label proportions with generative adversarial networks. In Proc. NeurIPS, pages 7167–7177.

Jiabin Liu, Bo Wang, Xin Shen, Zhiqian Qi, and Yingjie Tian. 2021. Two-stage training for learning from label proportions. In Proceedings of the Thirty-fifth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2737–2743. International Joint Conferences on Artificial Intelligence Organization. Main Track.

Jiexi Liu, Dehan Kong, Longtao Huang, Dinghui Mao, and Hui Xue. 2022. Multiple instance learning for offensive language detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7387–7396. Association for Computational Linguistics.

T. Lozano-Pérez and C. Yang. 2000. Image database retrieval with multiple-instance learning techniques. In Proc. ICDE, page 233.

Zhekun Luo, Devin Guillory, Baifeng Shi, Wei Ke, Fang Wan, Trevor Darrell, and Huijuan Xu. 2020. Weakly-supervised action localization with expectation-maximization multi-instance learning. In Computer Vision – ECCV 2020, pages 729–745, Cham. Springer International Publishing.

O. Maron. 1998. Learning from ambiguity. Ph.D. thesis, Massachusetts Institute of Technology.

Oded Maron and Tomás Lozano-Pérez. 1997. A framework for multiple-instance learning. NIPS’97, page 570–576.

Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. 2022. Teaching language models to support answers with verified quotes. Preprint, arXiv:2203.11147.
D. R. Musicant, J. M. Christensen, and J. F. Olson. 2007. Supervised learning by training on aggregate outputs. In Proc. ICDM, pages 252–261. IEEE Computer Society.

J. Nandy, R. Saket, P. Jain, J. Chauhan, B. Ravindran, and A. Raghuram. 2022. Domain-agnostic contrastive representations for learning from label proportions. In Proc. CIKM, pages 1542–1551.

Conor O’Brien, Arvind Thiagarajan, Sourav Das, Rafael Barreto, Chetan Verma, Tim Hsu, James Neufeld, and Jonathan J. Hunt. 2022. Challenges and approaches to privacy preserving post-click conversion prediction. CoRR, abs/2201.12666.

S. N. Ørting, J. Petersen, M. Wille, L. Thomsen, and M. de Bruijne. 2016. Quantifying emphysema extent from weakly labeled ct scans of the lungs using label proportions learning. In The Sixth International Workshop on Pulmonary Image Analysis, pages 31–42.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Preprint, arXiv:2203.02155.

Nikolaos Pappas and Andrei Popescu-Belis. 2014. Explaining the stars: Weighted multiple-instance learning for aspect-based sentiment analysis. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 455–466, Doha, Qatar. Association for Computational Linguistics.

Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit document modeling through weighted multiple-instance learning. Journal of Artificial Intelligence Research, 58:591–626.

G. Patrini, R. Nock, T. S. Caetano, and P. Rivera. 2014. (almost) no label no cry. In Proc. Advances in Neural Information Processing Systems, pages 190–198.

Laura Perez-Beltrachini, Yang Liu, and Mirella Lapata. 2019. Generating summaries with topic templates and structured convolutional decoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5107–5116, Florence, Italy. Association for Computational Linguistics.

N. Quadrianto, A. J. Smola, T. S. Caetano, and Q. V. Le. 2009. Estimating labels from label proportions. J. Mach. Learn. Res., 10:2349–2374.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. Preprint, arXiv:1910.10683.

Jan Ramon and Luc De Raedt. 2000. Multi instance neural networks. In Proceedings of the ICML-2000 workshop on attribute-value and relational learning, pages 53–60.

Soumya Ray and Mark Craven. 2005. Supervised versus multiple instance learning: an empirical comparison. In Proc. ICML, page 697–704.

S. Rueping. 2010. SVM classifier estimation from group probabilities. In Proc. ICML, pages 911–918.

Sivan Sabato and Naftali Tishby. 2012. Multi-instance learning with any hypothesis class. Journal of Machine Learning Research, 13(97):2999–3039.

Rishi Saket, Aravindan Raghuram, and Balaraman Ravindran. 2022. On combining bags to better learn from label proportions. In AISTATS, volume 151 of Proceedings of Machine Learning Research, pages 5913–5927. PMLR.

C. Scott and J. Zhang. 2020. Learning from label proportions: A mutual contamination framework. In Proc. NeurIPS.

Karan Sikka, Abhinav Dhall, and Marian Bartlett. 2013. Weakly supervised pain localization using multiple instance learning. In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pages 1–8.

Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2023. Asqa: Factoid questions meet long-form answers. Preprint, arXiv:2204.06092.

M. Stolpe and K. Morik. 2011. Learning from label proportions by optimizing cluster model selection. In ECML PKDD Proceedings, Part III, volume 6913, pages 349–364. Springer.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyan Fu, and 49 others. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.

Xi Wang, Fangyao Tang, Hao Chen, Carol Y. Cheung, and Pheng-Ann Heng. 2023. Deep semi-supervised multiple instance learning with self-correction for dme classification from oct images. Medical Image Analysis, 83:102673.

J. Wu, Yinan Yu, Chang Huang, and Kai Yu. 2015a. Deep multiple instance learning for image classification and auto-annotation. In Proc. CVPR, pages 3460–3469.

Jiajun Wu, Yinan Yu, Chang Huang, and Kai Yu. 2015b. Deep multiple instance learning for image classification and auto-annotation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3460–3469.
Zequi Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. Preprint, arXiv:2306.01693.

Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. Docnli: A large-scale dataset for document-level natural language inference. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics.

F. X. Yu, D. Liu, S. Kumar, T. Jebara, and S. F. Chang. 2013. \( \alpha \)SVM for learning with label proportions. In Proc. ICML, volume 28, pages 504–512.

Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. 2024. Free process rewards without process labels. Preprint, arXiv:2412.01981.

Cha Zhang, John Platt, and Paul Viola. 2005. Multiple instance boosting for object detection. In Advances in Neural Information Processing Systems, volume 18. MIT Press.

J. Zhang, Y. Wang, and C. Scott. 2022. Learning from label proportions by learning with label noise. In Proc. NeurIPS.

Qi Zhang and Sally Goldman. 2001. Em-dd: An improved multiple-instance learning technique. Advances in neural information processing systems, 14.

A Choice of probAGG

The extension for MIN is:
\[
\text{probAGG}(\mathcal{M}(x_1), \ldots, \mathcal{M}(x_k)) = \mathcal{M}(x_{i^*})
\]
where \( i^* := \argmin (\hat{y}(x_1), \ldots, \hat{y}(x_k)) \) is the index of the instance with the minimum soft-label as defined in Section 4.1. We use the in-built TensorFlow (TF) tf_math_argmin function to compute argmin. For MAX we use the tf_math_argmax function. We investigate other popular differentiable approximations of the MIN function and present experimental results for the binary classification setting in Appendix D. For AVG, we simply take the average of the model predictions i.e., \( \text{probAGG}(\mathcal{M}(x_1), \ldots, \mathcal{M}(x_k)) = (1/k) \sum_{i=1}^k \mathcal{M}(x_i) \).

B Details on Tasks and Datasets

Additional details for Retrieval and Summarization tasks described in Section 5

Retrieval. We use two datasets for retrieval tasks; MultiSpanQA and FiRA. MultiSpanQA (Li et al., 2022) consists of question and retrieved context pairs, with annotated discontinuous answer spans for the train and validation splits (See Table 1 for example). We randomly select 25% of the train-split as the test-split. Context is treated as a bag, with sentences as instances labeled 1 if they overlap with annotated spans, and 0 otherwise. The MAX aggregation is used to indicate the presence an answer in the context. Given all MultiSpanQA samples contain answers, we create negative bags for half of the samples by extracting context chunks without answers. Thus, both the instance and bag labels are {0, 1}-valued.

The FiRA dataset (Hofstätter et al., 2020) comprises word-level relevance annotations using {0, ..., 4}-valued labels. We derive the sentence-level scores by taking the word-level average across annotators and then the maximum across all words in a sentence. Similar to the previous setup, we treat the paragraph as a bag, its sentences as instances, and employ MAX as the aggregation function. The instance and bag-level belongs to the set {0, 1, 2, 3, 4}, with the goal of optimizing a cross entropy loss.

For both datasets, we integrate a correlation prior between sentence pairs and a cosine-similarity prior (see (8), (9)) between the query and each sentence of the context.

Summarization. We utilize two datasets: WikiCatSum (Perez-Beltrachini et al., 2019) and AquaMuSe (Kulkarni et al., 2020).

We adopt the binary entailment metric for this task. The reference summaries already provided in these two datasets serve as the entailed summaries2 with each sentence considered positively entailed. To generate non-entailed summaries, we synthesize negatives similar to (Yin et al., 2021). Firstly, we perturb the reference summary through sentence replacement. This involves randomly selecting k sentences, where k is less than the total sentences in the summary, and iteratively feeding their left context to an PALM-2 (Anil et al., 2023) to predict the next sentence. The predicted sentence is then used to replace the selected one. Additionally, we explore the standard word replacement technique, which randomly masks k words whose POS tags are among proper nouns, numbers, and verbs, to introduce factual errors in the summaries. The masked words are then predicted using BERT (Devlin et al., 2018). The perturbed sentences within the summary are considered non-entailed. Thus,

2In this work, we do not filter any noise present in the existing data splits.
the sentence as well as bag labels are {0, 1}-valued with 1 indicating entailed and 0 non-entailed, with MIN as the aggregation function. Examples of entailed and non-entailed summaries are provided in Appendix H. As in previous tasks, we incorporate sentence-document cosine similarity and sentence correlation priors into our methods. Additionally, we experiment with NLI entailment scores (Honovich et al., 2022) as priors for this task.

C Task Specific Discussion of Experimental Results

Following is the detailed per-task analysis:

Long-form Question Answering. As presented in Table 4, the model trained using our PriorsBagLoss method on preference labels with the cosine-similarity prior has the best AUC-ROC, AUC-PR and accuracy scores in the experiments on the QA-feedback dataset. It outperforms both BagLoss as well as the cosine-similarity based baselines, the latter one by a significant margin. However, we observe that the performance of the correlation prior based variant is worse than that of the BagLoss baseline which itself is worse than the Response-level trained baseline.

Retrieval. In the MultiSpan-QA dataset experiments (Table 4) we observe that the model trained by applying PsLab on the predictions of the model trained on PriorsBagLoss with a combination of the cosine-similarity and correlation priors achieves the best AUC-ROC and AUC-PR scores (by a significant gap) among the bag level baselines, while the variant with only the cosine-similarity prior performs the second bet on these metrics. However, the Response-level trained model and BagLoss achieve marginally higher accuracy scores. All these methods also handily outperform the cosine-similarity baseline. From the FiRA dataset results (Table 5) we observe that the our prior augmented BagLoss method, specifically the using both the priors, performs the best on mae as well as mse metrics.

Summarization. The experimental results on the WikiCatSum dataset, presented in Table 4, show that our PriorsBagLoss method with different combinations of the cosine-similarity and the correlation priors, or using NLI as a prior, as well as the PsLab method on these models yield the best performance (by a significant margin) in terms of AUC-ROC, AUC-PR and accuracy metrics. The comparative baselines are the BagLoss, NLI and cosine-similarity. A similar trend is observed on the AquaMuse dataset (Table 4) on which the our methods significantly outperform the bag-level baselines, in particular the PsLab applied to the PriorsBagLoss method yields the best performing model.

Math Reasoning. On the PRM800k dataset, we observe from the experimental evaluations (Table 4) that the BagLoss and PriorsBagLoss methods are best performing among the bag-level baselines and also outperform the cosine-similarity baselines. While PriorsBagLoss using a combination of cosine-similarity and correlation priors achieves better AUC-ROC scores, BagLoss has significantly better accuracy while the AUC-PR scores are similar.

<table>
  <tr>
    <th>Model</th>
    <th>AND Approx</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
  </tr>
  <tr>
    <td>Instance Baseline</td>
    <td>-</td>
    <td>0.837</td>
    <td>0.894</td>
  </tr>
  <tr>
    <td rowspan="3">BagLoss</td>
    <td>Mult</td>
    <td>0.449</td>
    <td>0.785</td>
  </tr>
  <tr>
    <td>GM</td>
    <td>0.463</td>
    <td>0.824</td>
  </tr>
  <tr>
    <td>tf.reduce_min</td>
    <td>0.478</td>
    <td>0.829</td>
  </tr>
  <tr>
    <td rowspan="3">PriorBagLoss(0.2, 0.1)</td>
    <td>Mult</td>
    <td>0.599</td>
    <td>0.858</td>
  </tr>
  <tr>
    <td>GM</td>
    <td>0.631</td>
    <td>0.862</td>
  </tr>
  <tr>
    <td>tf.reduce_min</td>
    <td>0.643</td>
    <td>0.877</td>
  </tr>
</table>

Table 8: Results for differentiable AND approximations on WikiCatSum dataset

D Results for Differentiable Minimum Approximations

In the binary-label case, the standard baseline is Mult which is just the product of the soft-labels. More sophisticated approximations that we include in our study are LSE (Ramon and De Raedt, 2000), ISR, NOR and GM (Zhang et al., 2005) (see Section 2.4.1 of Babenko (2008) for details). For the binary case, we can employ the in built TensorFlow (TF) approximation tf_reduce_min over the soft-label (which is used for the loss functions) in our experiments, noting that MAX can be derived from MIN applied to flipped variables in the binary case.

Table 8 has an ablation of Mult, GM and tf_reduce_min for the BagLoss and PriorBagLoss methods on the WikiCatSum dataset, demonstrating that tf_reduce_min outperforms the others in AUC-ROC and AUC-PR metrics.

E Aggregate and instance-level evaluations

We also include evaluations of the various methods on a test set of bags w.r.t. bag-level metrics using the corresponding AGG approximations. Tables 9,
10, 11, 12, 13 and 10 contain the aggregate as well as instance evaluations.

Table 15 compares the performance achieved on the WikiCatSum dataset by varying the percentage of 1-bags in the training set.

F Hyperparameter Tuning

The key hyperparameters in our approach include the weights of the bag loss and prior loss terms, as well as the learning rate. We conducted a grid search over various values for these parameters to identify the optimal combination for each dataset. The learning rates considered were \( \{1e-2, 1e-3, 1e-4, 1e-5, 1e-6\} \), and the weights for each term were selected from \( \{0, 0.1, 0.2, \cdots 1\} \). As noted, we only consider convex combinations of the different loss terms, so combinations where the sum of the coefficients exceeds 1 were excluded. Table 16 contains the best weights for our prior augmented BagLoss method on different datasets, along with the best learning rates and batch size in the bag-level training.

G Details on Test Splits

We use the original test split for all datasets except WikiCatSum, whenever applicable.
PRM800K: We use both Phase 1 and 2 of the PRM800K dataset and maintain the same test splits as the original dataset. We split the train set into 75% for training and 10% for validation.
MultiSpanQA: We randomly select 25% of the train split to form the test split. This is because the original dataset’s test split lacks annotated answer spans.
FiRA: We partitioned the samples into 75% for training, 10% for validation, and 15% for testing.
WikiCatSum: We subsample 750 samples from test splits of the Animal and Film domains.
AquaMuSe: The original test split of the abstractive version of AquaMuSe has been utilized for testing purposes.
QA Feedback: Preference annotations were available for both the training and development sets. We reserved the dev set for validation purposes and divided the original training dataset into an 80:20 ratio for training and testing. We only consider wins and losses and have removed any ties before splitting the dataset.

H Generation of Perturbed Summaries from the WikiCatSum and Aquamuse Datasets

We utilize two datasets: WikiCatSum and Aqua-MuSe for the entailment (or summarization) task. The WikiCatSum dataset (Perez-Beltrachini et al., 2019) is specifically designed for multi-document summarization tasks, focusing on generating Wikipedia-style lead sections for entities within three domains: Companies, Films, and Animals out of which we focus on the Films and Animals domains. On the other hand, the AquaMuSe dataset (Kulkarni et al., 2020) is tailored for multi-document, question-focused summarization.

We adopt the binary entailment metric for this task. The reference summaries already provided in these two datasets serve as the entailed summaries\footnote{In this work, we do not filter any noise present in the existing data splits.}. Each sentence in these summaries is considered positively entailed. To generate non-entailed summaries, we synthesize negatives by employing various manipulations, similar to (Yin et al., 2021). Firstly, we perturb the reference summary through sentence replacement. This involves randomly selecting \( k \) sentences, where \( k \) is less than the total sentences in the summary, and iteratively feeding their left context to an PaLM 2 Bison Chat (Google et al., 2023) to predict the next sentence. The predicted sentence is then used to replace the selected one. Additionally, we explore the standard word replacement technique, which randomly masks \( k \) words whose POS tags are among proper nouns, numbers, and verbs, to introduce factual errors in the summaries. The masked words are then predicted using BERT. The number of replaced sentences and words is randomly selected for each sample. The perturbed sentences within the summary are considered non-entailed, while the remaining unchanged sentences are deemed entailed. Thus, the sentence as well as bag labels are \( \{0, 1\} \)-valued with 1 indicating entailed and 0 non-entailed, with MIN as the aggregation function.

Tables 17 and 18 contain the entailed and non-entailed (perturbed) summaries for the Aquamuse and WikiCatSum datasets respectively.

I PsLab for Binary Classification

The main idea behind PsLab is as follows: given a bag-level label for a collection of sentences
<table>
  <tr>
    <th>Evaluation</th>
    <th>Model</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
  </tr>
  <tr>
    <td rowspan="6">Aggregate</td>
    <td>Cosine Similarity</td>
    <td>0.488</td>
    <td>0.287</td>
    <td>0.698</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>0.6855</td>
    <td>0.522</td>
    <td>0.7453</td>
    <td>0.7883</td>
    <td>0.319</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.681 ± 0.012</td>
    <td>0.525 ± 0.081</td>
    <td>0.726 ± 0.033</td>
    <td>0.752 ± 0.011</td>
    <td>0.428 ± 0.063</td>
  </tr>
  <tr>
    <td>Sentence-level Model</td>
    <td>0.653 ± 0.076</td>
    <td>0.462 ± 0.050</td>
    <td>0.723 ± 0.015</td>
    <td>0.599 ± 0.093</td>
    <td>0.435 ± 0.187</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.678 ± 0.082</td>
    <td>0.525 ± 0.072</td>
    <td>0.718 ± 0.057</td>
    <td>0.717 ± 0.014</td>
    <td>0.391 ± 0.033</td>
  </tr>
  <tr>
    <td>0.8 BagLoss + 0.2 P1</td>
    <td>0.683 ± 0.053</td>
    <td>0.527 ± 0.02</td>
    <td>0.722 ± 0.035</td>
    <td>0.748 ± 0.009</td>
    <td>0.461 ± 0.01</td>
  </tr>
  <tr>
    <td>0.7 BagLoss + 0.2 P2 + 0.1 P1</td>
    <td>0.665 ± 0.061</td>
    <td>0.491 ± 0.04</td>
    <td>0.727 ± 0.029</td>
    <td>0.693 ± 0.018</td>
    <td>0.316 ± 0.037</td>
  </tr>
  <tr>
    <td rowspan="7">Instance</td>
    <td>Cosine Similarity</td>
    <td>0.455</td>
    <td>0.135</td>
    <td>0.851</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>0.631</td>
    <td>0.366</td>
    <td>0.859</td>
    <td>0.872</td>
    <td>0.178</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.583 ± 0.187</td>
    <td>0.217 ± 0.094</td>
    <td>0.852 ± 0.003</td>
    <td>0.529 ± 0.074</td>
    <td>0.086 ± 0.04</td>
  </tr>
  <tr>
    <td>Sentence-level Model</td>
    <td>0.729 ± 0.016</td>
    <td>0.354 ± 0.008</td>
    <td>0.861 ± 0.046</td>
    <td>0.717 ± 0.011</td>
    <td>0.438 ± 0.072</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.661 ± 0.092</td>
    <td>0.309 ± 0.127</td>
    <td>0.852 ± 0.133</td>
    <td>0.711 ± 0.074</td>
    <td>0.24 ± 0.188</td>
  </tr>
  <tr>
    <td>0.8 BagLoss + 0.2 P1</td>
    <td>0.669 ± 0.063</td>
    <td>0.311 ± 0.059</td>
    <td>0.838 ± 0.071</td>
    <td>0.65 ± 0.089</td>
    <td>0.189 ± 0.112</td>
  </tr>
  <tr>
    <td>0.7 BagLoss + 0.2 P2 + 0.1 P1</td>
    <td>0.625 ± 0.07</td>
    <td>0.271 ± 0.039</td>
    <td>0.851 ± 0.021</td>
    <td>0.639 ± 0.189</td>
    <td>0.135 ± 0.098</td>
  </tr>
  <tr>
    <td>FGLAB</td>
    <td>0.693 ± 0.115</td>
    <td>0.326 ± 0.071</td>
    <td>0.842 ± 0.052</td>
    <td>0.676 ± 0.043</td>
    <td>0.228 ± 0.091</td>
  </tr>
</table>

Table 9: Comparison of aggregate and instance-level performance on MultiSpanQA Dataset

<table>
  <tr>
    <th>Evaluation</th>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
  </tr>
  <tr>
    <td rowspan="5">Preference</td>
    <td>Cosine Similarity</td>
    <td>0.4978</td>
    <td>0.3952</td>
    <td>0.477</td>
    <td>0.379</td>
    <td>0.4985</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.546</td>
    <td>0.4651</td>
    <td>0.553</td>
    <td>0.437</td>
    <td>0.539</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.543</td>
    <td>0.4644</td>
    <td>0.5463</td>
    <td>0.442</td>
    <td>0.5324</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2,0)</td>
    <td>0.568</td>
    <td>0.4658</td>
    <td>0.574</td>
    <td>0.439</td>
    <td>0.5467</td>
  </tr>
  <tr>
    <td rowspan="5">Instance</td>
    <td>Sentence-level Model</td>
    <td>0.647</td>
    <td>0.611</td>
    <td>0.686</td>
    <td>0.722</td>
    <td>0.418</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.535</td>
    <td>0.526</td>
    <td>0.483</td>
    <td>0.893</td>
    <td>0.134</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.491</td>
    <td>0.4643</td>
    <td>0.453</td>
    <td>0.882</td>
    <td>0.278</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.509</td>
    <td>0.5269</td>
    <td>0.5167</td>
    <td>0.814</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0)</td>
    <td>0.516</td>
    <td>0.4936</td>
    <td>0.508</td>
    <td>0.647</td>
    <td>0.715</td>
  </tr>
</table>

Table 10: Comparison of Preference and instance-level evaluation on QA Preference Feedback Dataset

and the model’s probabilistic predictions for individual sentence-level labels, the goal is to find the maximum-likelihood configuration of sentence-level labels that is consistent with the given bag label. The likelihood is defined with respect to the model’s predicted probabilities.

Below, we explain the algorithm using an example focused on the MIN aggregation function applied over binary {0,1}-sentence labels. Consider a task where the objective is to identify harmful sentences, labeled as 0, within a document. The document aggregate label is given which indicates whether it is harmful or not, i.e., whether it contains at least one harmful sentence.

• Case 1: Bag-label is 1
In this case, the only valid configuration is the one where all sentences are labeled as 1 (i.e., none are harmful). This is because the MIN function over the sentence-level labels must return 1, which can only happen if every sentence is labeled 1. The presence of even a single harmful (label 0) sentence would reduce the MIN to 0, contradicting the true bag label.

• Case 2: Bag-label is 0
Here, the MIN aggregation requires that at least one sentence must be labeled as 0. The unconstrained maximum-likelihood configuration would assign label 1 to all sentences for which the model’s predicted probabilities are biased towards 1. If this configuration already includes at least one sentence labeled 0, then it is valid and can be used as is.

However, if all sentence labels are 1 in this unconstrained configuration (thus violating the bag-label constraint), we identify the sentence whose predicted probability is least biased towards 1 (i.e., closest to 0), and flip its label to 0.

We show in Appendix J that this results in the maximum-likelihood valid configuration of sentence-labels.

J PsLab for Multiclass
Suppose that the label set is \( \{0, \ldots, C\} \) where \( C \in \mathbb{Z}^+ \) and \( C > 1 \). With the setup as in Section
<table>
  <tr>
    <th>Evaluation</th>
    <th>Model</th>
    <th>MAE</th>
    <th>MSE</th>
  </tr>
  <tr>
    <td rowspan="3">Aggregate</td>
    <td>Instance-level Model</td>
    <td>0.375 ± 0.09</td>
    <td>0.247 ± 0.113</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.320 ± 0.042</td>
    <td>0.197 ± 0.017</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.326 ± 0.021</td>
    <td>0.209 ± 0.007</td>
  </tr>
  <tr>
    <td rowspan="5">Instance</td>
    <td>Instance-level Model</td>
    <td>0.283 ± 0.072</td>
    <td>0.141 ± 0.088</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.319 ± 0.047</td>
    <td>0.186 ± 0.098</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.304 ± 0.007</td>
    <td>0.163 ± 0.002</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.3, 0)</td>
    <td>0.298 ± 0.002</td>
    <td>0.157 ± 0.004</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0.2)</td>
    <td>0.294 ± 0.003</td>
    <td>0.155 ± 0.001</td>
  </tr>
</table>

Table 11: Comparison of Aggregate and Instance-level evaluations on FirA Dataset

<table>
  <tr>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
  </tr>
  <tr>
    <td>Instance Baseline</td>
    <td>0.837 ± 0.062</td>
    <td>0.894 ± 0.085</td>
    <td>0.718 ± 0.085</td>
    <td>0.926 ± 0.001</td>
    <td>0.733 ± 0.003</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>0.639</td>
    <td>0.817</td>
    <td>0.648</td>
    <td>0.834</td>
    <td>0.559</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.408</td>
    <td>0.829</td>
    <td>0.362</td>
    <td>0.719</td>
    <td>0.276</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.477 ± 0.093</td>
    <td>0.831 ± 0.052</td>
    <td>0.562 ± 0.047</td>
    <td>0.769 ± 0.018</td>
    <td>0.319 ± 0.048</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0.1)</td>
    <td>0.641 ± 0.028</td>
    <td>0.879 ± 0.013</td>
    <td>0.651 ± 0.017</td>
    <td>0.897 ± 0.009</td>
    <td>0.658 ± 0.082</td>
  </tr>
  <tr>
    <td>PSLab</td>
    <td>0.645 ± 0.038</td>
    <td>0.879 ± 0.057</td>
    <td>0.663 ± 0.091</td>
    <td>0.884 ± 0.076</td>
    <td>0.661 ± 0.092</td>
  </tr>
  <tr>
    <td>0.6*BagLoss + 0.4*NLI</td>
    <td>0.642</td>
    <td>0.885</td>
    <td>0.653</td>
    <td>0.914</td>
    <td>0.619</td>
  </tr>
</table>

Table 12: Instance-level Evaluation on WikiCatSum

4.2, here we describe PsLab for MAX aggregation which is used in the FirA dataset. Note that MIN aggregation is equivalent to MAX with the ordering on the labels reversed.

**Case {0, ..., C}-labels and MAX aggregation.**
The algorithm \( \mathcal{A}_{\text{PsLab}}^{\text{MAX}} \) on a given bag \( B \) and \( y_B \), outputs \( \Gamma_B : B \to \{0, \ldots, C\} \) as
1. For each \( x \in B \), let \( \Gamma_B(x) = \argmax_{\ell \in \{0, \ldots, C\}} \mathcal{M}(x)[\ell] \).
2. If \( y_B > \max_{x \in B} \Gamma_B(x) \) then:
   (a) Let \( z := \argmax_{x \in B} \mathcal{M}(x)[y_B]/\mathcal{M}(x)[\Gamma(x)] \).
   (b) Assign \( \Gamma_B(z) = y_B \).
3. else if \( y_B < \max_{x \in B} \Gamma_B(x) \) then:
   (a) Let \( S = \{x \in B \mid \Gamma_B(x) > y_B\} \).
   (b) For each \( x \in S \): let \( \ell = \argmax_{k \in \{0, \ldots, y_B\}} \mathcal{M}(x)[k]/\mathcal{M}(x)[\Gamma_B(x)] \), and set \( \Gamma_B(x) = \ell \).
   (c) If \( y_B > \max_{x \in B} \Gamma_B(x) \) repeat Steps 2(a) and 2(b).
4. Output \( \Gamma_B \).

Note that \( \mathcal{A}_{\text{PsLab}}^{\text{MAX}} \) in the {0, 1}-label case, outputs the all 0s assignment for a bag \( B \) if \( y_B = 0 \), and if \( y_B = 1 \), it first finds the max likelihood assignment, and if it is all 0s, then it sets the label of \( z \) to 1 which maximizes \( \mathcal{M}(x)[1]/\mathcal{M}(x)[0] = \mathcal{M}(x)[1]/(1-\mathcal{M}(x)[1]) \) which is the same as maximizing \( \mathcal{M}(x)[1] \). Thus, this is equivalent to the algorithm for MIN aggregation described in Section 4.2 by flipping the labels. Therefore, it suffices to prove the correctness of \( \mathcal{A}_{\text{PsLab}}^{\text{MAX}} \), as we do in the following lemma. To aid our proof, we shall use the definition of likelihood
\[
G(\mathcal{M}, B, \Gamma) := \prod_{x \in B} \mathcal{M}(x)[\Gamma(x)]
\]
for bag \( B \) and \( \Gamma : B \to \{0, \ldots, C\} \).

**Lemma J.1.** *For any bag \( B \) with aggregate MAX label \( y_B \), the output \( \Gamma_B \) of \( \mathcal{A}_{\text{PsLab}}^{\text{MAX}} \) maximizes the likelihood over the set of labellings \( \mathcal{P} := \{ \omega : B \to \{0, \ldots, C\} \mid \max_{x \in B} \omega(x) = y_B \} \) i.e., it satisfies:*
\[
G(\mathcal{M}, B, \Gamma_B) = \max_{\omega \in \mathcal{P}} G(\mathcal{M}, B, \omega)
\]
*Proof.* Let us first define another set of labellings \( \mathcal{Q} := \{ \zeta : B \to \{0, \ldots, C\} \mid \max_{x \in B} \zeta(x) \leq y_B \} \). Clearly \( \mathcal{P} \subseteq \mathcal{Q} \). We have the following lemma.

**Lemma J.2.** *Let \( \zeta^* = \argmax_{\zeta \in \mathcal{Q}} G(\mathcal{M}, B, \zeta) \) such that \( \max_{x \in B} \zeta^*(x) < y_B \), and let \( z := \argmax_{x \in B} \mathcal{M}(x)[y_B]/\mathcal{M}(x)[\zeta^*(x)] \). Then, with \( \overline{\omega} \) defined as \( \overline{\omega}(z) = y_B \) and for all \( x \in B \setminus \{z\}, \overline{\omega}(x) = \zeta^*(x) \), we have that \( G(\mathcal{M}, B, \overline{\omega}) = \max_{\omega \in \mathcal{P}} G(\mathcal{M}, B, \omega) \).*

*Proof.* Let \( \omega^* \) be a maximizer of \( G(\mathcal{M}, B, \omega) \) in \( \mathcal{P} \), i.e. \( G(\mathcal{M}, B, \omega^*) = \max_{\omega \in \mathcal{P}} G(\mathcal{M}, B, \omega) \). Now since \( \mathcal{P} \subseteq \mathcal{Q} \), we have that \( G(\mathcal{M}, B, \zeta^*) \geq G(\mathcal{M}, B, \omega^*) \). Let \( z' \in B \) s.t. \( \omega^*(z') = y_B \) which must exist by the definition of \( \mathcal{P} \). Now, for all
<table>
  <tr>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
  </tr>
  <tr>
    <td>Sentence-level Model</td>
    <td>0.613 ± 0.028</td>
    <td>0.928 ± 0.021</td>
    <td>0.709 ± 0.051</td>
    <td>0.902 ± 0.018</td>
    <td>0.993 ± 0.004</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.528 ± 0.145</td>
    <td>0.895 ± 0.037</td>
    <td>0.521 ± 0.082</td>
    <td>0.851 ± 0.055</td>
    <td>0.577 ± 0.065</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.420</td>
    <td>0.873</td>
    <td>0.496</td>
    <td>0.888</td>
    <td>0.683</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.569 ± 0.019</td>
    <td>0.924 ± 0.011</td>
    <td>0.688 ± 0.071</td>
    <td>0.904 ± 0.014</td>
    <td>0.955 ± 0.048</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.5, 0)</td>
    <td>0.582 ± 0.034</td>
    <td>0.927 ± 0.009</td>
    <td>0.534 ± 0.044</td>
    <td>0.920 ± 0.014</td>
    <td>0.606 ± 0.037</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0, 0.1)</td>
    <td>0.579 ± 0.052</td>
    <td>0.925 ± 0.017</td>
    <td>0.603 ± 0.038</td>
    <td>0.908 ± 0.020</td>
    <td>0.794 ± 0.055</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.1, 0.1)</td>
    <td>0.580 ± 0.068</td>
    <td>0.926 ± 0.024</td>
    <td>0.563 ± 0.049</td>
    <td>0.914 ± 0.028</td>
    <td>0.708 ± 0.077</td>
  </tr>
  <tr>
    <td>psl</td>
    <td>0.597 ± 0.093</td>
    <td>0.927 ± 0.004</td>
    <td>0.578 ± 0.063</td>
    <td>0.911 ± 0.009</td>
    <td>0.713 ± 0.128</td>
  </tr>
</table>

Table 13: Instance-level Evaluation on PRM800K

<table>
  <tr>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
  </tr>
  <tr>
    <td>Sentence-level Model</td>
    <td>0.648</td>
    <td>0.611</td>
    <td>0.686</td>
    <td>0.722</td>
    <td>0.418</td>
  </tr>
  <tr>
    <td>Cosine Similarity</td>
    <td>0.535</td>
    <td>0.526</td>
    <td>0.483</td>
    <td>0.893</td>
    <td>0.134</td>
  </tr>
  <tr>
    <td>Response-level Model</td>
    <td>0.491</td>
    <td>0.4643</td>
    <td>0.453</td>
    <td>0.882</td>
    <td>0.278</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.509</td>
    <td>0.5269</td>
    <td>0.5167</td>
    <td>0.814</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0)</td>
    <td>0.516</td>
    <td>0.4936</td>
    <td>0.508</td>
    <td>0.647</td>
    <td>0.715</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0, 0.4)</td>
    <td>0.527</td>
    <td>0.515</td>
    <td>0.529</td>
    <td>0.519</td>
    <td>0.763</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0.5)</td>
    <td>0.532</td>
    <td>0.522</td>
    <td>0.521</td>
    <td>0.738</td>
    <td>0.469</td>
  </tr>
</table>

Table 14: QA Preference Feedback Results (Instance Evaluation)

\( \mathbf{x} \in B \setminus \{z'\}, \mathcal{M}(\mathbf{x}, \zeta^*(\mathbf{x})) \geq \mathcal{M}(\mathbf{x}, \omega^*(\mathbf{x})) \), otherwise if there is some \( \mathbf{x} \) violating this, then one could increase \( G(\mathcal{M}, B, \zeta^*) \) by changing \( \zeta^*(\mathbf{x}) \) to \( \omega^*(\mathbf{x}) \). Thus, we can define \( \omega' \in \mathcal{P} \) where \( \omega'(z') = \omega^*(z') = y_B \) and for all \( \mathbf{x} \in B \setminus \{z'\}, \omega'(\mathbf{x}) = \zeta^*(\mathbf{x}) \), so that \( G(\mathcal{M}, B, \omega') \geq G(\mathcal{M}, B, \omega^*) \). Now, in \( \omega' \) changing the label of \( z' \) to \( \zeta^*(z') \) followed by changing the label of \( z \) (defined in the statement of lemma) to \( y_B \) yields \( \overline{\omega} \) and by the definition of \( z \), this does not decrease the likelihood, thus completing the proof.

Using the above lemma, we complete the proof by observing that at the start of Step 2(a), \( \Gamma_B = \argmax_{\zeta \in \mathcal{Q}} G(\mathcal{M}, B, \zeta) \) s.t. \( \max_{\mathbf{x} \in B} \Gamma_B(\mathbf{x}) < y_B \). This of course is true when \( \Gamma_B \) maximizes the likelihood over all labelings. A simple argument shows that this is also true if \( \Gamma_B \) is obtained from Steps 3(b) followed by 3(c) satisfying the inequality in the latter: in this case \( \Gamma_B \) is obtained from a likelihood maximizer over all labelings, and then for each \( \mathbf{x} \) whose label is greater than \( y_B \), its label is changed to the one in \( \{0, \ldots, y_B\} \) which has the maximum probability.

K Discussion on Handling Noisy Priors

In each task we explored, it was possible to devise task-specific hints that very coarsely capture the propensity or probability of a sentence-level label. Motivated by this observation, we designed the FRACTAL framework to incorporate such hints in the form of prior losses. This design choice makes FRACTAL a highly flexible framework, allowing it to integrate any task-specific priors.

For this work, we focused on two types of priors: point and pair priors, which led to substantial performance gains over the BagLoss baseline which does not leverage any prior information. However, it is important to note that these priors are not universal; rather, they can be tailored and refined to better suit other tasks as needed.

For example, in the relevance task using the MultiSpanQA dataset, the objective is to identify sentences within a paragraph that are relevant to a given question, as opposed to verifying factual correctness. In this context, cosine similarity between the question and the candidate sentences serves as a useful signal for determining relevance. If the objective were instead to assess factuality, alternative and more suitable priors could be developed within the FRACTAL framework.

We emphasize that these priors are not equivalent to the model’s final predictions. Instead, we define priors as auxiliary signals based on information available a priori. To ensure that only useful priors are incorporated, we include a weighting term for each prior in the loss function. These weights are tuned via grid search, allowing the framework to down-weight or omit priors that do not contribute significantly to performance.
<table>
  <tr>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <th colspan="4">30% I-bags</th>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.735 ± 0.023</td>
    <td>0.888 ± 0.006</td>
    <td>0.762 ± 0.018</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.605 ± 0.011</td>
    <td>0.879 ± 0.006</td>
    <td>0.713 ± 0.015</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0)</td>
    <td>0.622 ± 0.018</td>
    <td>0.887 ± 0.009</td>
    <td>0.724 ± 0.032</td>
  </tr>
  <tr>
    <th colspan="4">40% I-bags</th>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.759 ± 0.028</td>
    <td>0.887 ± 0.01</td>
    <td>0.771 ± 0.015</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.569 ± 0.017</td>
    <td>0.866 ± 0.004</td>
    <td>0.694 ± 0.027</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0)</td>
    <td>0.618 ± 0.01</td>
    <td>0.885 ± 0.006</td>
    <td>0.723 ± 0.02</td>
  </tr>
  <tr>
    <th colspan="4">50% I-bags</th>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.831 ± 0.051</td>
    <td>0.889 ± 0.062</td>
    <td>0.714 ± 0.048</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.478 ± 0.065</td>
    <td>0.829 ± 0.038</td>
    <td>0.569 ± 0.036</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0.1)</td>
    <td>0.639 ± 0.021</td>
    <td>0.881 ± 0.009</td>
    <td>0.653 ± 0.013</td>
  </tr>
  <tr>
    <th colspan="4">60% I-bags</th>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.784 ± 0.02</td>
    <td>0.886 ± 0.011</td>
    <td>0.738 ± 0.027</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.617 ± 0.021</td>
    <td>0.882 ± 0.014</td>
    <td>0.697 ± 0.022</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0)</td>
    <td>0.629 ± 0.031</td>
    <td>0.883 ± 0.01</td>
    <td>0.668 ± 0.013</td>
  </tr>
  <tr>
    <th colspan="4">70% I-bags</th>
  </tr>
  <tr>
    <td>Supervised</td>
    <td>0.721 ± 0.035</td>
    <td>0.885 ± 0.013</td>
    <td>0.743 ± 0.028</td>
  </tr>
  <tr>
    <td>BagLoss</td>
    <td>0.563 ± 0.019</td>
    <td>0.861 ± 0.011</td>
    <td>0.688 ± 0.025</td>
  </tr>
  <tr>
    <td>PriorBagLoss(0.2, 0)</td>
    <td>0.626 ± 0.017</td>
    <td>0.886 ± 0.03</td>
    <td>0.724 ± 0.033</td>
  </tr>
</table>

Table 15: Comparison of performance achieved on the WikiCatSum dataset by varying the percentage of I-bags in the training set.

<table>
  <tr>
    <th>Dataset</th>
    <th>\( \alpha_1 \)</th>
    <th>\( \alpha_2 \)</th>
    <th>\( \alpha_3 \)</th>
    <th>Learning Rate</th>
    <th>Batch Size</th>
  </tr>
  <tr>
    <td>QA-Feedback</td>
    <td>0.3</td>
    <td>0.2</td>
    <td>0.5</td>
    <td>1e-5</td>
    <td>256</td>
  </tr>
  <tr>
    <td>MultiSpanQA</td>
    <td>0.8</td>
    <td>0.2</td>
    <td>0</td>
    <td>1e-3</td>
    <td>512</td>
  </tr>
  <tr>
    <td>WikiCatSum</td>
    <td>0.7</td>
    <td>0.2</td>
    <td>0.1</td>
    <td>1e-4</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>FiRA</td>
    <td>0.6</td>
    <td>0.2</td>
    <td>0.2</td>
    <td>1e-5</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>AquaMuSe</td>
    <td>0.7</td>
    <td>0.2</td>
    <td>0.1</td>
    <td>1e-3</td>
    <td>512</td>
  </tr>
  <tr>
    <td>PRM800K</td>
    <td>0.8</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>1e-4</td>
    <td>64</td>
  </tr>
</table>

Table 16: \( \alpha_1, \alpha_2, \) and \( \alpha_3 \) represent the coefficients of the BagLoss, cosine similarity prior and correlation prior terms in the loss function.

L Ablation Study on Hybrid Datasets

Table 19 shows the performance of the fully supervised model trained on a randomly sampled subset 20% labeled training instances, along with models trained using our methods on the hybrid dataset i.e., the 20% labeled training instances along with the remaining bag-level training set. From Table 19 we observe that with even 20% instance level labels, we are able to further improve the performance of FRACTAL across all datasets in comparison to Table 4, while improving for most metrics on the supervised model trained on 20% of the labeled instances.

M Comparison of FRACTAL-RLHF with Implicit PRM

In this section, we compare our method with the recently proposed approach, Free Process Rewards without Process Labels (Yuan et al., 2024). This method (Implicit PRM) shares a similar motivation to ours: both aim to capture the benefits of process reward models (PRMs) while relying solely on response-level supervision, thereby avoiding the need for costly step-level annotations.

The key feature of their method is the training of an outcome reward model (ORM) via implicit reward modeling. Specifically, they compute process-level rewards using the log-likelihood ratios between the policy and a reference model over partial responses, and thus considers the sequence of the steps in the response.

Their approach differs in several ways from FRACTAL in which a task-specific aggregation function explicitly links process-level labels to the final outcome label. Based on this structure, we define a loss function that incorporates both the aggregation function and additional prior information to disaggregate the outcome label into process-level pseudo-labels, without considering any sequence
<table>
  <tr>
    <th>Type</th>
    <th>Summary</th>
  </tr>
  <tr>
    <td>Reference Summary</td>
    <td>She also is the first ever woman in Indian History to be nominated as the Rajya Sabha member. She is considered the most important revivalist in the Indian classical dance form of Bharatanatyam from its original 'sadhir' style, prevalent amongst the temple dancers, Devadasis, she also worked for the re-establishment of traditional Indian arts and crafts.</td>
  </tr>
  <tr>
    <td>Word Replacement</td>
    <td>She also is the first ever woman in Indian History to be nominated as the Rajya Sabha <b>Independent</b> member. She is considered the most important revivalist in the Indian classical dance form of <b>Kathak</b> from its Nautch 'sadhir' style, prevalent amongst the temple <b>singers</b>. Furthermore, she also advocated for the re-establishment of traditional Indian arts and crafts.</td>
  </tr>
  <tr>
    <td>Sentence Replacement</td>
    <td>She also is the first ever woman in Indian History to be nominated as the Rajya Sabha member. She is considered the most important revivalist in the Indian classical dance form of Bharatanatyam from its original 'sadhir' style, prevalent amongst the temple dancers. <b>She was also a strong advocate for animal welfare and environmental protection, actively participating in campaigns and legislative efforts throughout her life.</b></td>
  </tr>
</table>

Table 17: Example of the entailed and non-entailed versions of the summary from AquaMuSe Dataset. We either use entailed or non-entailed version.

<table>
  <tr>
    <th>Type</th>
    <th>Summary</th>
  </tr>
  <tr>
    <td>Reference Summary</td>
    <td>the gold spangle ( autographa bractea ) is a moth of the family noctuidae . it is found in europe , across western siberia and the altai mountains , the northern caucasus , northern turkey and northern iran . its wingspan is 42 – 50 mm . the forewings are brown and gray with large rhomboid golden marks . the hindwings and body are lighter grayish brown . the moth flies from july to august depending on the location , and migrates long distances . the larvae feed on a wide range of plants including hieracium , tussilago farfara , plantago , crepis paludosa , taraxacum , urtica , lamium , stachys and eupatorium cannabinum .</td>
  </tr>
  <tr>
    <td>Word Replacement</td>
    <td>the gold spangle ( <b>autographa californica</b> ) is a moth of the family noctuidae . it is found in <b>western north america</b> , across <b>california</b> and the altai mountains , <b>south dakota</b> and new mexico . its wingspan is 16 - 25 mm . the forewings are <b>blue</b> and gray with <b>silver-white long lateral part</b> and a <b>patch of chestnut brown</b> . the hindwings and body are <b>a grayish tan</b> . the moth flies from <b>march to september</b> depending on the location , and migrates long distances . the larvae feed on a wide range of <b>herbaceous plants including legumes such as fabaceae , alfalfas , peas , taraxacum , urtica , lamium , stachys and eupatorium cannabinum</b> .</td>
  </tr>
  <tr>
    <td>Sentence Replacement</td>
    <td>the gold spangle ( autographa bractea ) is a moth of the family noctuidae . it is found in europe , across western siberia and the altai mountains , the northern caucasus , northern turkey and northern iran . its wingspan is 42 – 50 mm . the forewing has an inner line below middle finely golden in color, and the outer one is golden at the inner margin only . the hindwings and body are lighter grayish brown . the moth flies from july to august depending on the location , and migrates long distances . <b>Occupying waste ground, gardens and moorland, this species is widespread and fairly common in the north of Britain</b>.</td>
  </tr>
</table>

Table 18: Example of the entailed and non-entailed versions of the summary from WikiCatSum Dataset. We either use entailed or non-entailed version.
<table>
  <tr>
    <th>Method</th>
    <th>AUC-ROC</th>
    <th>AUC-PR</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td><i>MultiSpanQA</i> Supervised (20%)</td>
    <td>0.658 ± 0.019</td>
    <td>0.299 ± 0.013</td>
    <td>0.828 ± 0.031</td>
  </tr>
  <tr>
    <td>FRACTAL*</td>
    <td><b>0.691 ± 0.031</b></td>
    <td><b>0.325 ± 0.027</b></td>
    <td>0.844 ± 0.051</td>
  </tr>
  <tr>
    <td><i>QA Preference</i> Supervised (20%)</td>
    <td>0.576 ± 0.007</td>
    <td>0.539 ± 0.004</td>
    <td>0.603 ± 0.012</td>
  </tr>
  <tr>
    <td>FRACTAL**</td>
    <td><b>0.585 ± 0.015</b></td>
    <td><b>0.54 ± 0.006</b></td>
    <td><b>0.607 ± 0.008</b></td>
  </tr>
  <tr>
    <td><i>WikiCatSum</i> Supervised (20%)</td>
    <td>0.773 ± 0.041</td>
    <td>0.88 ± 0.019</td>
    <td>0.652 ± 0.039</td>
  </tr>
  <tr>
    <td>FRACTAL*</td>
    <td>0.662 ± 0.016</td>
    <td>0.881 ± 0.009</td>
    <td>0.674 ± 0.005</td>
  </tr>
  <tr>
    <td><i>PRM900K</i> Supervised (20%)</td>
    <td>0.592 ± 0.012</td>
    <td>0.897 ± 0.01</td>
    <td><b>0.686 ± 0.017</b></td>
  </tr>
  <tr>
    <td>FRACTAL*</td>
    <td><b>0.599 ± 0.007</b></td>
    <td><b>0.912 ± 0.004</b></td>
    <td>0.651 ± 0.009</td>
  </tr>
</table>

Table 19: Hybrid learning ablations with 20% instance-level data. * indicates PSLAB method and ** indicates PriorsPref-BagLoss.

information. These pseudo-labels are then used to train a process-level reward model, followed by Fine-Grained RLHF.

Our method can thus be viewed as a structured preprocessing step that explicitly infers process-level information before reward modeling and fine-tuning. This modular design enables flexibility and interpretability by allowing the integration of task-specific priors and constraints.

However, the Implicit PRM method utilizes the additional sequence information, which is important in tasks such as Math reasoning. Also, in the current implementation, there is a difference is the choice of model architecture: we finetune the T5-base model, whereas their method relies on more recent and powerful models (see Table 1 of (Yuan et al., 2024)).

Reviewer paper summaries: This paper proposes FRACTAL, which consists of bag-level loss-based model training and a pseudo-labeling method. By build upon techniques from MIL and LLP, the authors disaggregate RLHF scoring from the response level to the sentence level. Overall, the method aims to reduce human annotation costs while improving model optimization.
Summary of strengths: 1. FRACTAL is both novel and interesting. Traditional RLHF is solely based on the rewards of LLM's responses, while the authors extend it to the sentence level. Theoretically, this provides finer-grained feedback for subsequent LLM optimization. Additionally, the approach relies on automatically computed prior knowledge, eliminating the need for human sentence-level annotations.

2. The paper conducts extensive experiments across four tasks and six datasets, demonstrating that FRACTAL achieves 93% of full supervision performance using only 10% of the labeled data.
Summary of weaknesses: 1. Lack of Explanation for Pseudo-Labeling Algorithm. For instance, if a bag receives a score of 1, PSLAB will assign all its sentences to 1. However, if the response contains harmful sentences, how does PSLAB identify them? Moreover, when all sentences are assigned 1, PSLAB will set the least likely one to 0. What is the intuition behind this? Why only one sentence instead of two or three? 

2. Overly Simplistic Prior Assumptions. Point prior is derived from sentence similarity, but this is not always reliable. For example, in a QA task, a sentence that merely repeats the question would receive the highest score under this assumption, even though it is not useful. Furthermore, pair prior assumes that two sentences with a high Pearson correlation should have similar scores.
However, this is not always true again. Conversely, if two sentences are weakly correlated but have similar correctness, how does Pair Prior ensure they receive similar scores?
Comments, suggestions and typos: 1. Fig 1 and its referenced text are separated by two pages. Please consider adjusting the figure’s position for better readability.

2. MIL and LLP are reintroduced in lines 127–128.

3. Some key components such as probAGG and L_\text{pr-prior} are placed in appendix rather than the main text, making the paper harder to follow. Please consider integrating these explanations into the main content.
Overall assessment scores: 4
Metareview score: 4
Metareview: The paper introduces FRACTAL, a framework designed to generate fine-grained, sentence-level labels for responses instead of a single response-level label. This approach aims to provide precise and interpretable feedback for Reinforcement Learning from Human Feedback (RLHF) optimization. The authors conducted extensive experiments across six datasets and four tasks to evaluate FRACTAL. Findings indicate that the framework achieves 93% of the performance of fully supervised baselines using only 10% of gold labels and shows a 6% improvement on downstream QA tasks.
Summary of reasons to publish: 1/ The proposed framework is novel and promising, offering enhanced feedback during optimization and significantly reducing the reliance on human-generated sentence-level annotations by leveraging automatically computed prior knowledge.
2/ The experimental design is extensive and robust, utilizing multiple QA datasets and tasks.