Generative Modeling Reinvents Supervised Learning: Label Repurposing with Predictive Consistency Learning
Authors: Yang Li
Jiale Ma
Yebin Yang
Qitian Wu
Hongyuan Zha
Junchi Yan
Venue: ICML 2025 poster (2025)
Keywords: Supervised Learning
Generative Models
Consistency Models
Primary area: general_machine_learning->supervised_learning

Abstract: Predicting labels directly from data has been the standard in label learning tasks, e.g., supervised learning, where models often prioritize feature compression and extraction from inputs under the assumption that label information is less complex. However, recent prediction tasks often face predicting complex labels, exacerbating the challenge of learning mappings from learned features to high-fidelity label representations. To this end, we draw inspiration from the consistency training concept in generative consistency models and propose predictive consistency learning (PCL), a novel learning paradigm that decomposes the full label information into a progressive learning procedure, mitigating the label capture challenge. Besides data inputs, PCL additionally receives input from noise-perturbed labels as an additional reference, pursuing predictive consistency across different noise levels. It simultaneously learns the relationship between latent features and a spectrum of label information, which enables progressive learning for complex predictions and allows multi-step inference analogous to gradual denoising, thereby enhancing the prediction quality. Experiments on vision, text, and graph tasks show the superiority of PCL over conventional supervised training in complex label prediction tasks.

Generative Modeling Reinvents Supervised Learning: Label Repurposing with Predictive Consistency Learning
Yang Li Jiale Ma Yebin Yang 12 Qitian Wu?
Hongyuan Zha Junchi Yan
Abstract
prominent example. This direct label prediction paradigm has been widely applied across various domains, from im- age classification (Krizhevsky et al., 2012; He et al., 2016; Simonyan & Zisserman, 2014), natural language process- ing (Vaswani, 2017; Devlin, 2018; Radford, 2018), to struc- tured graph learning (Kipf & Welling, 2016; Veličković et al., 2017; Wu et al., 2022) where it is considered stan- dard practice. In such systems, it is very typical to em- ploy neural network to directly map the data inputs to labels, with a particular focus on the expressive capacity of (deep) models to compress high-dimensional inputs into latent representations while preserving relevant information for accurate predictions, viewed from an information the- ory perspective (Tishby et al., 2000; Tishby & Zaslavsky, 2015). This compression is believed to contribute to the generalization ability of deep learning models, particularly in nigh-ddmeensioal and noisy input scenarios.
Predicting labels directly from data has been the standard in label learning tasks, e.g., supervised learning, where models often prioritize feature compression and extraction from inputs under the assumption that label information is less com- plex. However, recent prediction tasks often face predicting complex labels, exacerbating the chal- lenge of learning mappings from learned features to high-fidelity label representations. To this end, we draw inspiration from the consistency train- ing concept in generative consistency models and propose predictive consistency learning (PCL), novel learning paradigm that decomposes the full label information into a progressive learning procedure, mitigating the label capture challenge. Besides data inputs, PCL additionally receives input from noise-perturbed labels as an additional reference, pursuing predictive consistency across different noise levels. It simultaneously learns the relationship between latent features and spec- trum of label information, which enables progres- sive learning for complex predictions and allows multi-step step inference analogous to gradual denois- ing, thereby enhancing the prediction quality. Ex- periments on vision, text, and graph tasks show the superiority of PCL over conventional super- vised training complex label prediction tasks.
This paradigm typically assumes that the labels contain a sig- nificantly lower dimensionality and less information than the data inputs, thus guiding model designs toward compressing and extracting relevant features from the input space for ef- ficient prediction (Tishby & Zaslavsky, 2015). The assump- tion further implies that transforming meaningful latent fea- tures to label outputs is relatively straightforward compared to the challenge of extracting expressive features. However, recent advanced scenarios involve much more complex la- bels, leading to new challenges. Examples include image prediction extending dense, pixel-level outputs (Long et al., 2015; Chen et al., 2017), natural language processing tasks generating complex sentences (Brown, 2020; Touvron et al., 2023), and predicting complex structured solutions based on graph representations (Li et al., 2023b; Satorras et al., 2021). These challenges expose predictive bottlenecks due the inherent complexity of transformations from fea- tures to labels, in addition to feature extraction. To address this challenge, one approach involves learning efficient representation of the complex labels, facilitating a more effective transformation within the low-dimensional feature space. Indeed, this can correspond to methods that leverage Variational auto-Encoder (VAE) (Kingma, 2013) to perform learning tasks within the latent space (Rombach et al., 2022; Hottung al., 2021). However, this approach necessitates that the transformation between labels and latent features
1. Introduction
Machine learning has long relied on the paradigm of map- ping input data to corresponding output labels by mini- mizing prediction error, where supervised learning is a
"School Artificial Intelligence, Shanghai Jiao Tong Univer- sity "Shanghai Innovation Institute 'Broad Institute of MIT and Harvard "The Chinese University of Hong Kong, Shenzhen. Corre- spondence to: Junchi Yan <yanjunchie @ wetechh@@ju..dei sjtu.edu.en This work partly supported by NSFC 62222607. Code available at github repository.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Predictive Consistency Learning
Data Input Noised Labels
fo
x
+
Y
Data Input
fo
Label
L d(fa(x V) d(fo(x
Sample
(
X
Y
Yr
Y₁
Ye
y
== (0,Y))
Random Noise
Noised Label
Noised Label
Label
(a) Classic Supervised Learning
(b) Predictive Consistency Learning
Figure 1. Illustration of predictive consistency learning (PCL.. Unlike traditional approaches that predict labels directly from inputs, PCI. predicts labels using inputs and noise-perturbed label hints and pursues predictive consistency across different noise steps.
be reversible, requiring training additional neural networks.
formation, especially when the labels are complex, while the prediction consistency serves mechanism to propagate prediction accuracy from low-noise to high-noise conditions and constrains the invariance of the learned representation across varying noise levels, promoting the information lever- age from the input data. Fig. illustrates the pipeline.
In this paper, drawing inspiration from the concept of con- sistency mapping in generative consistency models (Song et al., 2023; Song & Dhariwal, 2024), we propose an alter- native approach to better capture complex label information by introducing fundamentally different learning paradigm. This learning scheme aims to decompose the full label in- formation into a gradual process, where the model begins by simultaneously capturing partial label details and pro- gressively approximates the complete label. To schedule the label information to be learned, we resort to the noising process of the diffusion and consistency models (Ho et al., 2020; Song et al., 2020; 2023) and generate noisy labels as additional imput hints, enabling the model to learn the complementary information alongside the noisy part. The proposed Predictive Consistency Learning (PCL) frames the learning process as learning the predictive consistency from input label hints of various noise levels toward the target labels. This process can be viewed as a label generation process conditioned on the input data. However, to align with prediction scenarios, each training instance is paired with reference target label, and the model learns to guide all denoising trajectories toward this target by enforcing pre- dictive consistency across different noise timesteps, which we define as prediction consistency.
We demonstrate the effectiveness of our approach across a range of scenarios involving complex labels from diverse domains, including vision learning, e.g., semantic segmen- tation (Long et al., 2015; Chen et al., 2017), graph learning, e.g., constrained N-body simulation (Satorras et al., 2021), I), and natural language processing, e.g., supervised fine-tuning via next-token prediction in large language models (Brown, 2020; Touvron al., 2023). The empirical results highlight the superiority of PCL traditional supervised learning across various mainstream network backbones.
2. Related Work
Learning with Deterministic Labels. Learning with deter- ministic labels, often framed within the supervised learning (SL) paradigm, focuses on training models using datasets where input-output pairs unambiguously annotated. In SL, the models typically learn the direct mapping from the data inputs to labels by minimizing the discrepancy between predicted and ground truth labels. Beyond pure supervi- sion, hybrid frameworks integrate deterministic labels with auxiliary learning objectives, e.g., semi-supervised meth- ods (Zhu & Goldberg, 2009) combine scarce labeled data with abundant unlabeled samples, and weakly supervised learning addresses scenarios with incomplete noisy su- pervision (Zhou, 2018). For theoretical analysis of these systems, the Information Bottleneck (IB) principle (Tishby et al., 2000; Tishby & Zaslavsky, 2015) indicates that mod- cls balance the trade-off between compression and predic- tion accuracy. this paper, we mainly focus on classic supervised learning scenarios to demonstrate and analyze the effectiveness of the proposed PCL learning paradigm.
Specifically, during training, unlike conventional methods predicting labels directly from inputs, PCL maps noisy la- bels varying noise levels back the true label conditioned on the input data and enforces different noise timesteps map- ping to the same target. By enforeing predictive consistency across multiple noise levels, it captures a rich spectrum of label information from entirely noisy to accurate predictions, fostering a more expressive mapping. During inference, the inherent multi-step denoising mechanism also facilitates pro- gressive refinement, resulting in more flexible and accurate predictions. Intuitively, this process can be seen learn- ing to predict with varying degrees of solution hints, which benefits learning by progressively understanding the label in-
2

---

Predictive Consistency Learning
Besides, different label utilization techniques have been proposed enhance model generalization and robustness. Examples include label smoothing that replaces hard la- bels with soft distributions to prevent overconfidence and improve calibration (Szegedy et al., 2016; Müller et al., 2019), mixup that generates virtual training examples by interpolating both inputs and labels, augmenting data and smoothing decision boundaries (Zhang et al., 2017), curricu- lum learning which organizes training samples from easy to hard, effectively exploiting label difficulty to stabilize and speed up model convergence (Bengio et al., 2009; Wang et al., 2021), and focal loss that reweights the contribution of hard-to-classffy examples by modulating label-related loss terms to address class imbalance (Lin et al., 2017). How- ever, these methods still rely on using labels solely align neural predictions, whereas we attempt to incorporate label information into the model input as reference for learning.
how much information X provides about Y. Typically, X is high dimensional variable whereas Y has significantly lower dimensionality, which generally means that most of the entropy of X is not very informative about Y and the relevant features are difficult to extract (Tishby & 7aslavsky, 2015). In deep learning (LeCun et al., 2015), deep neural networks create a compressed representation X of X through an encoder, which discards irrelevant information while preserving as much of the mutual information I(X F; Y) as possible. The compression is optimized by minimizing T(X; XT:) while maximizing T T(XE;Y).
This formulation typically assumes that Y is a low- dimensional vector (e.g. class labels) where the informa- tion content is relatively limited. However, many real-world tasks, especially in structured prediction (e.g.. image seg- mentation, sequence generation), involve predicting high- dimensional outputs. In these tasks, the mutual information I(XE: e: Y) can be difficult to maximize because the high- dimensional labels themselves contain redundancies. More- over, the space of possible outputs Y could involve complex correlations that are hard to capture directly. These learning tasks with complex labels can be characterized by label space that exhibits high complexity due at least one of the following characteristics: (i) high dimensionality, (ii) intricate internal structure, or (iii) the presence of significant dependency patterns among labels.
Diffusion Models and Consistency Models. Generative models serve widespread applications for their powerful distribution learning capacity (Cheng et al., 2022; Li et al., 2022; 2023e;a; Chen et al., 2024; Guo et al., 2024). Diffu- sion models are characterized by forward process of noise injection and a reverse process of learnable denoising, where neural networks iteratively predict data distributions condi- tioned on increasingly noisy inputs. Diffusion models have been verified in continuous space based on the Gaussian noise (Sohl-Diae Dickstein et al., 2015; Song& & Ermon, 2019; Ho et al., 2020; Song et al., 2020) and extended to discrete data with noise distributions modeled as binomial or cate- gorical variables (Sohl-Dickstein et al., 2015; Austin al., 2021; Hoogeboom et al., 2021). Building the advance- ments of diffusion models, instead of iteratively refining noisy samples through reverse diffusion process, consis- tency models (Song et al., 2023; Song & Dhariwal, 2023) leverage a self-consistency mechanism across different time steps, directly learning the mappings from noise to data in single step. This approach has shown promise in reduc- ing computational overhead while maintaining high sample quality. Recently, diffusion models have shown promise in more challenging data generation tasks like solving NP combinatorial problems (Sun & Yang, 2023; Li et al., 2023b; 2024; 2025; Zheng et al., 2024).
In contrast to traditional tasks with simple scalar or categor- ical labels, complex labels encode rich, multi-dimension or structured information. Consequently, these tasks require models to capture sophisticated relationships and dependen- cies within the label space, transcending straightforward mappings from input features. In such cases, the mutual in- formation 7 exxx; Y) becomes harder to maximize due to the exponential growth of label entropy HY) with dimension- ality, which further complicates the balancing between mini- mizing I [Xx; and maximizing (Xg; Y). One solution to this challenge is to learn an effective latent representation YE of the target Y. This concept aligns with existing ap- proaches (Rombach et al., 2022; Hottung et al., 2021) that handle high- nigh-dimensional outputs in latent spaces. However, for prediction purposes, these methods rely on the invertibil- ity of the mapping from Y to its latent representation YE, and necessitate learning additional networks to manage la- tent representations. In the following section, we propose to enhance the model's 's ability to capture [(Xg, Y) directly by leveraging the mechanism of the learning paradigm itself.
3. Preliminary and Background
Supervised learning aims to train the model to extract compressed features or representations of input data x E X while retaining the most relevant information about the target label y E Y (Tishby et al., 2000; Tishby & Zaslavsky, 2015). This is based on the assumption that the data provides sufficient information about the labels, which means the data is abundant. From an information-theorriicc perspective, the mutual information YY) quantifies
4. Predictive Consistency Learning
4.1. Intuition and Overview
To more effectively capture T(XE; Y), where Y contains substantial information, directly maximizing the mutual in-
3

---

Predictive Consistency Learning
formation between Xg and Y can be challenging. Rather than attempting to learn all of Y's information once, we propose a structured learning process that progressively captures this information. To break down the label infor- mation into a more gradual learning process, we introduce an additional noisy label Y, to regulate the amount of label information learned at each iteration. Using Y,, the original mutual information I (Xp: Y) can be decomposed as:
We start by introducing the noising processes for different labels and then thoroughly explain the training and inference scheme in the consistency learning paradigm.
4.2. Noise Scheduling for Labels
This section elucidates the noising processes designed to gradually incorporate noise across various label spaces.
Diffusion on Categorical Labels. For multi-dim categor- ical labels in {1,... KJ where K denotes the category number and N denotes the dimension, we follow discrete diffusion models (Sohl-Dickseer et al., 2015; Austin et al., 2021; Hoogeboom et al., 2021) model the noising pro- cess introducing multinomial noise to the label each timestep t. We represent the label as E {0, 1JNXX which is concatenation of N one-hot vectors. The noise can be understood as transitioning between different categories for each of the N dimensions. Starting from the initial point yo = y, the noising process is defined as:
((X=;Y)= = 1-0--I(xa I(xE; (I)) + Y.) - - ((XE;
(1)
Since Yú is derived from Y, it does not provide any more information about X g if Y is given, thus the redundancy term T(Xe: YI(Y)= = 0, simplifying to:
Ex Y) ([x + I()::Y)
(2)
The decomposition reveals two key components, where the first term, [[x;; Y|Y)), captures the incremental informa- tion about Y that can be learned given y, This term serves as a lower bound of / [(Xg: Y), and the gap between them can be controlled by the information content of Y,. By optimizing Xg through maximizing Eg (x( Y[Y)), the model learns to progressively capture the full information content of Y. Specifically, when i + T, Y, provides lit- tle information, and the model is forced to fully capture T(XE: F; Y). When i. + 0, Yr approximates Y, allowing the model to focus on refining details of the label. During training. by sampling a batch of random / values, the model simultancously learns to capture different aspects of the label. Initially, Xg is expected to easily capture partial de- tails of the label, and through iterative training, gradually accumulates the full information content of Y.
g(velve-1)
(4)
where Cat(y: p) is categorical distributions over N one- hot vectors with probabilities given by p, and Q: = (1 Be I 8/KK1 E IR K is the transition matrix, which determines the corruption introduced at timestep 4, where 3, is the corruption rate at timestep t. This ensures that with probability Se, the corresponding label category can transition to any other category. Over time, as / approaches the final timestep T, the labels converge towards uniform distribution over the K categories. The cumulative effect of the diffusion process after t steps is:
For implementation, the model is exposed to noisy versions of Y. The inputs to the model include both X and Y,, where Y2 serve as the condition. While introducing Y, as an aux- iliary input facilitates learning, the ultimate goal is for the model to make predictions relying on Y, as little as possible. Formally, we aim to minimize the noise-conditional depen- dency To (Y; Xi)X), which measures the extent to which the model's 's predictions depend on the noisy label Y., Ideally, this term should be zero, indicating that the model's pre- dictions are independent of Y given X and the model 0. Mathematically, it can be measured as:
(yyr = Cat(y;; Pa Qr),
(5)
where Q2 = Q1 Q2 ... Q2 represents the accumulated tran- sition matrix from yo to yr-
Diffusion on Continuous Labels. For multi-dim contin- uous labels in R.V, where N denotes the dimensionality, we follow Gaussian diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Nichol & Dhariwal, 2021) to model the diffusion process as introducing Gaussian noise to the label at each timestep. At each timestep t, Gaussian noise is applied to corrupt the label, progressively pushing it toward noisy distribution. The noising process is defined as:
TY;Y; Y| = Ex,y, (DKL (Y|x,
(3) To achieve this, we introduce the prediction consistency term, which enforees (Y |X, Y,,t) = pe (Y Yv, t') for all t' This regularization ensures that the model's predic- tions are consistent across different noise levels, reducing the dependency on Y; and encouraging XE to encode all necessary information for accurate predictions.
g(y:ly=-1) = N(y: / 3,,--1,
(6)
where AV μa, >) is a Gaussian distribution with mean μ and covariance Σ. and 3r controls the variance of the added noise at timestep l. The factor vT 3r ensures that the label retains some of its original value, while the noise is introduced with variance 3, Over time, as t approaches the final timestep T. the labels converge towards a Gaussian cen- tered at zero The cumulative effect of this diffusion process
The left part of the section outlines the detailed design of the proposed predictive consistency learning framework.
4

---

Predictive Consistency Learning
1 p(y)) =
label, i.e., fo y1, () so y2, y for distinct tra- jectories and (from independent noising processes) distinct steps & and t', as shown in Fig. 2.
N
Trajedories
p(ylx) =1 1
:
To6x v..
To achieve the consistency to learn f x N, given that the target y is certain and explicit, we do not have to merely rely on optimizing the expectation of the variation of the consistency mappings over two noise points Yt and Yv to propagate the label information across different noise levels generative consistency models do (Song et al., 2023). Instead, we additionally introduce y to optimize the triadic distance to achieve prediction consistency:
14.77)
fx
(5.0
Conditioned on Data Input x
LPCI. (0) =B =B + (ox (),)))
Figure2 2. Prediction consistency enforces all trajectories condi- tioned on x consistently map to the same initial point, i.e. label y.
+
(8) Here .) is distance metric function and X1, X2 are loss weights Specifically, to align with the traditional supervised training paradigm, we retain the original task-defined loss function for the distance metric d, such as cross cross-entropy for classification tasks and mean squared error for regres- sion tasks. This is because the design of the loss function is orthogonal to our learning framework, allowing them to complement each other. The main modification in our ap- proach lies in that the model predicts y based on both x and the noise-perturbed versions of y, while ensuring predictive consistency across different noise levels. In practice, rather than randomly sampling two timesteps i and f2, we explic- itly control the noise gap between them by setting to at11 This ensures that the predictive consistency remains strong and informative constraint to learn. We then independently sample from the noise distribution to obtain yi, and yi, to ensure that the two noisy samples are independent with respect to both the time steps and the diffusion trajectories. Then Eq. 8 can be effectively optimized to learn the consis- teney predictive mapping, and the whole training process presented in Alg. and Fig. 1.
after t steps is described by the marginal distribution:
q(yrlyo)= = NN Varyo, (1 a,)1),
where ā₂ -
B.) is the accumulated noise scale from the original label yo the noisy label yr- As in- creases, 0g decreases, leading to increased label corruption.
Diffusion on Embeddings. In scenarios where the label is too complex to be directly expressed as a categorical or continuous value, or when the number of categories is excessively large, we introduce Gaussian noise directly to the latent embeddings of the labels in a manner consistent with the noising process used for continuous labels.
4.3. Predictive Consistency Training Scheme
We draw inspiration from the consistency mapping concept in consistency models (Song et al., 2023; Song & Dhariwal, 2024) and define the predictive consistency mapping for prediction scenarios as fo (x, Fr, + y. Correspond- ing to the discussions in Sec. 4.1, we expect the training scheme to satisfy two properties: 1) fo can nearly recover the label y given the inputs x, yr, L, where t informs the model of the noise level; 2) the prediction consistency that fo(x, yt,t) yx,, t') holds for all f, The key dis- tinction between the proposed predictive consistency map- ping and the generative consistency mapping is that in the former, the diffusion trajectories are conditioned on the data input x with a reference target label y. PCL aims to recover the exact y given x, where the target distribution converges to an exact target point, and the model trades the output diversity to better capture y. This requires the prediction to focus on the condition x, with predictive consistency forced across all possible label noising trajectories instead of satisfying self- self-consistency (Song et al., 2023) within merely each generative trajectory. The complete predictive consis- tency condition can be expressed as follows: conditioned on data input x, all points along any trajectory map to its
4.4. Multistep Inference with Consisteney Mappings
With a well-trained fe(:...), we obtain predictions for a given by sampling yr from the uniform distribution and then evaluate yo = fo (x, yr, T'). This standard single-step inference requires only one forward pass through the model, akin to conventional direct predictions. During training, the accuracy tends to be higher when is small, as the label hints contain a richer amount of information. Our objective is to progressively transfer this high accuracy larger values of through training, thereby enhancing overall model perfor- mance. In the ideal case that the consistency loss converges to zero, optimal results can be achieved in a single step, yet in practice, gradually decreasing t from I" to 0 can lead to accuracy improvements. To achieve such enhancements, a multistep inference strategy can be adopted, which itera- tively alternates between denoising and reintroducing noise.
5

---

Predictive Consistency Learning
total number of time steps 10 and use the validation identify the step that best balances this tradeoff and achieves the highest prediction accuracy. During testing, we perform early stopping at this identified inference step. The specific multistep prediction procedure is in Alg. 2.
Algorithm Predictive Consistency Training
1: Input: Dataset D, model fo. noise function q(-), leam- ing rate 7/, loss weights A1, A2, noise gap a
2: repeat
3: Sample (x D, and U[1, 12 ah
4: Sample Yt, ~ q(yr, (y), V12 ((v)2ly)
5. Experiments
5: vi t (1,
6: 2 t xx, Y12, t2)
Since PCL is proposed as a novel training paradigm, the major baseline for comparison is conventional supervised learning. The comparisons are conducted across various classic and representative model backbones for different modalities to showcase the general applicability of PCL. We test the proposed PCL. framework on tasks involving com- plex labels from diverse domains, including semantic seg- mentation (high-dimensional categorical outputs in vision learning), N-body simulation (high- nigh-diimensioal continuous outputs in graph learning), and next-token prediction (high- dimensional sequential outputs in language modeling).
7:
t A1d( yoi y) + A y) + A2d(58 s88)
8:
0 + 0-- - VaL
9:
until convergence
Algorithm 2 Multistep Prediction
Input: model fg. data input x, noise function q()), time steps T > T2 > ... > TN, -1 preset inference step N;
Sample random noise YT Yo + fx(x, yr, T) for n = 1 to N; - do Sample yr, ~ (0yr 180) ýo < (x. Tr)
Experiments for constrained n- u-body simulation are con- ducted on a single GPU of NVIDIA RTX 4090. For se- mantic segmentation, a single NVIDIA H100 GPU was employed, and experiments for next-token prediction performed on 8 GPUs of NVIDIA H800.
end for
Output: Prediction ýo
5.1. Constrained N-body Simulation
This approach effectively trades off runtime for enhanced prediction quality, allowing the model to refine its outputs over multiple inference steps and leverage increasingly rich information embedded in earlier predictions.
The constrained N-body simulation (Huang et al., 2022) task involves predicting the future positions of a set of interacting particles over time based on initial conditions such as their positions, velocities, and their interactions from constraints like sticks and hinges indicating connection relations, as well as the inherent physical forces. The evolution of par- ticle positions and velocities follows physical laws. We incorporate tasks with varying complexities by controlling the number of isolated particles p. the number of sticks and the number of hinges h. The configuration is abbre- viated as (p, s, h) and we follow (Huang et al., 2022) to incorporate five tasks corresponding to configurations of (1, 2, 0). (2,0, 1). (3, 2, 1). (0, 10, 0), and (5,3,3)
Given a sequence of time points Ti > T2 >.... -N at each step Tu. the current prediction Y-.-1 is perturbed by noise function a state Yr,. The noise level decreases with each step, meaning Tn < Tn-1- The model then denoises the corrupted label by applying fo(x, Yr., Tr), producing refined prediction. This process is repeated over successive steps, where each newly refined label incorporates progres- sively more accurate information from the previous step. This enables the model gradually recover the whole infor- mation of y by taking the perhaps approximated prediction as the label hints and leveraging the incrementally informa- tive hints for the final prediction.
Dataset. We collect 5000 trajectories for training, 2000 for validation, and 2000 for testing for each configuration. Each trajectory spans 1000 timesteps. For each trajec- tory, the initial conditions include the particle positions p(0) E Rxx3 the initial velocities v(0) E R and the re- spective charges E {-1, There exist stick and hinge connections among the particles. The task to predict the positions of the five particles after 1000 timesteps.
In practice, we observe that as the number of inference steps increases, the model's prediction error initially continues decreasing, but may then rise for a period, as shown in Fig. 3. This phenomenon may result from the fact that during training, the model is only exposed to the noise- perturbed versions of ground truth labels as inputs. However, during inference, the intermediate predictions of the model may contain errors, which can accumulate over multiple steps of inference. Consequently, there exists a tradeoff between gradually capturing finer prediction details and accumulating prediction errors. To address this, we set the
Metrics. We evaluate the prediction error by Mean Square Error (MSE), i.c., the average of the squares of the errors between the predicted values and the true values.
Model Design. We consider the state-of-the-art graph mod-
6

---

Predictive Consistency Learning
Epoch
Epoch 200
Epoch 400
Epoch 600
012
00403
0.03000
Prodiction Enter
Preditionrre Ereer
5.3225
Prodicion Eicor
Predicion
00400
0.33000
0.284
0.81
00292
503160
0.10
00390
093125
0.290
5 10
B 10
Number of Inference Steps
Number of inferencce Steps
Number of Infarence Shups
Number of Infarer os Staps
Figure The influence variation inference steps across the training process.
Table 1. Prediction error (x 4) of SL and PCL on top of graph models on various types of N-body simulation systems. The header of each column "p, 8, ht" denotes the scenario with p isolated particles, sticks and & hinges.
Backbone Model
Training
1,2,0
2, 2, 1
3, 1
0, 10,0
5,3,3
SL
2.5344000061 2.865++0.021
3.479:10:110
4.705:10.046
4,303::0.002
GCN
PCL
2.406.1...000
2.268:10.012
2.795.10.06
3.162.10.251
3.228.102221
SL
2.921110.198
2.707:10.024
3.351
3.478.10.422
3.407
GAT
PCL
2.771 2.5811:.0266
2.802:1.216
2.481=0.059
2.534
SL
3.013::0.022 2.716:00068
3.2934:0023
4.426#:.044
4.148:0.035
GGNN
PCL
2.61440.031
2.297+1.003
2.974+0.011
3.191=:.290 3,457+:0213
(GAT) (Veličković al., 2017), and Gated Graph Neural Networks (GGNN) (Li et al., 2015). Table shows the superiority of PCL on quantitative results with significantly lower estimation error across five tasks with different com- plexity. As can be discovered, the performance gains are more pronounced on more complex datasets.
Table Ablation study loss construction top of different backbones reflected by prediction error (x10 1x10-3). 2).
GCN
GAT
GGNN
Traditional Ateremm w/ Ag-term
3.479:10.1 6.784:10.86 862
3.35110.111
3.293:10.023
4.23410.170
3.73140.915
termm, Ay-term
2.881=0.110
3.001=:0.074
3.045::..00
A1 heeeem, Ag-term
2.795
2.974+0.011 0.80
Table 2 provides ablation the effects of the A1-term and Az-term in Eq. 8. Az-term corresponds to the design from the generative consistency models that enforce the self-consistency loss, which is less suitable for this scenario. Compared to merely adopting the Az-term, Az-term can serve as a regularization term to more directly enforce pre- dictive consistency, resulting in certain performance gains.
eling solution for this task, and the model is optimized by minimizing the averaged MSE between the predicted posi- tions and the ground truth positions. We input the concate- nation of the initial positions and the velocities as the node features. The edge feature is provided by concatenation of the product of charges G.2y and an edge type indicator Tij. where 4jj is valued as 0 if node i and j are disconnected, 1 if connected by stick, and 2 if connected by hinge. We take the model outputs as the estimated positions. To introduce PCL, adopt two linear layers encode the input attributes and the noised label, respectively, and then concatenate them to form the input hidden feature to the sub- sequent graph neural layers. We adopt 4 graph neural layers, and for each layer's output, we integrate the timestep em- bedding extracted by the sinusoidal position embedding and linear layer through addition. In this task, y R2xx and adopt the Gaussian noising process to produce noised labels as shown in Eq. 7.
Fig. 3 illustrates the testing performance across different inference steps in a 10-step prediction process. In the early stages, increasing the number of inference steps clearly re- duces prediction error. However, as training progresses, the trend becomes more variable, first decreasing, then increas- ing, and finally decreasing again. This behavior may stem from the trade-off between capturing finer details and the accumulation of errors as discussed in Sec. 4.4.
5.2. Semantic Segmentation
Semantic segmentation classic dense vision task that involves classifying each pixel of an image into prede- fined category (Long et al., 2015; Zhao et al., 2017). Unlike classification tasks that categorize entire images, semantic segmentation analyzes the finer granularity of images to identify the boundaries and relationships between objects.
Results. For graph modality, we compare PCL and the conventional supervised learning on top of classic graph neural networks, including Graph Convolutional Networks (GCN) (Kipf& Welling, 2016), 6), Graph Attention Network

---

Predictive Consistency Learning
Test Image Ground Truth & Classic SL
PCL
t= 1000
691
= 413
= 191
Label Hint
Output
Figure Predictions across varying timesteps based on the last step's predictions the multistep inference procedure. In each step, the model receives the input and label hint and predicts the output.
Label Hint ye from Last Step
Difference between ye', 1) and the Last-Step Prediction
Ground Truth
800
600
400
200
Figure 5. The visualization of the prediction improvements with different t controlling the prediction granularity.
plays a critical role in various applications, such as au- tonomous driving (Badrinarayanan et al., 2017), medical imaging (Ronneberger et al., 2015), and scene understand- ing (Long et al., 2015).
Table 3. Results on Semantic Segmentation.
Backbone Model
Training
Pixel Acc.t
mloUt
Scoref
SL
75.53
33.13
54.33
MobileNerV2dilated
PCL
76.42
35.27
55.85
Dataset. We utilize the ADE20K dataset (Zhou et al., 2019), which is a commonly used large-scale scene parsing dataset that contains over 20K images with pixel-level annotations. The dataset annotated with 150 different object classes, and make the unannotated pixels -1, which are ignored during training and testing. Following Zhou et al. (2017; 2019), resize the original images during training while maintaining a constant aspect ratio, randomly scaling the shorter side to one of the sizes: 300, 375, 450, 525, or 600.
SL
78.98
41.49
60.24
PPsNer500diateedPPPM
PCL
82.33
48.56
65.45
sampled labels are then fused with the image features ex- tracted by the encoder. Finally, the fused image features are fed into the decoder for further processing. In this task, E {-1,.., 149] HW where each entry will be converted into one-hot vector of length 151, indicating the classifica- tion of pixels. We adopt the categorical noising process in Eq. 5 using transition matrices of Q2 E [o, []111x15]
Metrics. Following Zhou et al. (2017; 2019), we adopt three evaluation metrics to measure model performance: 1) Pixel Accuracy: the proportion of correctly classified pixels. 2) Mean IoU (mloU): the -overcctioonovee-uiiin between the predicted and ground-truth pixels, averaged over all the classes. 3) Score: the average value of Pixel Accuracy and Mean loU. During the testing phase, we use Multi-Scale Test: evaluate at multiple sizes and then take the average.
Results. For the encoder, we choose ResNeet0Odilated (He et al., 2016) and MobileNetV2ddiltte (Sandler et al., 2018). For the decoder, Pooling Module) (Zhao et al., 2017) with DeepSup (deep supervision trick) and C1 (one convolution module) with DeepSup. Table 3 shows PCL's 's superiority over SL. Fig. 4 visually demonstrates how increasing inference steps fur- ther improves predictions, particularly for large background areas. Fig. 5 shows the prediction improvements on top of the last-step prediction with varying t. Compared to the previous step's prediction ýo with its noised version yr as the label hint, for the prediction fx, ye, t), setting a larger tends to encourage the model to improve broader structural relationships, while setting a smaller t encourages the model to focus on finer details, such as object boundaries.
sequentially selected PPM (Pyramid
Model Design. We adopt an encoder-decoder framework. The encoder compresses the input by extracting high-level features using CNN backbone, reducing the spatial res- olution while capturing important semantic information. The decoder then progressively upsamples the compressed features to recover the original resolution. To introduce PCL, we first downsample the noised labels processed by the embedding layer using the timestep embeddings ex- tracted through sinusoidal position embedding. These down-
8

---

Predictive Consistency Learning
5.3. Supervised Next-token Prediction Fine-tuning
Table 4. Evaluation on I.IM fine-tuning. Relative performance
The next-token prediction task is cornerstone in natural lan- guage processing, forming the foundation for transformer- based large language models (LLMs) such as GPT (Radford, 2018), LLaMA (Touvron et al., 2023). The objective of the task is to predict the next token in a sequence, given the preceding context. The task's outputs retain similar dimen- sionality and sequence complexity as the inputs. Models trained on next-token prediction tasks have proven to be highly effective at capturing both short-term and long-range dependencies in language, enabling them to generate coher- ent, contextually appropriate text. This section investigates the effectiveness of PCL in the full fine-tuning task on the pre-trained I. L-aMa-2-7B (Touvron et al., 2023) models.
improvements compared to the raw model are marked in brackets.
Backbone
Training
MMLU
CRASS
BBH
Raw Model
41.90
37.59
32.93
SL.
46.22
58.29
33.38
Full FT
PCL
47.10
59.48
34.75
relationships between different entities or events. We use 3-shot direct prompting to assess model reasoning.
Model Design. Our modified LLaMA2-7B retains the orig- inal embedding and decoder layers but introduces a novel mechanism to predict the next token. Instead of the standard approach where hidden states directly generate token proba- bilities, we inject Gaussian noise into token embeddings in training to enhance robustness. This noise is combined with the tokens hidden state to form an augmented vector, which is passed via an MLP and classification head for predic- tion. Additionally, we enforce consistency by minimizing the mean squared error between logits at randomly selected time steps. the generation phase, noise is iteratively re- duced across steps generate tokens sequentially. More detailed descriptions can be found in Appendix B.3.1.
Dataset and Metries. The Alpaca (Taori et al., 2023) dataset is based on the self-instruct method (Wang et al., 2022), utilizing the OpenAl text-davinci-005 engine to gen- erate a collection of instructions and demonstrations, These instruction data can be employed for fine-tuning language models. By filtering out low-quality data, such as hallucina- tions, incorrect answers, and unclear instructions, we obtain the Alpaca-cleaned dataset, which serves as the sole training data for all models discussed in this paper.
To evaluate the performance of models, we employ IN- STRUCTEVAL (Chia et al., 2023), comprehensive evalua- tion suite designed specifically for instruction-ture tundd models. INSTRUCTEVAL aims to assess models across dimensions such as problem-solving ability, writing proficiency, and alignment with human values. Following INSTRUCTE- VAL, the evaluation of large language model performance in this paper includes benchmarks:
Results. We compare methods for fine-tuning LLMs on the next-token prediction task. The baselines include the pre-trained LLaMA2-7B model and the LLaMA2-7B model fine-tuned using traditional full- -ull-parmmeter supervised learn- ing (Taori et al., 2023). In contrast, our method also tunes the full parameters of the model but operates under the pre- dictive consistency learning paradigm. Table 4 shows the superiority of PCL on quantitative results across MMLU, GRASS and BBH.
MMLU (Hendrycks al., 2020) assesses models' world knowledge and reasoning abilities across 57 academic disciplines. Questions range in difficulty, presented in a multiple-choice format. The evaluation primarily uses few-shot settings to test the generaliza- tion capabilities. A 5-shot direct prompting utilized for evaluation.
6. Conclusion and Future Work
This paper has proposed novel predictive consistency leam- ing framework beyond previous methods with direct predic- tion, aiming to explore the full potential of label information for supervision during the learning process. It has shown superiority in experiments on various tasks across vision, text, and graph modalities with complex labels. It resembles, to a certain degree, the residual learning scheme by treating the noisy label as the input, which serves as counterpart to the raw signal of the input data. Future work will explore the potential of applying label diffusion to other fundamen- tal deep learning paradigms, such as semi-supervised and weakly supervised learning. It will also explore more effec- tive label granularity, e.g., treating entire sentences as label hints rather than individual tokens, and design more natural label noising mechanisms to enhance learning.
BBH (Srivastava et al., 2022) is a subset of 23 chal- lenging tasks from the BIG-Bench benchmark. It eval- uates the ability to handle challenging reasoning and problem-solvine tasks that go beyond simple language understanding. We apply 0-shot direct prompting to measure the model's capability dealing with unseen questions without additional contextual examples.
CRASS (Frohberg & Binder, 2022) evaluates models' ability to handle complex relational reasoning tasks, specifically in the context of causal structures and re- lationships. It includes a variety of problems that test how well the model understands and predicts causal

---

Predictive Consistency Learning
Impact Statement
Chia, Y. K., Hong, P., Bing, L.., and Poria, S. Instructeval: Towards holistic evaluation of instruction-tunet large lan- guage models. arXiv preprint
This work introduces a novel training paradigm that funda- mentally rethinks the traditional direct mapping approach in supervised learning and beyond. Specifically designed to address the growing challenge of capturing complex, high- dimensional labels in modern prediction tasks, the method decomposes full label information into a progressive learn- ing procedure and leverages noise-perturbed labels as ad- ditional references. As a generalizable framework, PCL has the potential to fundamentally improve wide range of applications in way that orthogonal to network ar- chitectures and loss function designs. Beyond its practical impact, this work also serves as exploration to answer fundamental question: When labels contain rich informa- tion, can they be leveraged to facilitate learning rather than simply being treated as prediction targets? By demonstrat- ing the effectiveness of incorporating label information into model input for reference, this study opens new avenues for rethinking how labels are utilized in machine learning.
2023.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: htn::880.aa 1810.04805, 2018.
Frohberg, J. and Binder, F. Crass: A novel data set and benchmark to test counterfactual reasoning of large lan- guage models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2126-2140, 2022.
Guo, Z., Li, Y., Liu, C., Ouyang, W., and Yan, J. Acm-milp: Adaptive constraint modification via grouping and se- lection for burdness-preservve milp instance generation. The Forty-first International Conference on Machine Learning, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-777777777777777 2016.
References
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete state-spaces, Advances in Neural Information Processing 2021.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXi::2009.03300 2020.
Systems,
Badrinarayanan, V., Kendall, A., and Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for im- age segmentation. IEEE transactions on pattern analysis and machine intelligence. 39(12):2481-2495 2017.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems,
2020.
Hoogeboom, E.,, Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34: 12454-124655 2021.
Bengio, Y., Louradour, J., Collobert, R., and Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Hottung, A., Bhandari, B., and Tierney, K. Learning a
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
latent search space for routing problems using variational autoencoders. In International Conference on Learning Representations, 2021.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. I.. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 8488))834-848 2017.
Huang, W., Han, J., Rong, Y., Xu, T., Sun, F,, and Huang, J. Equivariant graph mechanics networks with constraints. arXiv preprint arXiv:2203.06442 2022.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013.
Chen, X., Li, Y., Wang, R., and Yan, J. Mixsatgen: Learning graph mixing for sat instance generation. In The Twelfth International Conference on Learning Representations, 2024.
Kipf, T. N. and Welling, M. Semi- supissuper classifica- tion with graph convolutional networks. arXiv preprint 2016.
Cheng, R., X.,
Ye, J., Hao, J., and Yan, J. The policy- -rrcc-ggrdienttt placement and generative routing neural networks for chip design. Advances in Neural Information Processing Systems, 35:26350-2633 26362, 2022.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
10

---

Predictive Consistency Learning
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436 444, 2015.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Li, Y. Tarlow, D., Brockschmidt, M., and Zemel, R. Gated graph sequence neural networks. arXiv preprint 2015.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-asisted intervention-
Li, Y., Mo, Y., Shi, L., and Yan, J. Improving generative adversarial networks via adversarial learning in latent space. In Advances in Neural Information Processing Systems, 2022.
2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 241. Springer, 2015.
Li, Y., Chen, X., Guo, W., Li, X., Luo, W., Huang. J., Zhen, H.-L., Yuan, M., and Yan, J. Hardsatgen: Understanding the difficulty of hard sat formula generation and strong structure- hardness-aware baseline. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2023a.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Li, Y., Guo, J., Wang, R., and Yan, J. T2t: From distri- bution learning in training to gradient search in testing for combinatorial optimization. In Advances in Neurul Information Processing Systems, 2023b.
Satorras, V. G., Hoogeboom, E., and Welling, M. E (n) equivariant graph neural networks. In International con- ference on machine learning, pp. 2021.
PMLR,
Li, Y., Shi, L., and Yan, J. lid-gan: an iid sampling perspec- tive for regularizing mode collapse. International Joint Conference on Artificial Intelligence, 2023c.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Li, Y., Guo, J., Wang, R., Zha, H., and Yan, Fast 12t: Opti- mization consistency speeds up diffusion- training- to-testing solving for combinatorial optimization. Ad- vances in Neural Information Processing Systems, 2024.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan. N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. International Conference on Machine Learning, pp. 2256-2265, 2015.
Li, Y., Ma. J.. Pan. W., Wang, R., Geng, H., Yang, N., and Yan, J. Unify ml4tsp: Streamlining the design space of ml4tsp suggests principles for learning and search. International Conference on Learning Representations, 2025.
Song, J., Meng. C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Song, Y. and Dhariwal, P. Improved techniques for training consistency models. arXiv preprint vrXiv22310.14189 2023.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, 2017.
Song, Y. and Dhariwal, Improved techniques for training consistency models. In The Twelfth International Confer- ence Learning Representations, 2024. URL https:
pp.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1431-3440, 2015.
/ / /oppneevieww... net forum? htt:///ommmmm//////////aaa Mi--NNZyyRRDDS
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Müller, R., Kornblith, S., and Hinton, E.
E. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. arXiv preprint arXiv:2303301 2023.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. International Conference on Machine Learning, pp. 8162-8171, 2021.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Sarriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04610,,04611.... 2022.
Radford, A. Improving language understanding by genera- tive tra-tingning. 2018.

---

Predictive Consistency Learning
Sun, Z. and Yang, Y. DIFUSCO: Graph-based diffusion solvers for combinatorial optimization. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL
Zheng, X., Li, Y., Fan, C., Wu, H., Song, X., and Yan, J. Learning plaintext-ciphertext cryptographic problems via anf-based sat instance representation. Advances in Neural Information Processing Systems, 2024.
http://pwweeieeeeeeeeeeeeeo net/ id=Jv8Ff0lgVV.
Zhou, B., Zhao, H., Puig. X., Fidler, S., Barriuso, A., and
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 633 001, 2017.
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso. A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-3211 -2,30, 2019.
Taori, R., Gulrajani, I. Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction followiinn Ilama model, 2023.
Zhou, Z.-H. A brief introduction to weakly supervised learning. National science review, 5(1):44-53, 2018.
Tishby, N. and 7aslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. IEEE, 2015.
Zhu, X. and Goldberg, A. Introduction to semi-supervised learning. Morgan & Claypool Publishers, 2009.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004557, 2000.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2300. 2023.
Vaswani, Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. arXiv preprint arXiv:1710, 10903, 2017.
Wang, X., Chen, Y., and Zhu, W. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(5):4555 4576, 2021.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint 12 10560, 2022.
Wu, Q., Zhao, W., Li, Z., Wipf, D., and Yan, J. Nodeformer: A scalable graph structure learning transformer for node classification. In Advances in Neural Information Pro- cessing Systems, 2022.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0941,, 2017.
Zhao, H.. Shi, J., Qi. X., Wang, X., and Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881- 2890, 2017.
12

---

Predictive Consistency Learning
Appendix
A. More Discussions
A.I. Computational Overhead in Training and Inference
For training, since the loss calculation requires two inference predictions with different noise levels during training, requires twice the training cost of the traditional supervised learning paradigm. Moreover, typically takes less than twice the iterations for the model converge.
For inference, PCL can produce much superior predictions with merely single forward pass SL does. Meanwhile, PCL can achieve better performance with more forward passes. As shown in Fig. 3 using a GCN backbone on the 3,2,1 dataset, even single-step PCL. inference achieves superior performance (prediction error 0.02835) compared to SL. (0.03478), while additional inference steps (e.g., 5-step) can further refine results (0.02795).
A.2. Comparison to Curriculum Learning Strategies that Gradually Increase Label Complexity
The key differences between predictive consistency learning (PCL) and curriculum learning (CL) can be summarized as follows:
Learning Target Consistency. CL predefines staged learning targets (e.g., coarse-to-fine fine labels), where each stage learns approximation of the true label. This risks error accumulation, i.e., biased features from early stages may propagate to later stages. In contrast, the objective of PCL remains stable: always predicting the true label, with complexity dynamically adjusted by the input noise level.
Progressive vs. Simultancous Learning. CL follows fixed, sequential progression (e.g., easyhard labels). PCL, however, randomly samples time steps during training, enabling the model to learn various label information levels (from partial to complete) simultaneously, where the time step acts as controller for label granularity.
Controllable Prediction. PCLs noise-conditioned framework allows explicit control prediction granularity via time step As evidenced in Fig. 5, setting larger I tends to encourage the model to improve broader structural relationships, while setting a smaller encourages the model to focus on finer details. Further, PCL supports multi-step inference (Sec. 4.4), where predictions are iteratively refined (analogous to diffusion denoising), boosting final accuracy.
Leveraging Partial Labels Input. One of the motivations of PCL is to treat labels not just as targets but learning facilitators. By feeding partially noised labels (e.g., "hints") during training, the model learns to exploit intermediate information (similar to how humans use reference solutions to help solve problems).
B. Experiment Details
B.1. Experimental Details for N-body Simulation
B.1.1. NOISING PROCESS
At timestep the label y e 5x3 represents the original 3-dimensional coordinates of the 5-body system. We introduce Gaussian noise that gradually transforms the coordinates points from the standard Gaussian distributions. The noising process simply follows Eq. 6 and Eq. 7.
B. 1.2. MODEL ARCHITECTURE
We follow the implementation of (Satorras et al., 2021) to generally implement 4-layer GNNs. With its learnable edge operation function de and node operation function dr. the graph convolutional layer follows:
my = de (h), h1 aiz))
(9)
m; = Σ mij
(10)
jEN(
[+1 = on (h/ m,)
(11)
13

---

Predictive Consistency Learning
Where h IRa is the nf-dimensional embedding of node vi at layer I. Ozy are the edge attributes. Miii represents the set of neighbors of node U/-- Here, de and Ob are approximated by 2-layer Multilayer Perceptrons (MLPs).
The initial position p" and velocity v from the particles are passed through linear layer to obtain the input feature. The label hint is passed through another linear layer, and the obtained feature is concatenated with the input feature and inputted into the GNN first layer h The particle's charges are inputted as edge attributes dij = Gzj-- The time step is first embedded through the sinusoidal position embedding
concat
sin T
COS
sin
COS
COS
(12)
T
T
T
T
T
and then processed through linear layers and activation functions. Here d is the embedding dimension, T is large number (usually selected as 10000), concat(-) denotes concatenation. Then aggregate the timestep feature with the node convolutional feature and reformulate the update for node features, i.e., Eq. 11 as:
h on Φn m.) + (t)
(13)
where 6h is a linear layer model. The output of the GNN h is passed through a two layers MI. that maps it to the estimated position.
B.2. Experimental Details for Semantic Segmentation
Given image information x extracted through the encoder, hint label and time step 1, PCL first embeds the latter two and then down-sample the the embedded hint label j-with the time step embeddings t to accommodate the dimensions of x. Then PCL merges the image features and the noise labels containing time step information into a new image information X.
B.2.1. EMBEDDINGS OF HINT LABEL AND TIME STEP
The hint label for semantic segmentation generated by adding categorical noise the ground truth labels. Given the hint y, it is first passed through an embedding layer that maps each class {0, 1,.,,, C) (where C is the number of classes) to gigee--dimensional vector y. y is then processed through linear layers.
Time step t is first embedded through the sinusoidal position embedding and then processed through linear layers and activation function as below, where d is the embedding dimension, T' is a large number (usually selected as 10000).
cos 1
= concat sin
cos sin
cos T T T ..... sin
(14)
T
T
T
B.2.2. DOWN-SAMPLE
To perform down-sampling of the hint label ý using the time step embedding in the provided code, we first process them through two residual blocks below. Then the hint label is then down- down-sampled using convolution with stride of 2.
Conv2D Conv2D (iLU(()())) Linear (SILU(())
(15)
where G-(-)) denotes group normalization, SiLU(-)is is the sigmoid-weightt linear unit activation function, Conv2D(.) represents a 2D convolution operation, and Lincar(-) is fully connected layer that transforms the time step embedding.
B.3. Experimental Details for Next-token Prediction
B.3.1. MODEL DESIGN
In our approach, we adopt the LLaMA2-7B model as the backbone, preserving the structure of the embedding and decoder layers. However, we modify the prediction mechanism for the next token using the hidden states. In a conventional decoder-only language model (LLM), the prediction of the next token is achieved by leveraging the preceding context, encoded in a high-dimensional hidden state h. This hidden state is then passed through a linear layer, typically referred to as the language modeling head (Im head) or unembedding layer, to yield the probability distribution Pyy) over the next token.
14

---

Predictive Consistency Learning
In contrast, our model introduces noise injection mechanism to perturb the token embeddings, aiming enhance robustness and generalization.
Training Phase. During the training phase, when the model obtains the last hidden states for each token, instead of directly passing them through the Im head to generate logits and compute the cross-entropy loss with the ground truth labels y, we transform these labels back into its corresponding embedding Yemb- Then add Gaussian noise ~ N (N00, σ²) to Yemb. resulting in a perturbed embedding:
naisy
Cryymb + Bec.
(16)
To inform the model of the noise magnitude, we also perturb the time step t, obtaining corresponding time embedding tmmb. The noisy token embedding y; is combined with emmb to form new noisy information vector,
poisy h
noisy + cmb
(17)
This noisy information is concatenated with the hidden state h, resulting in the augmented vector
aug h [h: h, poisy]
(18)
Finally, hang is passed through a multi-layer perceptron (MLP) and new classification head to generate the probability distribution for the next token:
ply h,"h) = sotmax(LM.HEADD
(19)
is worth noting that for each batch, addition to aligning p(y h,"")) with the next token ground truth by minimizing the cross-entropy loss, randomly generate time steps, /1 and 22, and obtain logits logits(y ha and logits(y h!
We then minimize the mean squared error (MSE) loss between them to ensure as much consistency possible.
Generation Phase Now we describe the generation process. After the input passes through the decoder layers and obtains the last hidden states, we encounter a challenge during the inference phase since the next token ý is unknown. To address this, we input a complete Gaussian noise vector h (i... c), which denoised by model to generate the next token y1- This process then iterated, with yı serving as ý for the subsequent iteration. Following the steps outlined Eq. 16, 17, 18, and 19, we generate y2. and on, iteratively.
The noise addition time step for each iteration is predetermined hyperparameter. In our experiments, employ linearly decreasing schedule for the time steps. For instance, with maximum noise step of 1000 and 5 iterations, the time steps are set as [1000, 800, 600, 400, 200]. ensuring a gradual reduction of noise over the course of iterations.
C. Supplementary Experimental Results
C.1. Supplementary Comparison to Label Smoothing
We conducted experiments on the N-body simulation task (3 isolated particles, 2 sticks, and hinge) using GCN as the backbone model. For label smoothing with continuous outputs, we implemented Gaussian noise injection via reparame- terization: ý y NO(0: 3I) where 3 controls the noise intensity and use ý to calculate loss. This optimization is verified to induce better representations [1]. Then, we further fine-tune the trained model exact labels to ensure precise regression. The Prediction eтror (x10- -) results of SL, PCL, and SL label smoothing with various 3 are shown in Table 5, Experimental results demonstrate that while label smoothing achieves moderate improvements (best B = 0.3, 3.256 error) over standard supervised learning (SL, 3,453), PCL (2.795 error) maintains a significant performance advantage over even the optimal label smoothing configuration.
15

---

Predictive Consistency Learning
Table 5. Prediction error (x10-3) of GCN model on N N-body simulation task with 3 isolated particles, 2 sticks and 1 hinges. The comparison includes SL, PCL, and SL with label smoothing with varying 3 values smoothing intensity.
Method
SL
PCL
B 0.01
B = 0.03
/ 0.05
0.1
/ = 0.3
B 0.5
1.0
Prediction Error 3.453
2.795
3.460
3.461
3.428
3.418
3.256
3.258
3.368
C.2. Supplementary Comparison to Ensemble Methods
Given that PCL requires multiple forward passes for inference, we supplement the comparison to ensemble methods, We conducted experiments on the N-body simulation task (3 isolated particles, 2 sticks, and hinge) using GCN as the backbone model. comparing PCL against a bagging ensemble approach where independent models were trained on the full dataset and their predictions averaged. The prediction error (x (x10-2) results under matched inference passes (1-5) are presented in Table 6.
Table 6. Prediction error (x (x10=3) of GCN model on N- N-body simulation task with 3 isolated particles, sticks and 1 hinges. The comparison includes PCL and SL with the bagging technique under varying numbers of inference model passes.
# Inference Passes
5
PCL
2.834
2,818
2.808
2.796
2.786
SL. Bagging
3.453
3.210
3.132
3.084
3.065
C.3. Unconstrained N-body Simulation
The N-body simulation task involves predicting the future positions of set of interacting partieles over time based initial conditions such as their positions, velocities, and the inherent physical forces governing their interactions (Satorras et al., 2021). The task's outputs retain the same dimensionality and complexity as the inputs. The evolution of particle positions and velocities follows fundamental physical laws such as gravitational or electrostatic interactions. We follow (Satorras et al., 2021) to solve the eecharged-partccle system in 3-dimensional space. The system consists of five particles, each with either positive or negative charge, and each particle has an associated position and velocity.
Dataset. We collected 3000 trajectories for training, 2000 for validation, and 2000 for testing. Each trajectory spans 1000 timesteps. For each trajectory, the initial conditions include the particle positions p(0) = initial velocities v(0) = {v1 (0). Us (0)} E R5x3 and the respective charges c fci...., cs} (-1, The task is to predict the positions of the five particles after 1000 timesteps. The model is optimized by minimizing the averaged Mean
..... (0)) E 5xx the
Squared Error (MSE) between the predicted positions and the ground truth positions.
Metries. We adopt two evaluation metrics evaluate the regression quality for test data: 1) Mean Square Error (MSE): the average of the squares of the errors between the predicted values and the true values; 2) Mean Absolute Error (MAE): the average of the absolute differences.
Model Design. We consider the state-of-the-an graph modeling solution for this task, where we input the concatenation of the initial positions and the velocities as the node features. The charges are input as edge attributes aij c.cy. We take the
Table 7. Comparison of traditional supervised learning and PCI. for MSE.
Method
MSEL
MAEL
GCN (Kipf& & Welling, 2016)
1.006440000014
0.04322+00000822
GCN-PCL (Ours)
4.009770.00022
0.00081
GAT (Veličković et al., 2017)
0.00949=0.00040
0.03996:10.001998
GAT-PCL (Ours)
0.00910=0.00038
0.03726=00.00068
GGNN (Li et al., 2015)
0.01220=0.00020
0.0461440.00146
GGNN-PCI. (Ours)
0.01143 0.00042
0.04336+0.00127
16

---

Predictive Consistency Learning
0.018
GCN
GCN-PCL
0.016
GAT GGNN
GAT-PCL
GGNN-PCL
MSE
0.014
0.012
0.010
2000
4000
6000
Epochs
Figure 6. MSE curves on test data.
Table 8, Ablation study on loss construction.
Method
MSEL
MAE
Traditional SL
0.01064
0.04322
w/o Yr -teerm, w/ Xe-merm 4.34559
1.62437
w/ A--temm, w/o X2-term
0.00956
0.03895
w/ -temm, w/ Az-term
0.00927
0.03783
model outputs the estimated positions. To introduce PCL, we adopt two linear layers encode the input attributes and the noised label, respectively, and then concatenate them to form the input hidden feature to the subsequent graph neural layers. We adopt 4 graph neural layers, and for each layer's output, we integrate the timestep embedding extracted by the sinusoidal position embedding and a linear layer through addition. In this task, E R5x3 and we adopt the Gaussian noising process to produce noised labels as shown Eq.
Results. We compare the model with the classic graph neural networks, including Graph Convolutional Networks (GCN) (Kipf & Welling, 2016), Graph Attention Network (GAT) (Veličković et al., 2017), and Gated Graph Neural Networks (GGNN) (Li et al., 2015). For each model, we compare the performance with the models trained by the classic SL and our proposed PCL. Table 7 shows the superiority of PCL on quantitative results with lower estimation errors on both MSE and MAE under same settings, and Fig. 6 shows performance gain on the test MSE curves within the training process. Table 8 provides ablation studies on the effects the Attterm and Ae-term and verifies the effectiveness of every loss term Eq. 8.
C.4. Supplementary Results for Semantic Segmentation
For semantic segmentation, to vividly illustrate the effects of PCL, we present a comparison of the Intersection over Union (IoU) metries for all three models across the ADE20K dataset's 150 categories in Table The results indicate that MobileNetV2ddilan ResNet500hiltted, and HRNetV2 have achieved advantages IoU 70.00%. 58.67%, and 64.67% of the categories respectively after using PCL.
Fig. 7 illustrates the differences in predicting semantic segmentation maps among various models and training methods, visually reflecting the performance differences between the models as outlined in Table 3.
17

---

Predictive Consistency Learning
Table 9. Comparison of Intersection over Union (IoU) for classic supervised learning (SL.) versus the proposed predictive consistency learning (PCL) in semantic segmentation across various neural backbones. Bold indicates better performance in that category. MoibleNet: MobileNetV2dilatee ResNe150: ResNet50dilated. Order: Ranked from top to bottom based on the probability of each category in the
ADE20K dataset
MobileNet
ResNet50
MobileNet
ResNer50
MobileNet
ResNer50
Object
Object
Object
SL
PCL
SL
PCL
SL
PCL
SL
PCL
SL
PCL
SL
PCL
wall
69.01
67.29
73.21
77.57
building
75.72
77.00
80.89
85.21
building
92.78
93.14
93.50
93.36
floor
71.95
71.85
77.43
79.61
tree
68.86
69.99
70.12
72.07
tree
77.41
78.59
80.22
84.16
road
75.96
75,48
80.24
80.98
bed
80.13
80.04
84.53
86.09
bed
53.45
53.41
57.96
64.24
grass
64.91
67.28
66.54
75.82
cabinet
49.12
50.57
56.45
59.47
cabinet
54.10
54.77
60.93
63.53
person
70.96
70.75
75.13
72.22
earth
30.30
31.84
32.37
44.69
earth
32.27
32.86
39.90
52.50
table
45.08
46.04
52.93
57.66
mount
48.37
52.20
54.45
$5.61
mount
45.41
47.46
45.40
55.84
curtain
61.68
63.74
65.41
71.90
chair
45.15
46.72
53.01
55.42
chair
76.57
75.96
81.05
79.83
water
49.54
51.51
52.56
60.90
picture
63.22
64.17
66.01
70.79
picture
55,32
56.09
59.25
67.82
shelf
34,45
35.85
40.74
48.58
house
26.73
37.92
54.76
59.03
house
$1.19
55.54
54.69
17.62
mirror
42.59
43.43
54.74
59.80
nug
40.69
44.73
53,59
61.40
rug
20.00
23.24
24.74
46.71
armchair
33.24
33,23
38.42
49.39
seat
37.37
42.45
52.93
50,21
seat
27.98
29.42
31.38
48.18
desk
28.57
29.67
47.64
41.48
rock
31.21
33,04
35.79
50.45
rock
35
38.37
39.80
45.83
lamp
49.66
49.48
58.79
63.01
bath
60.63
64.66
69.45
70.42
bath
27.08
29.41
32.58
37.84
cushion
38.03
39.14
47.27
43.30
base
10.83
13,20
26.47
35.80
base
13.79
15.00
15.39
20.79
column
32.52
33,72
40.73
49.83
signboard
26.02
26.75
30.75
38.02
siguboard
34.45
36.27
37.04
54.18
counter
17.63
21.31
24.55
51.29
sund
17.37
22.90
35,48
45.76
sand
50,58
50.17
65.30
59.64
skyscraper
41.18
45.66
53.45
67.52
hearth
63.57
63.18
62,90
70.00
hearth
51.17
52,91
68.53
70.14
grandstund
37.60
47.05
39.08
33.67
path
15.70
14.58
19.93
23.39
path
26.81
28.95
18.78
44.63
runway
49.63
50.55
57.36
60.86
case
34.37
39.37
44.64
54.33
case
88.14
88,33
89.86
85.98
pillow
42.76
41.31
46.58
37.59
screen door
45.60
48.47
58, 18
63.30
screen door
28.34
29.06
19.93
48.86
river
12.43
12.92
11.70
27.58
bridge
32.22
36.03
55.18
37.89
bridge
29.15
31.21
37.23
39.30
blind
13.24
26.42
39.79
54.50
coffee table
48.14
43.74
57.95
63.86
coffee tahle
73,80
74.33
84,20
77.64
flower
26.52
30.22
33.88
42.41
book
36.39
39.23
41.95
44.01
book
5.31
5,82
5.35
27.00
bench
35.06
36.14
38,70
42.32
countertop
35.03
38.29
49.59
48.98
countertop
55.27
55.90
76.17
65.01
palm
36.59
38.48
47.14
45.50
kitchen island
25.78
27.30
36.10
23.53
kitchen island
47.56
48.16
53.04
46.73
swivel chair
35.47
33,70
47.66
41.91
hoat
26.55
29.40
53,28
59.06
boat
21.39
29.08
23.8% 23.88
47.72
arcade machine
22.63
32.14
47.78
47.26
hovel
25.65
26.17
12:55
15.46
hovel
57.20
62.79
76.84
70.25
towel
31.85
34.53
53.05
52.35
light
32.29
34,86
43.61
45.28
light
3.52
5.69
23.40
24.88
tower
26.91
25.90
38.69
51.83
chandelier
57.94
59.60
59.92
62.95
chandelier
8.82
5.80
21.74
42.11
strectlight
7.74
7.19
22.78
22.20
booth
26.28
32.90
41.30
$7.26
booth
45.35
49.25
63.16
68.86
airplane
39.72
39.50
50.81
54.55
dirt uack
11.28
10.38
6.97
22.89
dirt track
24.60
23.67
28.22
39.82
pole
12.51
12.20
13.94
16.40
land
0.94
0.69
0.46
18.23
land
1.44
2.17
9.50
24.90
escalator
5.27
7,88
21.89
42.77
ottoman
28.07
26.99
38.95
38.36
ottoman
8.29
13.26
34.92
10.97
huffet
33.48
33.36
37.03
49.54
poster
10.05
7.30
20.61
37.01
poster
9.03
9.04
5.45
24.44
van
25.37
23.07
41.20
45.06
ship
1.72
26.34
24.27
83.96
ship
7.06
15.23
16.68
71.20
conveyor
46.77
46.93
38.53
44.23
canopy
9.79
14.49
15.35
29.41
canopy
53.79
59.10
59.56
76.23
plaything
7.95
13.39
20.22
16.25
natatorium
22.26
25.75
24.61
21.81
natatorium
26.78
27.75
37.90
36.75
barrel
6.54
6.47
23.75
22.49
hasket
15.64
16.77
27.50
27.00
basket
35.83
38.82
49.53
35.14
tent
76.36
82.72
77.62
84.89
bag
1.93
4.91
9.18
16.85
bag
35,53
35.81
49,52
40.93
cradle
66.27
64.05
74.45
65.07
oven
33.26
30.10
35.46
44.45
oven
29.42
30.76
30.93
32.86
food
33.55
45.09
43.66
40.65
step
1.58
3.30
0.14
17.22
step
12.07
27.30
22.64
44.84
brand
18.18
20.85
20.95
32.78
microwave
33.01
32.82
34.16
27.20
microwave
26.52
28.90
39.99
51.19
animal
42.52
43.54
54.43
56.01
bicycle
31.74
31.13
47.76
39.18
bicycle
3.41
12.23
37.02
1.47
dishwasher
44.65
45.73
66.85
44.28
screen
56.52
57.84
61.08 0..08
68.69
0.30
1.74
5.77
29.02
sculpture
2.79
9.56
18.52
29.25
hood
31.01
35.97
49.64
34.65
hood
22.72
21.80
38,07
36.85
vase
18.52
16.69
36.89
32.72
stoplight
5.44
1.98
26.40
18.00
stoplight
0.83
0.93
3.42
6.23
ashean
20.05
25.89
39.94
35.85
fan
39.64
33.16
52.23
$6.63
fan
23.69
28.73
31.10
17.93
crt screen
0.75
1.15
0.70
17.78
plate
33.26
31.50
42.14
41.12
plute
5.91
21.66
1.96
58.89
notice board
18.69
23.73
37.39
23.26
shower
0.00
0.01
0.00
0.20
shower
26.37
28.49
47.29
60.18
glass
2.08
3.87
10.87
8.51
clock
4.92
5.91
25.50
21.65
clock
9.82
10.00
21.03
41.11
18

---

Predictive Consistency Learning
Test Image & Ground Truth
ResNetsodilated & PPM
HRNetV2
Classic SI
PCT (Ours)
Claosicc SL
PCT. (Ouns)
Classic SL
PCL (Our)
Figure 7. Comparison of classic supervised leamingg and PCL on semantic segmentation across different neural backbones.
19


Reviews:
[{"review_id": "rRRUDcNI40", "reviewer": "Reviewer_Fty8", "summary": "The paper introduces Predictive Consistency Learning (PCL), a paradigm for tasks with complex labels. The paper starts with arguing that traditional supervised learning struggles with high-dimensional or structured labels due to the difficulty of mapping compressed input features directly to rich label spaces. Inspired from image diffusion, PCL addresses this by decomposing label learning into a progressive process: it introduces noise-perturbed labels as additional inputs during training and enforces consistency across predictions at different noise levels. Experiments on vision (semantic segmentation), graph (N-body simulation), and text (LLM fine-tuning) tasks demonstrate PCL's effectiveness over standard supervised learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments, nice suggestions, and for acknowledging our work. Below we respond to your major comments.\n\n> **Q1: Qualitative comparison to methods that project the output space to a simpler latent space.**\n\nIntroducing an encoder-decoder structure with a  $Y\\to Y_E$ encoder and a $Y_E\\to Y$ decoder may help handle the complexity of $Y$. This approach is commonly used in generative models like Latent Diffusion Models (LDMs), where data (e.g., images or videos) are first compressed into a latent space for generation before being decoded back into the original domain. However, for predictive mapping from $X$ to $Y$, which is a fundamental component in ML, explicitly introducing an encoder-decoder stage adds computational burden. During inference, the encoder (mapping $Y \\to Y_E$) is redundant, yet its inclusion increases training complexity. Additionally, the two-stage learning process (first $X \\to Y_E$, then $Y_E \\to Y$) can lead to error accumulation, degrading overall performance. \n\nIn contrast, our method retains the simplicity of traditional supervised learning while enhancing performance through progressive label decomposition. This approach allows for more effective learning of label information with minimal yet refined modifications to the standard training framework.\n\n\n> **Q2: How does PCL differ from curriculum learning (CL) strategies that gradually increase label complexity?**\n\nThe key differences between PCL and CL can be summarized as follows:\n\n1. **Learning Target Consistency**. CL predefines staged learning targets (e.g., coarse-to-fine labels), where each stage learns an approximation of the true label. This risks error accumulation, i.e., biased features from early stages may propagate to later stages. In contrast, the objective of PCL remains stable: always predicting the true label, with complexity dynamically adjusted by the input noise level.\n2. **Progressive vs. Simultaneous Learning**. CL follows a fixed, sequential progression (e.g., easy→hard labels). PCL, however, randomly samples time steps during training, enabling the model to learn various label information levels (from partial to complete) simultaneously, where the time step t acts as a controller for label granularity.\n3. **Controllable Prediction**. PCL’s noise-conditioned framework allows explicit control over prediction granularity via time step t. As evidenced in Fig. 5, setting a larger t tends to encourage the model to improve broader structural relationships, while setting a smaller t encourages the model to focus on finer details. Further, PCL supports multi-step inference (Sec. 4.4), where predictions are iteratively refined (analogous to diffusion denoising), boosting final accuracy.\n4. **Leveraging Partial Labels as Input**. One of the motivations of PCL is to treat labels not just as targets but as learning facilitators. By feeding partially noised labels (e.g., \"hints\") during training, the model learns to exploit intermediate information (similar to how humans use reference solutions to help solve problems).\n\n\n> **Q3: Is there a train / inference time difference in the SL vs PCL training / inference?**\n\nFor training, since the loss calculation requires two inference predictions with different noise levels during training, it requires twice the training cost of the traditional supervised learning paradigm. Moreover, it typically takes less than twice the iterations for the model to converge.\n\nFor inference, PCL can produce much superior predictions with merely a single forward pass as SL does. Meanwhile, PCL can achieve better performance with more forward passes. As shown in Fig. 3 using a GCN backbone on the 3,2,1 dataset, even single-step PCL inference achieves superior performance (prediction error 0.02835) compared to SL (0.03478), while additional inference steps (e.g., 5-step) can further refine results (0.02795).\n\n> **Q4: Experiments are sound but can be made more thorough with comparison against advanced baselines such as diffusion models for structured labels.**\n\nThanks for the acknowledgment and suggestion. Our method primarily takes traditional supervised learning as our main framework, where PCL is constructed within. The generative modeling approach you mentioned essentially reformulates the prediction task as a generation task, representing one specific implementation under the broader supervised learning framework ($X\\to Y$ mapping). PCL operates at a different conceptual level, whose methodology for label utilization could potentially be integrated into diffusion-based prediction as well. In a word, our current framework is a bit orthogonal to and can be combined with the diffusion models.\n\nIn our revision, we will discuss relevant works in this direction and explore subsequent attempts in future work.\n\n---\n\nWe hope this response could help address your concerns, and we are more than happy to address any further concerns you may have.", "claims_and_evidence": "The claims are supported by empirical evidence across diverse tasks.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experiments are sound but can be made more through with comparison against advanced baselines such as diffusion models for structured labels (for e.g. SegDiff [1] for segmentation).\n\n---\n\n[1] SegDiff: Image Segmentation with Diffusion Probabilistic Models, https://arxiv.org/pdf/2112.00390", "relation_to_broader_scientific_literature": "PCL builds on consistency models and diffusion-based training but adapts them to deterministic label prediction. It connects to curriculum learning (progressive label complexity) and residual learning (noise as input).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths\n- Paper is well presented with clear details\n- Experiments encompass multiple modality benchmarks\n\n### Weakness\n- The thesis of the paper is that for complex output spaces standard supervised learning is not appropriate, it also acknowledges that there are methods which can project the output space to a simpler latent space but provides no qualitative comparison against them.", "other_comments_or_suggestions": "I think [extreme classification](http://manikvarma.org/downloads/XC/XMLRepository.html) can be a good benchmark for this work (as it inherently has a high dimensional predictive output space)", "questions_for_authors": "1. In what *predictive* cases is learning an invertible function from $Y_E \\rightarrow Y$ not straightforward (hence necessating the need of the proposed approach)? dual encoder style modeling (which has a very simple inverse map) should be possible for most predictive tasks, right?\n2. How does PCL differ from curriculum learning strategies that gradually increase label complexity?\n3. Is there a train / inference time difference in the SL vs PCL training / inference?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tlo73KYZsV", "reviewer": "Reviewer_FjUw", "summary": "This paper introduces a new prediction paradigm for more complex label spaces than the ones that are traditionally assumed in supervised learning. The paper draws inspiration from the \"generative consistency learning\" learning paradigm to produce the predictive consistency learning (PCL) paradigm, which the authors suggest is more suited to complex label spaces. The technique involves predicting the label with progressive amounts of label noise so that models can make predictions even when the ground truth labels contain varying amounts of noise, and enables multi-step denoising techniques within the PCL framework. Experimentally, the authors demonstrate PCL compared to supervised learning in a variety of settings: N-body simulation, semantic segmentation, and language modeling.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments, nice suggestions, and for acknowledging our work. Below we respond to your specific comments.\n\n> **Q1: How this work relates to the field of structured prediction.**\n\nThanks for your insightful suggestion. Indeed, our work also handles the challenge of structured prediction, i.e., learning to predict complex, structured outputs. However, we observe that with the widespread adoption of deep neural networks, many structured prediction problems (like sequence or graph prediction) can now be effectively handled through standard feed-forward architectures trained end-to-end, often being naturally subsumed under the broader umbrella of supervised learning. While classical structured prediction methods may focus on specific output structures (e.g., trees or sequences), our approach makes no assumptions about certain structures, while being particularly effective for handling complex labels. This is why we emphasize label complexity within the supervised learning paradigm rather than through the lens of structured prediction methods. We sincerely appreciate your suggestion and will expand our discussion of related work in this field during revision.\n\n\n> **Q2: Does the noise distribution have to be similar to the natural label noise distribution?**\n\nThe noise distribution does not necessarily need to match the natural label noise distribution, as our primary goal in Sec. 4.2 is to design a noise process that systematically degrades label information in a controlled manner. The key is to tailor the noise to the structure of the label space (whether continuous or discrete) rather than replicating real-world noise patterns. For continuous labels, we employ Gaussian noise to gradually corrupt the signal, while for discrete labels, we use categorical noise that diffuses the probability mass of one-hot vectors across classes. In cases where the discrete label space is excessively large or complex, we shift to perturbing the latent representations directly with continuous noise. These approaches comprehensively cover the spectrum of label types (continuous or discrete), which is also why we choose these tasks in experiments, which can exemplify different scenarios.\n\n> **Q3: What noise distribution was used for the N-body simulation problem or the language modeling task?**\n\nFor N-body simulation, the label $\\mathbf{y}\\in \\mathbb{R}^{p\\times 3}$ where $p$ denotes the number of particles. Since the labels maintain continuous values, we adopt Gaussion noise for the noising process: $q(\\mathbf{y}_t | \\mathbf{y}) = \\mathcal{N}(\\mathbf{y}_t; \\sqrt{\\bar{\\alpha}_t} \\mathbf{y}, (1-\\bar{\\alpha}_t)\\mathbf{I})$ where $\\bar{\\alpha}_t$ controls the noise level.\n\nFor language modeling, since the raw token space is extremely large, we directly add noise to the latent embeddings of the labels, which are multi-dimensional continuous vectors. we adopt Gaussion noise for the noising process: $q(\\mathbf{h}\\_{\\mathbf{y}\\_t} | \\mathbf{h}\\_{\\mathbf{y}}) = \\mathcal{N}(\\mathbf{h}_{\\mathbf{y}_t}; \\sqrt{\\bar{\\alpha}\\_t} \\mathbf{h}\\_{\\mathbf{y}}, (1-\\bar{\\alpha}\\_t)\\mathbf{I})$.\n\nThe details can be found in Sec. 4.2.\n\n> **Q4: It's not immediately clear how \"next-token prediction\" is a complex label space.**\n\nNext-token prediction, while seemingly maintaining simple labels, actually operates in a complex label space due to the vast vocabulary size (often tens or hundreds of thousands of tokens) and the semantic information encoded in each prediction. Though it only predicts one token at a time, the process implicitly models long-range dependencies and coherent reasoning, as each step conditions on the full context to generate meaningful continuations. The high dimensionality of the token embedding space (e.g., 4096-dimensional vectors), where we apply noise, also underscores its complexity. We chose this task for its practical relevance in modern LLMs and its ability to demonstrate our method’s scalability to large label spaces. Sequence-to-sequence tasks are indeed another compelling setting, and we appreciate the suggestion and will explore such extensions as a natural direction for future work.\n\n> **Q5: Are there other examples of label learning tasks beyond supervised learning and this work?**\n\nThe reason we haven’t strictly confined PCL to supervised learning is that the mapping from X to Y is fundamental across many learning paradigms, including semi-supervised, weakly supervised, and even self-supervised settings. While our current experiments focus on standard supervised learning for clarity and validation, PCL’s core mechanism (refining the X-to-Y mapping) is intentionally designed to be adaptable. We believe this flexibility opens doors for future extensions into broader learning frameworks, and we’re excited to see explorations in these directions in subsequent works.\n\n---\n\nWe hope this response could help address your concerns, and we are more than happy to address any further concerns you may have.", "claims_and_evidence": "The main claim of the paper is that the proposed PCL is better suited to complex label spaces compared to supervised learning. The experimental evidence for this suggests that this is true -- in all of the settings that the authors evaluated, N-body simulation, semantic segmentation, and language modeling, PCL indeed outperforms supervised learning, and these settings all correspond to what I would think of as having more complex label spaces.", "methods_and_evaluation_criteria": "The evaluation protocol makes sense to me. N-body simulations, semantic segmentation, and language modeling are all tasks which involve complex label spaces that are typically modeled using supervised learning. The proposed method is described clearly, although in the abstract, it was initially unclear what generative consistency models refer to -- more exposition in the abstract would help for readers who are unfamiliar with this area.", "theoretical_claims": "I did not carefully verify the correctness of any of the math introduced in the paper, but it seemed to make sense from my initial read.", "experimental_designs_or_analyses": "The experimental design seemed sound from my read-through. The N-body simulation is a graph learning problem, so the authors use appropriate supervised graph learning baselines. Similarly, the baselines were chosen appropriately for the semantic segmentation and language modeling tasks.", "relation_to_broader_scientific_literature": "The work is motivated by the need for learning paradigms targeting modern complex label spaces -- the current literature often comprises techniques that are largely based on traditional supervised learning, where complex or structured label spaces are perhaps handled in an ad hoc way.", "essential_references_not_discussed": "From what I can tell, the essential references were mostly covered. If anything, the related work seems to be missing a discussion of how this work relates to the field of structured prediction, as such label spaces seem to be a key motivating component of this work.", "other_strengths_and_weaknesses": "**Strengths**\n- The proposed PCL is a novel and interesting take on the problem of learning labels from inputs, and tackles the relevant challenge of learning from more complex and structured labels. \n- The proposed framework is described in a clear and easy-to-follow way. \n- The experimental setup seems to appropriately cover complex label spaces including graphs, pixel-level segmentation, and language modeling. \n\n**Weaknesses**\n- I might have misunderstood, but it was unclear to me how the choice of noise distribution would impact results. Does the noise distribution have to be similar to the natural label noise distribution? \n- Again I might have missed this, but I can't tell what noise distribution was used for the N-body simulation problem or the language modeling task. \n- It's not immediately clear how \"next-token prediction\" is a complex label space -- if the label space is defined as predicting a single token from the vocabulary, then it seems to simply be a large label space. On the other hand, I would think that that the task of predicting entire sequences (all at once) would be a more obvious choice for a more complex structured label space, perhaps applied to a sequence to sequence learning problem.", "other_comments_or_suggestions": "On my first read, I found the abstract to be a somewhat confusing read. \"Directly predicting labels from data inputs has been a long-standing default mechanism for label learning tasks, e.g., supervised learning\" seems like an unnecessarily roundabout way to describe supervised learning -- are there other examples of label learning tasks beyond supervised learning and this work? This statement seems to imply this. Later on in the abstract, the authors mention the \"generative consistency training concept in generative consistency models,\" which I was unfamiliar with -- this part should have somewhat more exposition.", "questions_for_authors": "As mentioned before, my main question is about the noise distributions used in the experiments and how the choice of noise distribution impacts results. Furthermore, for complex label spaces, it might be unclear or challenging to design a realistic noise distribution -- how might such cases be handled?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xu13u3wRWF", "reviewer": "Reviewer_bsic", "summary": "This paper aims to leverage more rich label information to enhance supervised learning. To achieve this goal, it proposes a new learning method called PCL (predictive consistency learning). Different from those conventional supervised learning approaches, PCL learns from both the labels as well as the noise-perturbed label information. Besides, PCL also optimizes the consistency across different noise levels to ensure the learning quality. Experiments on multiple tasks demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable comments and nice suggestions. However, we believe there may exist some misunderstandings regarding the methodology. We regret any confusion caused by our presentation and would like to clarify the core methodology of PCL.\n\nWhat PCL does is to leverage the concept of consistency modeling to enhance the traditional predictive paradigms (e.g. in supervised learning), which modifies the mapping manner in training from `data input x -> full label y` to  `data input x, noised label y_t (as additional hints) -> full label y`, and then enforce mapping consistency across different noise levels (controlled by t) to regulate label information content during training. The \"label modeling\" in the title refers to systematically modulating label information density (which is achieved by gradually adding noise to the raw labels and providing noised labels as input to make the model learn the complementary label information) and PCL does not employ an auxiliary model to predict or recover noise but rather introduces a novel training paradigm.\n\n> **Q1: There seems to be an overclaim regarding the label modeling. It is improper to claim that the label information is more complex than the inputs.**\n\nTo clarify, we have not claimed that label information is universally more complex than inputs in the paper. Rather, we highlight (e.g., in L30 right) that \"recent advanced scenarios involve much more complex labels\" and \"these challenges expose predictive bottlenecks due to the inherent complexity of transformations from features to labels, in addition to feature extraction\". The label complexity is for comparison to simple categorical annotation tasks, rather than compared to input complexity.\n\nIn many modern tasks, particularly those beyond traditional classification, labels and inputs can share closer dimensionality and information density, representing a significant departure from simple categorical annotation tasks. For instance, in LLM fine-tuning, both questions (inputs) and answers (labels) are natural language sequences requiring similar levels of semantic understanding and generation capability.\n\n\n> **Q2: The computational cost. PCL learns to model the noise using an additional model.**\n\nPCL is within the supervised learning paradigm and **does not employ an auxiliary model** to predict or recover noise. From the model perspective, it directly predicts the labels just as SL does. The mere difference lies in that it receives additional label hints as input. Thus, the additional computational overhead is minor in a forward pass.\n\nDuring training, since the loss calculation requires two inference predictions with different noise levels, it requires twice the training cost of the traditional supervised learning paradigm. Moreover, it typically takes less than twice the iteration for the model to converge. For inference, PCL can produce much superior predictions with merely a single forward pass as SL does. More forward passes can lead to further performance gains. As shown in Fig. 3 using a GCN backbone on the 3,2,1 dataset, even single-step PCL inference achieves superior performance (prediction error 0.02835) compared to SL (0.03478), while additional inference steps (e.g., 5-step) can further refine results (0.02795).\n\n> **Q3: The ablation studies are only conducted on the N-body simulation task. No ablation studies for other tasks.**\n\nThe purpose of our experiments across three modalities is to validate the broad applicability of our method. We chose to conduct a more detailed analysis on one task to ensure the depth of our analysis.\n\nHere we supplement the ablation study of $\\lambda_2$ term on the semantic segmentation task in [table](https://anonymous.4open.science/r/PCL-0F54/lambda2-segmentation.png) and we find excluding excluding the $\\lambda_1$ term causes the model to fail.\n\n> **Q4: Hyperparameter settings and sensitivity analysis.**\n\nThe key model-specific hyperparameters are set as $\\alpha=0.2$ and $\\lambda_1=\\lambda_2=1$ in our experiments, with $\\lambda$ values chosen for equal weighting rather than through extensive tuning. In response to your valuable suggestion, we conduct sensitivity analyses across reasonable parameter variations. All other parameters follow standard SL configurations as detailed in the paper, with all source code to be fully disclosed upon acceptance. \n\nThe experimental results for hyperparameters:\n\n$\\alpha$: [Table](https://anonymous.4open.science/r/PCL-0F54/alpha.png)\n\n$\\lambda_1$: [Table](https://anonymous.4open.science/r/PCL-0F54/lambda1.png)\n\n$\\lambda_2$: [Table](https://anonymous.4open.science/r/PCL-0F54/lambda2.png).\n\n> **Q5: Source code.**\n\nThe open-source release of PCL is crucial for its impact on the research community, and we commit to open-sourcing our code upon acceptance of the final manuscript.\n\n---\nWe hope this response could help address your concerns, and we are more than happy to address any further concerns you may have.", "claims_and_evidence": "Weaknesses:\n- There seems to be an overclaim regarding the label modeling. In supervised learning, the label tends to correspond to the annotations from crowd sourcing. The recovered noise from the label is also related to the recovering model, which is different from the label itself. Therefore, it is improper to claim that the label information is more complex than the inputs.", "methods_and_evaluation_criteria": "Strengths:\n- The proposed method has been adequately evaluated on multiple tasks. The authors have also provided thorough explanations for the experimental settings and criteria.\n\nWeaknesses:\n- The computational cost of the proposed method should be quite expensive, as it learns to model the noise using an additional model. Therefore, it is unfair to directly compare PCL with supervised learning. Moreover, it is necessary to analyze the complexity of the proposed method.", "theoretical_claims": "Weaknesses:\n- There is no theoretical evidence for supporting the proposed PCL.", "experimental_designs_or_analyses": "Strengths:\n- The experiments on multiple datasets demonstrate the effectiveness of the proposed method, as the performance gain is satisfactory.\n\nWeaknesses:\n- The ablation studies are only conducted on the constrained N-body simulation task. There is no ablation studies for other tasks (semantic segmentation and supervised next-token prediction fine-tuning).\n- It seems that the authors have not reported the hyperparameter settings. Besides, there is no hyperparameter sensitivity analysis.", "relation_to_broader_scientific_literature": "Strengths:\n- This paper is related to supervised learning and generative models such as Diffusion. The authors have properly discussed them.", "essential_references_not_discussed": "Weaknesses:\n- It seems that the references are a bit out-of-date. Most references are from or earlier than 2023. Is there any more recent related work?", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iD3T9E8jzI", "reviewer": "Reviewer_yH7y", "summary": "Traditional methods predict labels directly from data inputs, but recent tasks often involve more complex labels. To address this, the authors propose Predictive Consistency Learning (PCL), inspired by consistency training in generative models. Unlike traditional approaches, PCL incorporates noise-perturbed labels as additional references, ensuring consistency across varying noise levels. It learns the relationship between latent features and a range of label information, enabling progressive learning and multi-step inference akin to gradual denoising, thus improving prediction quality. Experiments on vision, text, and graph tasks demonstrate the superiority of PCL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments, nice suggestions, and for acknowledging our work. Below we respond to your specific comments.\n\n> **Q1: The proposed multi-step prediction approach brings up concerns regarding its efficiency. How much longer does the training process take compared to standard supervised learning? Additionally, what about the inference time?**\n\nFor training, since the loss calculation requires two inference predictions with different noise levels during training, it requires twice the training cost of the traditional supervised learning paradigm. Moreover, it typically takes less than twice the iteration for the model to converge.\n\nFor inference, PCL can produce much superior predictions with merely a single forward pass as SL does. Meanwhile, PCL can achieve better performance with more forward passes. As shown in Fig. 3 using a GCN backbone on the 3,2,1 dataset, even single-step PCL inference achieves superior performance (prediction error 0.02835) compared to SL (0.03478), while additional inference steps (e.g., 5-step) can further refine results (0.02795).\n\n> **Q2: Label smoothing involves applying some form of “perturbation” to the labels to help the model generalize better. How much of a performance advantage does PCL offer over label smoothing? The authors may report the results on a small dataset.**\n\nThanks for this valuable point. To evaluate this, we conducted experiments on the N-body simulation task (3 isolated particles, 2 sticks, and 1 hinge) using GCN as the backbone model.\n\nFor label smoothing with continuous outputs, we implemented Gaussian noise injection via reparameterization: $\\hat{\\mathbf{y}} = \\mathbf{y} + \\mathcal{N}(0;\\beta\\mathbf{I})$ where $\\beta$ controls the noise intensity and we use $\\hat{\\mathbf{y}}$ to calculate loss. This optimization are verified to induce better representations [1]. Then, we further finetune the trained model on exact labels to ensure precise regression. The Prediction error ($\\times 10^{-2}$) results of SL, PCL, and SL_label_smoothing with various 𝛽 are as follows:\n\n\n\n| |SL| PCL| 𝛽=0.01 | 𝛽=0.03 | 𝛽=0.05 | 𝛽=0.1 | 𝛽=0.3 | 𝛽=0.5 | 𝛽=1.0 |\n| -| -| -| - | -| -| -| - | -| -|\n| Prediction Error | 3.453 | 2.795 | 3.460  | 3.461  | 3.428  | 3.418 | 3.256 | 3.258 | 3.368 |\n\nThe results show that while label smoothing yields certain improvements over standard supervised learning (with the best performance at β = 0.3), it still falls short of PCL by a significant margin. Specifically, PCL achieves a prediction error of 2.795, clearly outperforming both SL and the best-performing label smoothing setup (3.256). This highlights PCL’s stronger ability to generalize through its learned iterative refinement process.\n\n\n[1] How Does a Neural Network’s Architecture Impact Its Robustness to Noisy Labels? NeurIPS 2021.\n\n\n> **Q3: Given that it requires multiple forward passes for inference, how does its performance compare to that of ensemble methods?**\n\n\nThank you for the insightful question. To evaluate this comparison, we conduct experiments on the N-body simulation task (comprising 3 isolated particles, 2 sticks, and 1 hinge), using a GCN backbone. We compare PCL to a standard bagging ensemble approach, where $n$ independent models are trained on the full dataset. At inference time, predictions from the $n$ models are averaged to reduce individual model biases. The prediction error ($\\times 10^{-2}$) across different numbers of inference passes (i.e., number of independent models or PCL steps) is summarized below:\n\n\n| #inference passes | 1 | 2| 3| 4| 5|\n| -| -| -| -| -| -|\n| PCL| 2.834 | 2.818 | 2.808 | 2.796 | 2.786 |\n| SL (Bagging) | 3.453 | 3.210 | 3.132 | 3.084 | 3.065 |\n\n\nAs the results show, while bagging does provide performance improvements with more inference passes, it consistently underperforms compared to PCL. Notably, even with five inference passes, bagging does not reach the accuracy of PCL’s single-pass result (2.786 vs. 2.834), highlighting the efficiency and effectiveness of PCL’s learned iterative refinement.\n\nThe computational trade-offs further emphasize PCL's advantages. Bagging incurs significant overhead—requiring training, storing, and maintaining $n$ separate models, resulting in O(n) training and memory costs. In contrast, PCL achieves its gains through a single model, keeping both training and deployment lightweight while delivering superior predictive performance.\n\n\n---\n\nWe hope this response could help address your concerns, and we are more than happy to address any further concerns you may have.", "claims_and_evidence": "The authors have demonstrated the effectiveness of their method by testing it across a variety of settings, including n-body simulation, semantic segmentation, and next-token prediction. These experiments highlight the versatility of PCL.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally make sense.", "theoretical_claims": "This paper does not make theoretical claims.", "experimental_designs_or_analyses": "The proposed multi-step prediction approach brings up concerns regarding its efficiency. How much longer does the training process take compared to standard supervised learning? Additionally, what about the inference time? The authors may provide this information.", "relation_to_broader_scientific_literature": "Label smoothing involves applying some form of “perturbation” to the labels to help the model generalize better. How much of a performance advantage does PCL offer over label smoothing? The authors may report the results on a small dataset.\n\n[1] Rethinking the Inception Architecture for Computer Vision. CVPR 2016", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "Given that it requires multiple forward passes for inference, how does its performance compare to that of ensemble methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper draws inspiration from diffusion models and introduces a predictive consistency learning paradigm for tasks with complex labels. The prior approaches often suffer in cases with high-dimensional or structured labels. This paper introduces noise perturbation to labels as additional inputs and forces the training model to be consistent in its predictions at different noise levels. Experiments on a variety of tasks (with different modalities too) demonstrate the effectiveness of the proposed method. All reviewers agreed on the clarity of the paper and its technical contributions.
Decision: Accept (poster)