MixTraining: A Better Trade-Off Between Compute and Performance
Authors: Zexin Li
Jiancheng Zhang
Yufei Li
Yinglun Zhu
Cong Liu
Venue: Accepted by TMLR ()

Abstract: Integrating self-supervised learning (SSL) prior to supervised learning (SL) is a prevalent strategy for enhancing model performance, especially in scenarios with limited labeled data. Nonetheless, this approach inherently introduces a trade-off between computational efficiency and performance gains. Although SSL significantly improves representation learning, it necessitates an additional and often computationally expensive training phase, posing substantial overhead in resource-constrained environments.  To mitigate these limitations, we propose MixTraining, a novel training framework designed to interleave multiple epochs of SSL and SL within a unified $\textit{mixtraining phase}$. This phase enables a seamless transition between self-supervised and supervised objectives, facilitating enhanced synergy and improved overall accuracy. Additionally, MixTraining consolidates shared computational steps, thereby reducing redundant computations and lowering overall training latency. Comprehensive experimental evaluations demonstrate that MixTraining provides a superior trade-off between computational efficiency and model performance compared to conventional training pipelines. Specifically, on the TinyImageNet dataset using the ViT-Tiny model, MixTraining achieves an absolute accuracy improvement of 8.81% (a relative gain of 18.89%) while concurrently accelerating training by 1.29$\times$.

Published in Transactions on Machine Learning Research (04/2026)
Mix MixTninni::: A Better Trade-Off Between Compute and Performance
Zexin Li"
z1:536 Suerr edu
University of Colifornia, Riverside
Jiancheng Zhang
5hhan745 edu
University of California, Riverside
Yufei Li
yli9270ucr.ed edu
University of Colifornia, Riverside
Yinglun Zhu
yzhu@ucr.edd edu
University of California, Riverside
Cong Liu
congl@ucr.ed edu
University of Colifornia, Riverside
Reviewed on OpenReview: http:://pppmmmiiie..eeee//////ts
Abstract
Integrating self-supervised learning (SST.) prior to supervised learning (SL.) prevalent strategy for enhancing model performance, especially in scenarios with limited labeled data. Nonetheless, this approach inherently introduces trade-off between computational efficiency and performance gains. Although SSL significantly improves representation learning, it neces- sitates an additional and often computationally expensive training phase, posing substantial overhead in resource-constrained environments. To mitigate these limitations, we propose MixT MIATNNNGGG, a novel training framework designed to interleave multiple epochs of SSL and SL within a unified miztraining phase. This phase enables a seamless transition between self-supervised and supervised objectives, facilitating enhancod synergy and improved overall accuracy. Additionally, MIXTRAINING consolidates shared computational steps, thereby reducing redundant computations and lowering overall training latency. Comprehensive ex- perimental evaluations demonstrate that MIXTRAINING provides a superior trade-off between computational efficiency and model performance compared to conventional training pipelines. Specifically, on the TinylmageNet dataset using the ViT-Tiny model, MIXTRAINING achieves an absolute accuracy improvement of 8.81% (a relative gain of 18.89%) while concurrently accelerating training by 1.29x.
1 Introduction
Self-supervised learning (SSL) has emerged as a powerful paradigm for learning general representations from unlabeled data, significantly improving performance on downstream lasks (Brown et al., 2020; He et. al., 2022). The standard approach combines SSL with supervised learning (SL) into a two-phase pipeline: first training a model on unlabeled data to learn general representations, then adapting it to specific tasks using labeled data. This SSL+SL pipeline has become the de facto standard across computer vision (Chen al., 2020; He et al., 2020, 2022;
Wang et al., 2023b), natural language processing (Devlin et al., 2019; Radford
"Equal contribution
1 "Corresponding author

---

Published in Transactions on Machine Learning Research (04/2026)
Supervised Leaming
Self-supperiiee--SSppervieed Learning
MixTraining
Use 10% of Dataset
Use 50% of Dataset
Use 100% of Dataset
24
48
59
21
42
55
Accuracy (%)
18
51
36
30
12
43
9-
39
3
7
11
15
19
23
27
18
18
35
52
69
86
103
120
35
35
71
107
143 179 215 251
Latency (10*25)
Latency (10*26)
Latency (10
Figure 1: MIx TIXTRAINIG demonstrates significant. accuracy and computation gains over standard self- supervised learning and supervised learning pipeline across various data limitations levels (10%, 50%, and 100%). Experiments are conducted on the TinylmageNet dataset with the ViT-Tiny model; Each set of three points on the same line represents results obtained with different training durations (50, 75, and 100 epochs).
et al., 2019; Brown et al., 2020; Ouyang et al., 2022; Min et al., 2023), and speech recognition (Hsu et al., 2021; Chen et al., 2022), particularly excelling in data-scarce scenarios where labeled examples are limited.
However, this two-phase pipeline introduces a fundamental trade-off between computational efficiency and model performance. While the first phase of SSL improves downstream accuracy, requires substantial additional computation that be prohibitive in resource-constrained environments. Tn fact, the SSL phase often demands hundreds or thousands of GPU-days before any task-specific training begins (Kaplan et al., 2020; Dosovitskiy et al., 2020; Chung et al., 2024). This computational overhead creates a significant barrier for practitioners, particularly those who must train models from scratch due to data privacy requirements, domain-specific constraints, or the absence of suitable pretrained models.
The abrupt transition between SSL and SL phases in the standard pipeline also presents optimization challenges. The model must suddenly shift from optimizing self-supervised objectives (e.g., reconstruction loss) to supervised objectives (c.g., classification loss), potentially causing training instability and suboptimal adaptation (Peters et al., 2019; Mosbach et al., 2020; Kumar et al., 2022). This discontinuous transition may prevent the model from fully leveraging the synergies between self-supervised and supervised learning signals.
To address these limitations, we propose MIXTRAINING, a novel training framework that introduces a dedicated mixtraining phase between the SSL and SL phases. Our key insight is that the rigid separation between SST and ST. phases in conventional pipelines is neither necessary nor optimal. As illustrated Fig. 2, the mixtraining phase jointly optimizes both self-supervised and supervised objectives, creating a smooth transition that better preserves learned representations while adapting to downstream tasks. Our design also consolidate what would be separate forward and backward passes in the standard pipeline into unified operations (see Fig. 3 for details), substantially reducing training time while improving model performance.
As shown in Fig. 1, MIXTRAINING consistently achieves superior accuracy and lower training latency across varying data availability levels and training durations. For example, on the full TinyTmageNett dataset (Te & Yang, 2015) with reonstruccion--asee SSL (Hinton & Zemel, 1993; He ct al., 2022), MIXTRAINING delivers an 18.89% relative improvement in accuracy (8.81% absolute) and achieves 1.29x speedup over the standard SSL+SL pipeline. The advantage becomes more pronounced under severe data constraints: at 10% data availability, TRAINING improves accuracy by 105.58% (relative) and reduces training latency by 1.24x.
In this work, we focus on the widely used reconstruction-baseed self-supervised learning (SSL) paradigm (e-g., MAE) followed by supervised learning (SL) with classification losses. Within this scope, MIXTRAINING consistently provides a better tradc-off by smoothing the transition between reconstruction- basod SSL and SL while reducing redundant computation. We view extending MIXTNAINING to other SSL objectives (c.g., contrastive methods) as an important direction for future work.
2

---

Published in Transactions on Machine Learning Research (04/2026)
Contributions. We introduce the MIXTRAINING framework, which enhances both computational efficiency and model performance over the conventional SSL++TL pipeline. Focusing on reconstruction-basel SST and classification-basel SL, our core contributions are:
Superior accuracy through smooth objective transition. MIXTRAINING introduces a mixtraining phase that creates a gradual transition between self-supervised and supervised objectives, avoiding the abrupt shift that can destabilize training in standard pipelines. This smooth transition leads to consistently higher accuracy across all evaluated settings. On TinyImageNet with ViT-Tiny, this design achieves an 18.89% relative accuracy gain over the standard SSL+SL baseline.
Computational efficioncy via unified bowaad/backwaan passes. By merging SSL and SL epochs into a joint optimization phase, MIXTRAINING consolidates separate forward and backward passes over the backbone into single operations. Since backbone computations typically dominate training cost (the backbone is usually much larger than task-specific heads), this system-level optimization yields substantial speedups (e.g., 1.29x for ViT-T on TinylmageNet) while simultancously improving accuracy.
Robust generalization across diverse settings. MIXTRAINING demonstrates consistent. Pareto improvements over SSL+SL baselines across different experimental configurations, including multiple architectures (ViT, ResNet), datasets (CIFAR-10, CIFAR-100, TinyImageNet), data limitation levels (from 10% to 100%), and training epochs. The framework's modular design enables integration with existing SSL+SL pipelines with minimal modifications.
Scope and Applicability. It is important to clarify that MIXTRAINING does not aim to replace the existing pretraining-fe fineiiniinieuuiinggggg paradigm when high-quality pretrained models are readily available. Instead, our framework primarily targets scenarios where practitioners must train models from scratch, such as when data privacy constraints prevent the use of pretrained models, when domain-specific requirements render generic pretrained models inadequate, or when suitable pretrained models are unavailable. In these settings, MIXTraINING provides a more efficient alternative to the standard SSL+SL pipeline, achieving superior performance while reducing training lateney. In Section 5, we provide initial evidence that MIXTRAINING mid-training strategy.
can be effectively applied to pretrained models
Organization. The remainder of this paper is structured as follows. Section reviews related work. Section 3 introduces the MIXTRAINING framework, detailing its design intuition and operational procedure. Section 4 presents comprehensive empirical evaluations across multiple datasets and architectures. Section 5 concludes with discussion of limitations and future directions.
2 Related Work
Learning Pipelines and the relied on supervised learning (SL.) with large labeled datasets (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016). More recently, self-supervised learning (SSL) has emerged as a powerful paradigm for learning transferable ixpresentations without labels, giving rise to a training pipeline that first leverages SSL for representation learning and then adapts the model to downstream tasks via supervised learning. This SSL+ pipeline has been widely adopted across domains, including computer vision (Chen et al., 2020; He et al., 2020, 2022; Wang et al., 2023b), natural language processing (Devlin et al., 2019; Radford al., 2019; Brown et al., 2020; Ouyang et al., 2022; Min et al., 2023), and speech recognition (Hsu et al., 2021; Chen et al., 2022). While SSL+SL consistently improves accuracy over SL alone, especially in low-label regimes, it incurs a substantial increase in compute due to the additional SSL phase (Kaplan et al., 2020; Chung et al., 2024), leading to a computee-errformaeee trade-off between the two approaches. Recent work has begun to examine aspects of this trade-off; for example, Liu et al. (2024) explore strategies to reduce the compute cost of the SSL stage. In this work, we directly study on the SL, and propose a new MIXTRAINING pipeline that adds a dedicated mixtraining phase between SSL and SL, yielding Pareto improvements in both accuracy and compute efficiency over the standard SSL+ pipeline.
Trade-off. Early deep learning models primarily
trade-off between SSL+SL and
3

---

Published in Transactions on Machine Learning Research (04/2026)
objsst
objst
objsst
obb Jat
ob fmix objst
objst
Self-supervised Phase
Supervised Phase
Self-supervised Phase
MixTraining Phase
Supervised Phase
(a) Standard SSL+SL Framework
(b) SSL+SL w/ MixTraining Framework
Figure 2: Comparison of MIXTRAINING with the standard SSL+SL framework. (a) The standard SSL+SL framework features an abrupt transition from self-supervised objective (objaa) to supervised objective (obja)- (b) Our MIXTRAININ framework creates an dedicated mixtraining phase the middle. The mixtraining phase optimizes towards mixed objective (objmi)), which enables smooth transition from the self-supervisod objective to the supervised objective.
Modifications to Learning Objectives. A broad class of methods modifies learning objectives to improve generalization, robustness, or stability. One important direction explores the synergy between different objectives, such as combining self-supervised learning and supervised learning objectives. Prior studies in this space have focused on domain adaptation (Pan et al., 2020; Berthelot et al., 2021), mitigating catastrophic forgetting (He al., 2021; Mchta al., 2023), and improving data efficiency (Zhai et al., 2019; Khosla et al., 2020; Yao al., 2022; Zhang al., 2024). Another related line of work mixes up data from different. sources domains to bridge distribution gaps regularize decision boundaries (Zhang et al., 2017; Yun al., 2019; Verma et al., 2019; Liu et al., 2022; Zou et al., 2023), often by constructing intermediate representations that incorporate information from multiple domains tasks. Our work is inspired by these lines of research but differs in both purpose and design: we introduce a dedicated mixtraining phase between SSL and SI. to smooth the transition between two learning different objectives, and we design this phase for compute efficiency by merging forward and backward passes over the shared backbone an aspect not addressed prior work.
Efficient Training of Deep Neural Networks. A substantial body of research has focused on reducing the computational and data costs of training deep neural networks while maintaining competitive performance. Model compression techniques (Han et al., 2015a,b; Jacob et al., 2018; Frankle & Carbin, 2018; Blalock et al., 2020; Li et al., 2023) reduce network size and complexity through pruning, weight sharing, quantization, other architectural simplifications, thereby lowering both memory footprint and FLOPs for more efficient training and deployment. Parameter-efficien finetuning methods (Hu et al., 2021; Ding et al., 2023; Han et al., 2024) freeze most of the backbone and introduce small trainable modules or low-rank adaptations, greatly reducing optimization cost while preserving downstream performance. Data-cfficient training strategies (Bengar et al., 2021; Mindermann et al., 2022; Wang al., 2023a; Bhatt et al., 2024) accelerate convergence by prioritizing informative examples, filtering redundant noisy data, leveraging pretrained models for improved initialization. These approaches target model, paramcter, or data-level efficiency, and are complementary to our pipeline-level strategy: MIXTRAINING could be integrated with these techniques to further improve its efficiency.
3 Methodology
We briefly introduce the standard self-supervised lcarning and supervised learning (SSL+LL) pipeline in Section 3.1. We introduce our MIXTRAINING framework in Scction 3.2, which consists of its design intuition (Section 3.2.1) and operational procedure (Section 3.2.2).
3.1 Background: The Standard Self-supervised Learning and Supervised Learning Pipeline
The standard self-super learning and supervised learning pipeline consists of two separate phases: a self-superoised learning (SSL) phase and a supervised learning (SL) phase. In the self-supervised lcarning phase, a backbone model with self-supervised learning head trained help the model learn general feature representations. Specifically, this process usually relies on learning from unlabeled data with reconstruction
4

---

Published in Transactions on Machine Learning Research (04/2026)
Forward
Backward
(*Usaally: Backbone size so SelFsuperviieed head size)
Forward
Backward
Standard SSL+SL
Fus
GPU utilization (%)
Rack
Self Self-supervised head
Backbone
R
Supervised bead
Timeline
Fur
GPU utilization (%)
SSL+SL Mix Training
Fict
Mudh
Sclf-supervised head
(Fucc Neee)
Backbone
Supervised bead
Fuus into Faid Merge B, R., into Bu..
Morge
Compustionn
Timeline
Corward-Backwwnr Flow (leff) Computation Graph Visualization Breakdown (right)
Figure 3: Comparison of MIXTRAINING with the standard SSL+SL framework. MIXTRAINING achieves computation gains over the standard SSL+SL framework. Top: Standard SSL+SL. Data first goes through a self-supervised learning pass (Fsa,1 + Fs + Bas + B and then goes through supervised learning pass (Fst1 + Fst, + Bst,2 + Bs.1). Bottom: MIXTRAINING. We merge two forward passes (Fsst. and Fa.1) over the backbone model together into single pass Fiix,1, and use its result for both sule-sueer head and supervised head; the backward passes (Bal,1 and Bst1)) are merged into B (bottom left). Our modifications reduce compulation requirements and allow better parallelization (bottom right).
tasks (Hinton & Zemel, 1993; Hinton & Salakhutdinov, 2006; Michelucci, 2022) predicting manually masked tokens (Devlin et al., 2019; He et al., 2022). The backbone model is further refined in the supervised learning phase, together with a supervised learning head. This step adapts the model to downstream tasks, usually achicving better performances than directly training the downstream tasks. Fig. 2(a) shows the standard SSL+SL framework, which has now become the go-to approach for improving the performance of AI models (Wang et al., 2023b; Min et al., 2023).
3.2 A New Framework: MixTraining
3.2.1
Design Intuition behind MixTraining
While the standard SSL+SL framework has achieved remarkable success, its self-supervised learning and supervised learning phases are completely separated, leaving limited opportunity for interaction or further optimization. To enable closer interactions between these two phases, we propose novel MIXTRAINING framework- as shown in Fig. 2-which effectively mergos several self-supervised learning and supervised learning epochs into additional mirtraining phase, featuring smooth transition between learning objectives.
TRAIIIIG introduces a new mixtraining phase that allows joint updates of self-supervised and supervised objectives, which is in contrast with the standard SSL+SL framework where one first updates the self- supervised objective and then updates the supervised objective. At a high level, the benefits of this phase are rooted in the dedicated joint optimization objective. The joint objective can be easily understood as a weighted average of the supervuue objective and the supervised objective balance these two objectives. We next ex- plain the design intuition behind the mixtraining phase to achieve both computation gains and accuracy gains.
Accuracy Gains. Since the self-supervised learning phase aims at learning general representations and the following supervised learning phase aims at learning task-specific information, intuitively, these two phases optimize the model in different directions. The standard SSL+SL framework features an abrupt change in optimization directions during the transition from self-supervised lcarning and supervised learning (Fig. 2(a)), which may cause instability in model performance (Mosbach el al., 2020). Indeed, there are also studies show that, in certain settings, supervised learning in the second phase can lead to worse model performance (Peters et al., 2019; Kumar et al., 2022). In our MIXTRAINING framework, the mixtraining phase creates middle ground, i.c., a weighted combination of two objectives, allowing a rather smooth transition from the self-supervised learning objectives the supervised learning objective, illustrated in Fig. 2(b). We
5

---

Published in Transactions on Machine Learning Research (04/2026)
hypothesize that such smooth transition avoids instability in phase transition, thus allowing the model to better adapt to the larget task and achieve higher accuracy. We validate the smooth-transition intuition by measuring the cosine similarity of the last-backbone-layer gradients induced by different learning objectives. The average cosine similarity between SSL and SL objectives is only 0.018, indicating that these two objectives are nearly orthogonal. In contrast, the cosine similarity between SSL and Mix MixTraining objectives is 0.791, and between SL and MixTraining objectives is 0.598, suggesting that the MixTraining objective indeod provides a smooth transition from SSL to SL. Our empirical results in Section 4 further support the hypothesis that smooth transitions lead to higher accuracy.
Computation Gains. In the standard SSL+SL framework, we first run self-supervised learning passes with data (Iasl: Vssl), and then run supervised learning passes with data (Zat, Ysi). This process involves forward/ forward/back- back- ward passes of both (Iast: Yei) and (Irat, ya); and strictly follows a sequential order to compute each sub-processer (top part of Fig. 3). In contrast, MIXTRAINING aims to jointly optimize self-supervised and supervised objectives. Specifically, it "merges" (Iaad, Wsaa) and (Ira, Val into a mized data (Imix; ymix) and thus merging the separate forward passes over the backbone model into a single pass, and use its result for both self sele-supereieed head and supervised head; from the computation aspect, we also merge the backward passes of self-supervised learning and supervised learning tasks over the backbone model into a single pass (bottom left part of Fig. 3). Since the size of the backbone model is usually much larger than the size of self-supervised and supervised heads (He et al., 2022; Du et al., 2021; Yang et al., 2023), the merge of forward/ hewarr/bbckweee backward passes over the backbone model allows us to reduce computation compared to the synchronous sctting substantially. Additionally, the merge of forward hrrard/bbawward passes over the backbone model allows better parallelization of forward forward/backward wark passes over the self-supervised and supervised heads (right part of Fig. 3), which further speeds up the computation. Empirically, we find the runtime of single merged mixtraining pass (0.317s) is lower than the combined runtime of an SL pass and an SSL pass (0.543s = 0.248s + 0.295s), validating our intuition.
3.2.2 The Mix Procedure
In this section, we introduce the operational procedure of our MIATRIINING framework in detail. Besides the number of self-supervised learning epoch ésal and the nunber of supervised learning epoch est, MIX- Training takes as input a hyperparameter MIX-RATIO E [0, 1] to determine the number of self-sueervised learning/superviseed learning epochs Emix to be merged into the mixtraining phase. We set
Emix = mininsss ea)]]
(1)
Our TRAINING framework then operates by running (i) the vanilla self-supervised learning phase for Caal - Cmix epochs, (ii) the mixtraining phase for Cmix epochs, and (iii) the vanilla supervised learning phase for eat - emix epochs, as shown in Algorithm 1.
Since the sulf-super learning and supervised learning phases are standard, in the following, we mainly discuss the mixtraining phase. In the mixtraining phase, we (i) design mixing function g to generate a mixed dalasel, and (ii) design joint oplimization objective as supervision signal. We next highlight the design choice for these two parts.
The Mixed Dataset. The goal of creating a mixed dataset Dmix - sel. Da) is to extract information stored in self-supervised learning dataset Dast = {ra}i and supervised learning dataset Da = {(2,, h)). featuring a smooth transition between two learning objectives Fig. 2 (b). In the simple case where self- supervised learning and supervised learning use the same input (i.c., xx;), we can simply sot g(Das, Dai) Dzi- We remark that the importance of this simple case is usually overlooked: conducting self-supervised learning and supervised learning on the same ImageNet dataset allows one to boost the top-1 classification accuracy from 82.5% to 84.9%, without using extra data (He et al., 2022).
We next discuss the general case where the self-supervised learning dataset is not the same as the supervised learning dataset, i.e., Dest # Dat- Inspired by the mixup method in machine learning to improve the gencralization and robustncss to adversarial examples (Zhang et al., 2017), we consider a randomized mixing function g, which randomly mixes up data points from both datasets. Specifically, we set
Dmix g(Dss, Da)= {Imix Vol) Imix Ax + (1 A) Tsst, (Irs Wa) E Dete Tesl E Deal)
(2)
6

---

Published in Transactions on Machine Learning Research (04/2026)
Algorithm 1 The MIXTRAINING Framework
Input:
Sepe-supe learning epoch Esa,, Supervised learning epoch est, MIX-RATIO P E [0.1].
1: Initialize model parameters 0.
2: Calculate mixtraining epoch émix via Eq. (1).
3: & Vanilla Self-supervised Learning Phase
4: for == to Esal Emix do
Conduct vanilla self-supervised learning phase w.r.t. self-supervised loss (1;();; to optimize
b MIXTRAINING Phase
for to emix do Optimize the model parameter 0 with respect to the joint optimization objective as in Eq. (3).
9: & Vanilla Supervised Learning Phase
10: for e = to fs emix do
11: Conduct vanilla supervised learning phase w.r.t. supervised loss fz(ffz),, y: to optimize 0.
where for each (Iss, Va)) E Dst, we randomly draw a self-supervised data point Tsst from Dest and generate a mixup feature Azg + (1 X) Twa: where is a hyperparameter of user's choice (we set X 0.5 in our experiments balance the contribution from both datasets). We adopt the supervised label ysl to provide supervised signal since the self-supervised part typically doesn't require labels.
The Joint Optimization Objective. Let 0 denote the model parameters. Let (z; 0) denote the self- supervised loss, e.g., MSE reconstruction loss for masked autoencoders (He et al., 2022), and let (a(f(() y 0) denote the supervised loss, c.g., cross-cntropy for image classification (Krizhevsky et al., 2012). Let Das represent the SSL dataset, which contains examples {si and Da represent the ST. dataset, which also contains examples ((x,, n. These datasets serve as the sources for self-supervised and supervised learning, respectively. The classical SSL+SL framework first optimizes ming En [<aa(;; (2; 0)] and then optimizes ming (ess y; 0).
To integrate selfssupervised and supervised learning objectives, MIXTRAINING considers a weighted combina- tion of these two objectives and optimizes the following goal:
min E la + (1 - (1) (((()))
where Dmix is the mixed dataset and a E 10 is a hyperparameter LOSS-RATIO designed to balance the focus between learning general representations (by optimizing self-supervised loss (sa(2;; 0)) and achieving specific target (by optimizing supervised loss (a(f(x). f)). Eq. (3) can be naturally extended to include a continuously annealed loss weight that interpolates between the SSL and SL objectives. For simplicity, retain the current form of Eq. (3); our ablation study in Section 4.2.4 shows that both linear and cosine annealing yield comparable performance.
4 Experiments
4.1 Setups
Datasets. We conduct experiments on standard computer vision datasets, including CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), and TinylmageNet (Le & Yang, 2015).
Models. Our model consists of three components: shared backbone model, classification head for SL, and a reconstruction head for SSL. For most of our experiments, we use the standard ViT-Tiny (ViT-T) (Dosovitskiy el al., 2020; Wu et al., 2022) as the backbone and the classification ion head. We adopt masked autoencoding (MAE) (He et al., 2022) as the reconstruction task and use an MAE decoder of depth 2 as the reconstruction head. In Section 4.4, we provide additional experiments with ResNet-18, where the reconstruction task standard autoencoding (AE) and the reconstruction head two-layer transposed convolution (Zeiler et al., 2010).
7

---

Published in Transactions on Machine Learning Research (04/2026)
Table 1: Accuracy and latency comparison using the ViT-T model. We evaluate performance across various data limitation levels p (10%, 25%, 50%, 75%, 100%}. The highest accuracy in each setting is highlighted in bold (if gain 2 0.5%). MIXTRAINING achieves a Pareto improvement over the SSL+SL baseline, achieving higher accuracy and lower latency in 15 out of 15 scttings.
Methods
10%
25%
50%
75%
100%
Accurscy1
Lstency(s)
Accuracyl
Litency(s]) [s]]
Accuracyl
Letency(s]
Arcuracyl
Latency(s)]
Accuracy!
Latency (s)_.
7:3456
1102.42
30.01%
2619.14
91.3655
96.9855
7700.22
1..969
10257
TimyImageNet
SSL+SL
10.215%
2582.00
21.78%
5813.27
$1.30%
11649.43
12.7355
17296.22
66 65.65
22917.20
MicIvalaing
20.99%
1913.75
31.415%
4616.00
42.77%
49.30%
12804.00
65.46%
17790.47
40%
1335.77
27
T9:2256
2844.27
02.52%5
5155.30
CIFAR-10
SSL+SL
36,7955
1253.65
67.98%
2774.24
77.005
535408
82:1158
7953.72
84 54.8295
10730.52
MicTeaining
60.46%
W02.1
72.61%
2206
79.90%
4247.78
83.97%
0234.03
87.18%
8274.46
19.05%
600.1
31.40%
1346.66
42.00%6
2580008
48.9756
2814.54
72%7
G1122.96
CIFAR-100
22:27%
1279.
2882.62
53.7156
MixTrsining
25.19%
1010.94
38.55%
2225-92
48.60%
4238. 76
56.8496
6354.70
60.99%%
8457 93
Baselines.
We evaluate the performance of our algorithm (Algorithm 1) against the following baselines:
Superoised learning (SL). Conduct standard supervised learning on the backbone and the classification head with cross-entropy loss for est epochs.
Self-supervised learning + supervised learning (SSL+SL). Conduct self-supervised learning on the backbone and the reconstruction head with MSE loss for esst epochs and then conduct standard supervised learning with cross-entropy loss for est epochs.
The comparison between SL and SSL+SL reflects the better performance at the cost of the added computation in the SST.-alone step. We aim to provide a better compute-perforannce trade-off with MIXTRAINING, i.c., a Pareto improvement over the SSL+SL baseline. We adopt a standard set of augmentations following the practices of He al. (2016); Cubuk al. (2020); Charisoudis et al. (2023), detailed in Appendix A.3. We remark that our goal is not to maximize benchmark accuracy, but to enable controlled and reproducible comparison across different training pipelines.
trade-off: SSL+SL achieves
Evaluation Metrics. For cach method, measure its performance by the accuracy the downstream classification task and its computation cost by the training latency (i.e., the total wall-clock time). We report the average accuracy and latency over 4 runs with different random seeds. We calculate the speedups of our method as the ratio between the latency of SSL+ and MIXTRAINING.
Other Implementation Details. We conduct experiments across various data limitation levels by randomly select fraction of p data points from the original dataset; we choose E (10% 25%, 50%, 75%, 100%). our main experiments, we set training epoch 6ş = Cea 100, LOSS-RATIO conduct detailed parameter studies for these quantities in Appendix A.6. We defer additional experimental details to Appendix A.
0.5, and MIX-RATIO 0.5; we
4.2
Main Results
We present our main experiment results in this section. Results across different data limitation levels are provided in Section 4.2.1, and results with varying SSL and SL datasets are presented in Section 4.2.2.
4.2.1 Performance Analysis across Various Data Limitation Levels
In this scction, we evaluate the performance of MIXTRAINING across various data limitation levels. We conduct experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet. datasets and present results in Table 1. As expected, the comparison between SSL+SL and SL reflects a conpute-performant trade-off: SSL+$L achieves better accuracy at the cost of higher training latency. Our MIXTRAINING method achieves a Parcto improvement over the SSL+SL baseline: MixTraining achieves higher accuracy and lower latency in 15 out of 15 settings.
8

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: Accuracy and latency comparison with different self-supervised and supervised datasets using the ViT-T model. We use TinylmageNet as the self-supervised learning dataset and use CIFAR-10/CIFAR-O00 as the supervised learning dataset. The highest accuracy in each setting is highlighted in bold (if gain 2 0.5%). TRAININN achieves a Pareto improvement over the SSL+SL bascline, achieving higher accuracy and lower latency in both settings.
TinyImageNe to CIFAR-10
TinyImageNet to CIFAR-100
Methods
Accuracyt
Latency(s)4
Accuracyt
Latency(s)
SL SSL+SL
81.76%
5203.76
54.22%
5149.24
84.48%
16815.33
56.22%
16582.02
MixTraining
89.19%
12098.50
58.49%
13742.15
Compared to baselines, MIXTRAINING achieves significant accuracy gains: for instance, the full Tinylma- geNet dataset with the ViT-T model, MIXTRAINING achicves 18.89% relative accuracy gain (8.81% absolute accuracy gain) over SSL+SL and 32.17% relative accuracy gain (13.50% absolute accuracy gain) over SL. The accuracy gains are more significant under limited data: for instance, on the Tiny Tinylmaeeeet dataset and at data limitation level of 10%, MIXTRAINING achieves 105.58% relative accuracy gain (10.78% absolute accuracy gain) over SSL+SL and 189.92% relative accuracy gain (13.75% absolute accuracy gain) over SL. MIXTRAININS also saves more compute (reflected as training latency) compared to SS.+ For instance, on the full TinyImageNet datasct with the ViT-T model, MIXTRAINING acheives 1.29x speedup compared to SSL+SL. These results show that MITTnIN provides better compule-performance trade-off compared to the standard SSL+SL pipeline across various data limitation levels.
4.2.2 Self-Supervieed Learning and Supervised Learning on Different Datasets
In this section, we evaluate the performance of MIXTRAINING in settings where the self-supervised learning dataset and supervised learning dataset are different. We perform self-supervieed learning on the TinylmageNet dataset and supervised learning on the CIFAR-10 or CIFAR-100 datasets. We present the results in Table 2. The comparison between SSL+SI, and SL still reflects the compute-performance trade-off: SSL,+SL achieves better accuracy the cost of higher training latency. Our MIXTRAINING method achieves a Pareto improvement over the SSL+SL baselinc: MixTraining achieves higher accuracy and lower latency both settings.
Compared to baselines, MIXTRAINING achieves significant accuracy gains: for instance, on the CIFAR-10 dataset with the ViT-T model, MIXTRAINING achieves 5.58% relative accuracy gain (4.71% absolute accuracy gain) over SSL+SL and 9.09% relative accuracy gain (7.43% absolute accuracy gain) over SL. In terms of computation cost (reflected as training latency), MIXTRAINING also saves more compute compared to SSL+ For instance, on the CIFAR-10 dataset with the ViT-T model, MIXTRAINING can achieve 1.39x speedup compared to SSL+SL. These results show that MIXTRAINING can provide better compute-performand trade-off compared to the standard SSL pipeline, even when using different datasets for self-supervised learning and supervised lcarning.
4.2.3 Mix MixTraining is More Powerful than Standard Data Mixups
To examine the advantages of MIXTRAINING over standard data mixups (Zhang et al., 2017), we further equip the SL and SSL+SL baselines with mixup data augmentation (Table 3) on both CIFAR-10 and CIFAR-100 datasets. While mixup improves the baselines' performance, our proposed MIXTRAINING method consistently achieves the best performance; see Table 15 for results across different data levels. This demonstrates that MIXTRAINING provides advantages that standard mixup alone cannot achieve the gains stem from the unified MIX TRAINING pipeline described in Section 3.2, rather than just from augmentation effects.

---

Published in Transactions on Machine Learning Research (04/2026)
Table 3: Accuracy comparison on CIFAR-10 and CIFAR-100 with mixup augmentations applied to the SL and SSL+SL baselines (with ViT-T). We report results under the 100% data setting. The highest accuracy is highlighted in bold (if gain 2 0.5%). MIXTRANIIG achieves the best performance over the baselines.
Datasets
CIFAR-10
CIFAR-100
SL w/ mixup
83.87%
58.96%
SSL+SL w/ mixup
85.81%
59.41%
MixTraining
87.13%
59.95%
4.2.4 Smooth Interpolation of a in the Mixtraining Phase
While Eq. (3) uses a fixed loss weight a [0. 1], it can be naturally extended to include a continuously annealed weight that interpolates between the SSL and SL objectives. In Table 4, conduct ablations with both linear and cosine annealing schedules. Since both variants achieve comparable performance to our original MIXTRAINING approach, this indicates that the key advantage lies in the existence of the mixtraining phase itself, rather than in the specific scheduling of the weight a.
Table 4: Ablation of linear and cosine annealing schedules for loss weight in MIXTRAINING. We conduct. experiments the CIFAR-10 dataset with the ViT-T model. All variants achieve comparable performance.
Methods
10%
25%
50%
75%
100%
Algorithm
60.45%
72.61%
79.95%
83.97%
87.13%
A Algorithm w/ linear
61.49%
72.21%
79.07%
84.82%
86.90%
Algorithm 1 w/ cosine
60.65%
72.03%
79.16%
83.40%
87.00%
4.2.5 Impact of Training Epochs
We study the impact of varying training epochs esst est on model accuracy and training latency. For simplicity, we Essl = est and choose its value from {50, 75, 1003. 1 We conduct experiments across various choices of data limitation levels p (10%, 25%, 50%, 75%, 100%). Due to space limitations, we present results for learning with full data (p 100%) in the main content, and defer the complete results to Appendix A.4.
We present results with full data in Table 5. We observe that, compared to SL. and SSL+SL, MIXTRAINING generally yields greater accuracy gains for added computation; for example, doubling the training epochs from 50 to 100 results in a larger improvement accuracy. Interestingly, MIXTRAINING with é = Eg = 50 outperforms ST. with 6 100 and SSL+ST. with est = fs 100 in terms of both accuracy and compute: MIXTRAINING simultaneously achieves higher accuracy and lower training latency. MINTRAINING consistently acheives a Pareto improvement over the SSL+SL baseline: MixTraining achieves higher accuracy and lower latency in all out of 9 settings. Combining results in Appendix A.4 with data limitations levels [25%, 50%, 75%, 100%]. MixTraining achieves higher accuracy and lower latency in all 45 out of 45 settings. These improvements again demonstrate that MIXTRAINING superior training pipeline compared to the standard SSL+SL approach.
4.3 Mix Training in New Settings
4.3.1
Learning in Semi-Supervised Settings
We extend our experiments to the semi-supervised sotting where 50% data are labeled and the other 50% data are unlabeled. In this setup, the SL baseline trains only on the labeled data, while the SSL+SL baseline
When Coll cy 100, all validation loss curves have convergod. Some validation losses may not fully converge when the number of training epochs reduced; however, this intentional, as we aim study compute- perfomane trade-offs under compute-constrainee settings.
10

---

Published in Transactions on Machine Learning Research (04/2026)
Table 5: Parameter study on training epochs with full data using the VTT-T model. Results under other data limitation levels p (10%, 25%, 50%, 75%) are deferred Appendix A.4. The highest accuracy each setting highlighted in bold (if gain 2 0.5%). MINTRAINING achieves Pareto improvement over the SSL+ baseline, achieving higher accuracy and lower latency in 9 out of 9 settings.
SL
SSL+SL
MixTraining
Datasets
Computation (epoch)
Accuracy?
Lateucy(s)4
Accuracyt
Latency(s).
Accuracyt
Latency(s)
39.75%
4957 .82
42.16%
10714.36
50.96%
8410.97
Net
41.74%
7560140
44.22%
16499.39
53.09%
13111.49
41.96%
10257.12
46.65%
22917.29 29
56.46%
17795.47
80.78%
2189.35
83.75%
5365.33
85.60%
1193.12
CIFAR-10
81.50%
3818.88
84.48%
8070.96
86.79 79%
6515.22
100
81.52%
5155. 59
84.69%
10730.52 52
87.13%
8274.45
50
51.25%
2713.55
56.91%
5763.71
58.67%
442163
CIFAR-100
75
54.71%
3881.84 181
56.95%
8104.50
59.11%
6459.23
100
51.72%
5022. 96
37.92%
10960.08
59.95%
8457 .33
Table 6: Accuracy comparison in semi-supervised setting with 50% labeled data and 50% unlabeled data. We conduct experiments on the CIFAR-10 dataset with the ViT-T model. MIXTRAINING achieves the best result over the baselines.
Metric
SL
SSL+SL
MixTraining
Accuracy!
75.59
83.89
87.35
first applies reconstruction-buseed SSL all data and then performs supervised learning on the labeled subset. In our MIXTRAINING framework, the SSL and SL phases remain the same as in the standard SSL+SL pipeline, while the intermediate mixtraining phase randomly mixes labeled data and train using the objective in Eq. (3). As shown in Table 6, MIXTRAINING achieves the highest accuracy in this semi-supervised setting, with even larger gains over the baselines. This further highlights the effectiveness of MIXTRAINING.
4.3.2 Learning with Data Noise
We simulate real-world environment by adding random label noise to the dataset: for each data point, with probability 0.2, we randomly change its label. Table 7 shows that our MIXTRAINING framework continues to achieve the best performance with data noise, verifying its robustness in practical settings.
4.4 When MixTraining Helps and When It Doesn't
We conduct additional experiments with ResNet-18 on CIFAR-10 (Table 8), a setting where the SL baseline is already highly optimized. Across all data limitation levels, MIXTRAINING consistently improves upon the SSL+SL baseline. However, when SSL+SL (with an added SSL stage) underperforms a strong SL baseline, MIXTRAINING- since also includes SSL-doos not fully recover ST. performance. In contrast, in regimes where SSL+SL improves over SL (e.g,, under limited data), MIXTRAINING further improves upon SSL+SL.
These results refine-bit do not contradict main conclusions. MIXTRAINING consistently improves upon SSL+SL and offers favorable compute-performanc trade-off;; yet in settings where SL is alroady highly optimized and adding SSL hurts performance, MIXTRAINING may not surpass SL.
5 Conclusion and Future Work
We introduced MIXTRAINING, an innovative framework that interleaves multiple self-supervised learning (SSL) and supervised learning (SL) epochs within a unified training phase, enabling a smooth transition between the two learning objectives. By enhancing the synergy between SSL and SL, MIXTRAINING achieves significant accuracy improvements while consolidating shared computation steps reduce computational cost. Extensive experiments demonstrate that MIXTRAINING offers superior compute-performmnce trade-off
11

---

Published in Transactions on Machine Learning Research (04/2026)
Table Accuracy comparison on CIFAR-10 (with ViT-T) with random data noise. We evaluate performance across various data limitation levels p E {10% 25%, 50%, 75%, 100%). The highest accuracy is highlighted in bold (if gain 2 0.5%). MINTRAINING achieves the best performance over the baselines.
Methods
10%
25%
50%
75%
100%
SL SSL+SL
55.80%
64.64%
72.96%
78.33%
81.27%
56.61%
68.45%
72.94%
80.98%
84.17%
Mix MixTraining 61.18%
70.44%
75.97%
84.01% %
86.13%
Table 8: Accuracy and latency comparison using the ResNet-18 model. We evaluate performance across various data limitation levels p E (10%, 25%, 50%, 75%, 100%]. The highest accuracy in each setting is highlighted in bold (if gain 2 0.5%). MIXTRAINING consistently improves over SSL+SL but may not match the SL performance if SSL+SL degrades performance relative to a strong SL baseline.
Datasets
Methods
10%
25%
50%
75%
100%
SL 60.42
79.86
85.31
87.97
92.05
CIFAR-10
SSL+SL
65.29
82.20
85.24
87.12
87.18
MixTraining
67.33
82.73
85.30
87.63
87.91
SL 28.76
40.67
50.25
56.56
77.87
CIFAR-100
SSL+SL 28.57
40.78
51.44
55.73
59.76
MixTraining
29.81
41.76
52.30
57.62
61.54
compared to the conventional SSL+SL pipeline: MIXTRAINING achieves substantial improvements in model accuracy while significantly reducing training latency.
While our work focuses on scenarios where practitioners need to train models from scratch due data privacy constraints domain-specific requirements, we believe MIXTRAINING also be leveraged in larger-scale settings when positioned as mid-(raining strategy, rather than a replacement for fully task-agnostic pretraining. As an initial validation of this idea, we conduct an additional experiment where a ViT-T model pretrained on Tiny Tetgeeeeeet is adapted to CIFAR-10. Compared to directly fine-tuning the pretrained model (84.8%% accuracy), applying MixTraining on the target dataset (as pretraining data is typically unavailable in this case) improves performance to 88.75% indicating that MixTraining can preserve general representation while enhancing task-specific adaptation. We believe this mid-training perspective opens an interesting direction for future work.
References
Javad Zolfaghari Bengar, Joost van de Weijer, Bartlomicj Twardowski, and Bogdan Raducamu. Reducing label effort: Self-supervised meets active learning. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 1631-1639, 2021.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint arXiv:2106. 0tff::2106004333. 2021.
Gantavya Bhatt, Yifang Chen, Arna M Das, Jifan Zhang, Sang T Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey Bilmes, Simon S Du, Kevin Jamieson, et al. An experimental design framework for label-efficient supervised finetuning of large language models. arXiv preprint ar xrXic:2401.06692 2024.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129 146, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. A Advances in neural information processing systems, 33:1877-1901, 2020.
12

---

Published in Transactions on Machine Learning Research (04/2026)
Athanasios Charisoudis, Simon Ekman von Huth, and Emil Jansson. [re] masked autoencoders are small scale vision learners: A reproduction under resource constraints. In ML Reproduciiiiktyy Challenge 2022, 2023.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavim: Largo-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1606):1505-1 1518, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling inteructionniinctuuee language models. Journal of Machine Learning Research, 25(70):1 53, 2024.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition workshops, pp. 702 703, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language underslanding. In Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. srrameter-effcient fine-tuning of large-scale pre-trained language models. Nature machine intelligence, 5(3):220-25) 235, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1100 11929, 2020.
Xianzhi Du, Barret Zoph, Wei-Chil Hung, and Tsung-Y Yi Lin. Simple training strategies and model scaling for object detection. ar preprint ar orX:::1107.0000 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arv preprint Xrxtc:1803.08 03635, 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:1510.00149, 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient. neural network. Advances in neural information processing systems, 28, 2015b.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-cfficion fino-tuning for large models: A comprehensive survey. arXiv preprint arxiv:2403. 14608, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 16000016000 2022.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. Analyzing the forgetting problem in pretrain-finetuingg of open-domain dialogue response models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1121-1133, 2021.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313 507, 2006.
Geoffrey E Hinton and Richard Zemel. Autocncoders, minimum description length and helmholtz free energy. Advances in neural information processing systems, 6, 1993.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and AbdeIrahman Mohamed. Hubert: Self-supervieed speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 11:3451-4460 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 2021.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-rithmmeti-onit inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0 083611, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Tsola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in newral information processing systems, 33:18661-1867, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
Ananya Kumar, Aditi Raghunathan, Robbic Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXie preprint arXiw:2202.1000 10054, 2022.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Zhuo Li, Hengyi Li, and Lin Meng. Model compression for deep neural networks: A survey. Compulers, 12 (3):60, 2023.
Andy Liu, Yi-Cheng Lin, Haibin Wu, Stefan Winkler, and Hung-yi Lee. Efficient training of sul--supeevieed speech foundation models compute budget. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 961 968. IEEE, 2024.
Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z. Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference on Computer Vision, pp. 441-458, 2022.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24(214):1-50 2023.
Umberto Michelucci. An introduction to autoencoders. arXiv preprint arXiv:2201.032 033898 2022.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veysch, Thien Huu Nguyen, Oscar Sainz, Encko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Sure., 56(2), sep 2023. ISSN 0360-0300. doi: 10. 10.1145/360594
14

---

Published in Transactions on Machine Learning Research (04/2026)
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt. Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649. PMLR, 2022.
Marius Mosbach, Maksym Audriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arxi2:0006. 04884, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Aduances in neural information processing systems, 35:27730-27744, 2022.
Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon. Unsupervised intra-domain adaptation for semantic segmentation through sell-supervision. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 3764 3773, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint S5t::1993.099877 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint orXie:1409.1556 2014.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International conference on machine learning, pp. 6438 6447. PMLR, 2019.
Peihao Wang, Rameswar Panda, and Zhangyang Wang. Data efficient neural scaling law via model reusing. In International Conference on Machine Learning, pp. 36193-36204. PMLR, 2023a.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained modcls: A comprehensive survey. Machine Intelligence Research, pp. 1-36, 2023b.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXio preprint or Xiv:221100000
2021.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In Europeun Conference on Computer Vision, pp. 68 85. Springer, 2022.
Yu-Qi Yang, Yu-Xiao Guo, Jian-Yu Yu Xiong, Vang Liu, Hao Pan, Peng-Shuai Wang, Xin Tong, and Baining Guo. Swin3d: A pretrained transformer backbone for 3d indoor scene understanding. arXie preprint arXiv:2304.06006 2023.
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. Nlp from scratch without large-scale pretraining: A simple and efficient framework, In International Conference on Machine Learning, pp. 25438-251 25451. PMLR, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on compuler vision pp. 6023 6032, 2019.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In 2010 TEEE Computer Society Conference on computer vision and paltern recogralion, pp- TEEE, 2010.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1476-1485, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. orXiv preprinl arXiv:1710.0000 2017.
Jifan Zhang, Yifang Chen, Gregory Canal, Arnav Mohanty Das, Gantavya Bhatt, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Shaolei Du, Kevin Jamieson, et al. Labelbench: comprehensive framework for benchmarking adaptive label-officient learning. Journal of Data-centric Machine Learning Research, 2024.
Yingtian Zou, Vikas Verma, Sarthak Mittal, Wai Hoh Tang, Hieu Pham, Juho Kannala, Yoshua Bengio, Arno Solin, and Kenji Kawaguchi. Mixupe: Understanding and improving mixup from directional derivative perspective. Uncertainty in Artificial Intelligence, pp. 2597-2601 PMLR, 2023.
16

---

Published in Transactions on Machine Learning Research (04/2026)
A Additional Implementation Details and Experimental Results
A.1
Software and Hardware Dependencies
All of the codes are based PyTorch² (Paszke et al., 2019) with timm library (Wightman et al., 2021). All experiments in this paper are conducted on one NVIDIA RTX 6000 Ada GPU.
A.2 Computation Reuse for Different Self-Supervieed Learning Approaches
MAE. To enable compute reuse in MIXTRAINING, we use a modified version of MAE. All input patches are first processed by the backbone to produce feature representations. For the classification head, use all feature representations directly. For the reconstruction head, we randomly mask a subset of the feature representations according to a predefined mask ratio and compute the reconstruction loss only on the masked patches. This modification changes the reconstruction behavior of MAE but leaves the supervised learning stage unchanged, which is the basis for performance evaluation in our experiments.
Standard AE. Computation reuse in the standard AE approach is straightforward: the backbone outputs are fed directly both the reconstruction head and the classification head.
A.3 Implementation Details of Computer Vision Models
VIT-T. Our ViT-T implementations are largely derived from the seminal work from Wu et al. (2022) and shrink the decoder to layers as suggested in the reproduction challenge by Charisoudis et al. (2023). Specifically, we set the embodding dimension ('cmb_dim'') to 12 and 2 layers respectively, alongside 3 heads each for both encoder and decoder. The masking ratio is maintained at 0.75 He et al. (2022) suggest. For the CIFAR-10, CTFAR-100 datasets, the image resolution is standardized to 32 32 pixels with a patch size of 2, while the image resolution is 64 64 pixels with a patch size of 4 for TinylmageNet to ensure uniform computational complexity across all experiments.
192, with the encoder and decoder configured
ResNet-18. Our 18 implementations are derived from the seminal work from He et al. (2016) on deep residual networks. To facilitate self-supervised learning through autoencoders (AE), we augment the ResNet architectures with two additional two-dimensional transposed convolution layers bost-backbone for the reconstruction task.
Data Augmentation and Preprocessing. For the preprocessing of CIFAR-10 and CIFAR-100, we adopt simple preprocessing as in He et al. (2016), which randomly crops the images to size of 32 x 32 pixels, with padding of pixels each side of the image, then randomly flips the images horizontally with a 50% probability. For TinylmageNet, (scale (0.6, 1.0) and ratio (0.9, 1.1)), and RandAug(2,9) (Cubuk et al., 2020; Charisoudis et al., 2023) for data augmentation.
apply random horizontal flip (probability 0.5), random resized crop
Hyperparameters. Detailed training hyperparameters used in our experiments are summarized in Table 9 (ViT-T) and Table 10 (ResNet--8). For experiments on the TinylmageNet dataset using ViT-T, follow- ing Charisoudis et al. (2023), we slightly modify the hyperparameters Table 9: we set the base learning rate to 1 x 10-3 and 2 10-3, the Adam betas to (0.9, 0.95) and (0.9, 0.999), and the weight decay to 0.15 and 0.05 for self-supervised and supervised learning, respectively.
A.4 Additional Experiment Results
We report additional experimental results across various data limitation levels and training epochs. Results with full data are provided main text (Appendix A.6), and results with p E (10% 25%, 50%, 75%) data limitation levels are presented in Tables 11 to 14. These results show that MIXTRAININ provides better
2nttps://pytarch.org 3 Sntsss://hugiingffcecoo///t
17

---

Published in Transactions on Machine Learning Research (04/2026)
Table Hyperparameters for ViT-T across self-supervised and supervised settings.
Hyperparameters
Self-Supervised learning
Supervised Learning
Batch Size
256 1.5 x 10
256 1 10
Base Learning Rate
Learning Rate Scheduler
CosineAnnealligg AdamW
Aosealnn AdamW
Optimizer
Betas
(0.9, 0.95) 0.05
(0.9, 0.95) 0.05 5
Weight Decay
Warmup Epoch
20
Table 10: Hyperparameters for ResNet-18 across self-supervised and supervised settings.
Hyperparameters
Self-Supervised learning
Supervised Learning
Batch Size
256
256 1 x 10-3 3
Base Learning Rate
1 x 10
Learning Rate Scheduler
Cosine Annealing AdamW (0.9, 0.95)
CosineAnnealinn Adamw
Optimizer
Betas
(0.9, 0.95) 0.05
Weight Decay
0.05
Warmup Epoch
20
5
compute-perfommacce trade-off compared to the standard SSL+SL pipeline: MixTraining achieves higher accuracy and lower latency in all 36 out of 36 scttings over SSL+SL
Table 11:
Parameter study on training epochs under 10% data limitation using the ViT-T model. The highest accuracy in each setting highlighted in bold (if gain 0.5%). MIXTRAINING achieves Pareto improvement over the SSL+SL baseline, achieving higher accuracy and lower latency in 9 out of 9 settings.
SL
SL
MixTraining
Datasets
Computation (epoch)
Accuracy?
Lateucy(s)
Accuracyt
Latency(s)
Accuracyt
Latency(s)
7.69%
536.82
8.67%
1136.96
16.53%
896.16
ImayImageNet
7.62%
809.11
9.89%
1762.56
20.23%
1376.22
100
7.24%
1102.42
10.21%
2382.06
20.99%
1913.75
53.28%
287.83
54.75%
637.27
57.60%
473.80
C-FAR-10
75
53.07%
434.86
56.59%
937.01
58.59%
742.53
100
53.40%
589.44
56.79%
1253.65
60.45%
962.16
50
19.08%
291.14
20.26%
677.20 727.26
23.36%
516.54
C-FAR-100
19.39%
464.97
21.15%
981. .43
24.27%
770.
609.11
22.27%
1279.78
25-19%
1010.94
A.5 Comparison of Baselines with Mixup Augmentations across Different Data Levels
We present in Table 15 of experiment comaprison against baselines with mixup data augmentations across different data limitation levels.
A.6 Parameter Study
In this section, we explore the impacts of varying LOSS-RATIO a, MIX-RATIO p, and training epochs esgl and ea for MIXTRAINING
18

---

Published in Transactions on Machine Learning Research (04/2026)
Table 12: Parameter study on training epochs under 25% data limitation using the ViT-T model. The highest accuracy in each setting highlighted in bold (if gain 0.5%). MIXTRAINING achieves a Pareto improvement over the SSL+SL baseline, achieving higher accuracy and lower latency in 9 out of 9
settings.
Datasets
Computation (opoch)
SL
SSL+SL
MixTraining
Accuracy?
Latencyys) (+)
Accuracy
Latency(s))
Avcuracy1
Latency (s)
17.00%
1268.
17.31%
2771.72
31.69%
2129.89
Tiny Net
19.80%
4223.24
31.90%
3281 100
20.91%
2619.14
21.78%
5813.27
31.43%
4516.69
65.03%
646.45
67.65%
1409.32
70.69%
1076.16
CIFAR-10
65.60%
985.54
68.37%
2135.05
71.29%
1719.24
100
66.03%
1335. 79
67.95%
2771.21
72.61% 61%
2206.13
50
30.87%
651.25
34.46%
1517.26
36.87%
1161 89
CIFAR-100
75
31.21%
1033.91
35.05%
2237.72
38.43%
1720.56
100
31.49%
1346.96
34.93%
2882.62
38.55%
2226.92
Table 13: Parameter study training epochs under 50% data limitation using the ViT-T model. The highest accuracy in each setting is highlighted in bold (if gain 0.5%). MIXTRAINING achieves Pareto improvement over the SSL+SL baseline, achieving higher accuracy and lower latency in 9 out of 9 settings.
Datasets
Computation (epoch)
SL
SSL+SL
MixTraining
Accuracy!
Latency(s]
Accuracyt
Lateney(s))
Accuracy1
Latency(s)
21.95%
2196.22 29
27.27%
5401.40
40.37%
1280.53
Tiny ImageNet
28.81%
3827 33
31.15%
8363.36
41.99%
6530.16
100
30.36%
5181.44
34.30%
11449.43
42.77%
8868.85
50
73.70%
1252.78
75.26%
2703.06
78.83%
2133.61
CIFAR-10
75
74.63%
1931.89 19
76.40%
4155.73
79.31%
3255.72
100
75.00%
2503.27
77.06%
5354.08
79.95%
4247.78
42.32%
1248.00 022
44.95%
29/4.08
48.53%
2246.85
CIFAR-100
42.46%
1981.46
45.23%
4355.99
48.87%
3275.14
42.50%
2580.08
46.02%
48.60%
4298.75
Table 14: Parameter study on training epochs under 75% data limitation using the ViT-T model. The highest accuracy in each setting is highlighted in bold (if gain 2 0.5%). MIXTRAINING achieves a Pareto improvement over the SSL+SL baseline, achieving higher accuracy and lower latency out of 9 scttings.
SL
SSL+SL
MixTriining
Datasets
Computation (epoch)
Accuracy?
Lateucy(s)]
Accuracy
Latency(s)
Accuracyt
Lateacy(s)
50
33.16%
3713.97
34.81%
8062.47
46.44%
6369.23
Nmt
36.64%
5708.06
40.31%
12538.22
49. 29.299
9779.95
36.98%
7709.21
42.73%
17296.22
49.30%
13304.60
78.08%
1868.81
79.60%
4033.65
82.91%
3163.51
CIFAR-10
78.74%
2856.25
81.24%
83.
4762.06
79.22%
3844.27
82.11%
7993.72
83.97%
6234.03
48. 99.99%
51.85%
43:15.17
5-1.30%
3329 29
CIFAR-100
49.00%
29291.78
53.25%
6312.75
55.05%
4857.74
100
48.97%
3814.54
53.79%
8226.83
55.84%
635-1776
19

---

Published in Transactions on Machine Learning Research (04/2026)
Table 15: Accuracy comparison on CIFAR-10 (with ViT-T) with mixup augmentations applied the SL and SSL+SL baselines. We evaluate performance across various data limitation levels p e {10%, 25%, 50%, 75%, 100%). We conduct mixup for both of the other two baselines. The highest ac- curacy is highlighted in bold (if gain 0.5%). Mix TRAINING achieves the best performance over the baselines.
Methods
10%
25%
50%
75%
100%
SL w/ mixup
56.45%
68.41%
76.18%
80.91%
83.87%
SSL+SL w/ mixup
57.92%
70.08%
77.50%
82.83%
85.81%
MixTraining
60.45%
72.61%
79.95%
83.97%
87.13%
Table 16: Parameter study LOSS-RATIO We train the VIT-T model with full data and set 0.75; other experimental settings remain the same as Table 1. The best and second-best accuracies are highlighted in bold and underline, respectively.
a
TinyImagell
CIFAR-10
CIFAR-100
0.01
41.58%
79.61%
52.58%
0.1
53.17%
82.19%
55.21%
0.5
53.86%
85.75%
58.88%
0.9
48.18%
86.13%
58.47%
0.99
42.67%
83.75%
56.04%
Table 17: Parameter study MIX-RATIO p. We train the ViT-T model with full data and set 0.5; other experimental settings remain the same Table 1. The best and second-best accuracies are highlighted in bold and underline, respectively.
P
TinyImagen
CIFAR-10
CIFAR-100
0.25
55.08%
85.49%
60.98%
0.50
55.40%
85.54%
60.15%
0.75
53.86%
85.75%
58.88%
1.00
49.21%
84.33%
55.80%
A.6.1 Impact of loss-ratio a
We study the impact of varying hyperparameter LOSS-RATIO a model accuracy in this section. We conduct experiments with a E (0.01, 0.1, 0.5, 0.9, 0.99) and report the accuracy in Table 16; we didn't report the latency since varying a docsn't change the overall computation cost. As shown in Table 16, LOSS-RATIO a = 0.5 generally leads to good accuracy gains either achieving the highest accuracy (2 out 3) or achieving the second-best accuracy out of 3). This indicates that well-chosen a should appropriatcly balance self-supervised learning and supervised learning objectives MINTRAINING. An 0.5 allows the model to focus both self-supe supervisdd learning and supervised learning objectives. Therefore, recommend setting = 0.5 in experiments.
A.6.2 Impact of mix-ratio p
We study the impact of varying the hyperparameter MIX-RATIO p on model accuracy in this section. We conduct experiments with p E (0.25, 0.5, 0.75, and report the accuracy in Table 17; didn't report latency since it is straightforward to see that large P reduces latency (Section 3.2). Values of 0.5 p 0.25 generally leads to good accuracy gains. Since larger reduces latency, its selection should be guided by individual priorities, such as accelerating the learning process or achieving higher accuracy.
20


Reviews:
[{"review_id": "up3QsJdoLF", "reviewer": "Reviewer_YG8C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel training strategy that combines SSL and SL objectives by introducing an intermediate *mix training* phase that is done in between the SSL and SL phases. During mix training, SSL and SL are treated as two different tasks, and the backbone is trained in a \"Multi-task Learning manner\" (i.e. a shared backbone and an independent head for each \"task\" -- i.e. the SSL and SL objectives), additionally they propose a \"mix-up\" fusion of data in case the SSL and SL data distributions are different. \n\nThe work also presents several experiments that show that MixTraining achieves better performance than sequential SSL + SL training, albeit in small-scale networks and datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claims (i.e. MixTraining achieves better pareto-optimal performance than SL or SSL + SL strategies) are well supported by experiments in section 4, where the different strategies are compared for CIFAR10, CIFAR100 and Tiny Imagenet with ViT-Tiny and ResNet18.\n\nAlthough the evaluation is small-scale (both in terms of network and dataset size) for the standards in SSL pretraining works, this is acknowledged and discussed by the authors, so within this scope I would say claims are supported.", "audience_yn": "Yes", "audience_explanation": "I think the idea of combining the SSL and SL objectives in a multi-task manner seems reasonable and can be of interest to the community. For starters, this work can be relevant to the \"efficient-training\" community, where limited data is available. However, of probably more relevance, I would be interested in seeing how this MixTraining strategy can be:\n 1) Scaled to larger settings, since to my understanding there is no limitation to use this in larger networks/datasets besides budget.\n 2) Combined with already SSL pre-trained models (e.g. DINOv3) and use MixTraining during downstream fine-tuning to further boost performance.", "requested_changes": "I propose two adjustments that are not critical for acceptance yet in my opinion would strengthen the work:\n\n1. Although claims are overall supported and ablations extensive, I missed one ablation on the mixed-dataset strategy (i.e. when SSL and SL datasets are different). I could not find any ablation comparing the mix-up strategy with any other strategy. \n\nFor instance, the simpler baseline of using only the SL data for both SL and SSL (instead of mixing them), or mixing them with other strategies (e.g. dividing the batch between SL and SSL data and then splitting the features and sending them to the relevant heads). \n\n2. Given that the authors have not performed experiments on larger-scale settings due to computational constraints, I would encourage the addition of an additional section to discuss how the authors think the MixTraining strategy could be leveraged beyond the scope of constrained budget.\n\nFor instance, do they consider the MixTraining strategy is suitable for large-scale training as well? One potential drawback I can see is that, when training at scale, the strategy is usually  to spend a lot of compute to train a single large \"foundation\" model whose features are extremely versatile (i.e. useful even with frozen backbone for many downstream applications). In that sense, one would try to make the pre-training as task-agnostic as possible to keep the generalization of features and introducing MixTraining at this phase might be harmful. However, when fine-tunining a general backbone like DINOv3 for specific application, I can see MixTraining as a way to try to keep the generality of the encoder while fine-tuning it for a specific task.\n\n3. Following-up on point 2. It might be interesting to leverage an already pre-trained model with SSL (at large scale) e.g. DINOv2-small or Dinov2-Base if budget allows and apply MixTraining as part of a fine-tuning strategy.", "broader_impact_concerns": "No concerns", "additional_comments": "No additional comments", "extra_scores": {}, "extra_text": {}}, {"review_id": "HwrRMs2elV", "reviewer": "Reviewer_Dw8m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a training method where semi-supervised learning and supervised learning is done jointly for faster convergence and better accuracy.", "claims_and_evidence_yn": "No", "claims_explanation": "The forward/backward pass claims are informal, and not substantiated with empirical evaluations.\n\nThe results lack standard deviations. Not all results are converged. The results show weak performance that is not typical of these networks and datasets. \n\nThe results only show simple image benchmarks of CIFAR and TinyImageNet. This is sufficient to describe the method in these simple image regimes, but does not tell us about more realistic images or datasets.", "audience_yn": "Yes", "audience_explanation": "Any training improvements are always welcome to the community.", "requested_changes": "- The intuition of fig 3 is sensible, but how much memory, runtime, or utilisation do the different passes actually take in real-world hardware and software (ie. jax/torch)? It’s not obvious that GPU’s in real world behaves as nicely as depicted in fig3. This should be studied empirically.\n- Furthermore, “merging” the two passes into one is described quite loosely. What does this merging actually do, and what properties it has? Can you formalise this more concretely?\n- Alg 1 shows that the method has three phases. Is this necessary? Why not have a single mix-phase that linearly interpolates between the SL/SSL modes? Did you consider using a cosine schedule for this?\n- The results need standard deviations over repeated training runs\n- The results show oddly weak results. One should easily get around 95% on CIFAR10 and around 80% on CIFAR100 using common ResNet architectures. The results need to achieve realistic, competetive levels to be meaningful.\n- Not all results seem to have converged. All results should show converged runs.\n- I understand that running larger experiments are costly. However, full ImageNet results have been a standard part of deep learning benchmarks for over a decade, and without them it’s difficult to assess how well the method performs in more realistic data regimes or in more realistic image sizes. On the other hand, if the paper claims to only operate in low data regimes, it should also then explore some more realistic ones that CIFAR and TIN.\n- In the mixup setting we interpolate between a labelled image (eg. of a cat) and unlabelled image (eg. of a dog). The prediction task is oddly to only predict a cat, and NOT to predict dog (since it was unlabelled). It then seems that the mix-phase trains only a partially meaningful objective; and the final SL phase needs to then fix this issue. Can you comment on this?\n- In modern DL we would not train from scratch, but instead start from pretrained CIFAR or ImageNet checkpoints. It would strenghten the paper to include this type of finetuning experiments.", "broader_impact_concerns": "No issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jCbMkg6jMj", "reviewer": "Reviewer_Coby", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces \"MixTraining,\" a novel strategy for the two-phase training pipeline of self-supervised learning (SSL) followed by supervised learning (SL), or fine-tuning. The method is presented in the context of reconstruction-based SSL objectives. MixTraining introduces an intermediate phase, where the SSL and SL objectives are optimized *jointly* as a weighted sum. The authors provide an empirical validation of their approach, demonstrating its potential for improved accuracy and reduced training time on downstream tasks.\n\n**Strength:**\nThe primary strength of this paper is its simple yet effective proposal of a weighted, joint optimization phase. The reported improvements in accuracy are noteworthy and suggest a promising potential for better integration of reconstruction based self-supervised and supervised learning workflow.\n\n**Weakness:**\nI believe the paper requires further strengthening by addressing a few points, primarily concerning the alignment between the stated motivation and the experimental validation. Critically, this appears to lead to partially missing empirical support for the claims made. Yet, I believe it also raises opportunities to enhance quality and clarity of the manuscript. I will outline these points in the following:\n\nThe authors formulate the **Scope and Applicability** of their work as: *“It is important to clarify that MixTraining does not aim to replace the existing pretraining-finetuning paradigm when high-quality pretrained models are readily available. Instead, our framework targets scenarios where practitioners must train models from scratch due to data privacy constraints, domain-specific requirements where generic pretrained models prove inadequate, or the absence of suitable pretrained models. In these settings, MixTraining provides a more efficient alternative to the standard SSL+SL pipeline, achieving superior performance while reducing training latency.”*  — While I find this to be a relevant motivation and problem formulation, the respective empiric investigation and results are targeting standard benchmarks such as TinyImageNet. It would be interesting to see the method evaluated in a scenario that mirrors stated scope more closely. I fear the authors choice of validation weakens some of the central claims of this work.\n\n**(1)** Regarding the **\"accuracy gains,\"** the use of mixup data-augmentation in MixTraining is a neat mechanism. For the sake of a clear comparison, it would be helpful to clarify whether mixup was also applied to the SL and SSL+SL baselines?\n\nAlso, I would appreciate a bit more or deeper intuition regarding the mechanism behind the accuracy gains. The authors state that: *“Since the self-supervised learning phase aims at learning general representations and the following supervised learning phase aims at learning task-specific information, intuitively, these two phases optimize the model in different directions.”*  and “[…] *creating a smooth transition that better preserves learned representations while adapting to downstream tasks*”. -- I believe the term \"different direction\" needs to be defined more clearly here. Assuming it refers to the gradient direction, the positive cosine similarity shown in Figure 3 is an interesting intuition. Yet, it might be beneficial to discuss whether a negative cosine similarity between the “different directions” is possible, and if this intuition holds true across different layers of the model, e.g. layers at different depth. My initial thought was that features learned for a reconstruction loss would be quite distinct from those learned for a discriminative task. Can those gradients be aligned? Or is it the diversity of the joint training objective that yields benefit here? To me, it is not obvious that features to reconstruct data are aligned to those that allow to discriminate data. After all this is why often a subsequent finetuning stage is necessary for reconstruction-based SSL methods.\n\n**(2)** Concerning the **\"computational gains,\"** the authors note that: *“the merge of forward/backward passes over the backbone model allows us to reduce computation compared to the synchronous setting substantially.”* It would be helpful to clarify if this implies that MixTraining requires fewer gradient steps to converge? I believe readers would appreciate a respective ablation study. In relation to the improved “GPU utilization” shown in Figure 3, I wonder if this metric might be influenced by the smaller scale of the datasets used. Using lower resolution data could lead to utilization below 100% on modern hardware. As I understand the potential for computational efficiency gain is partly tied to under-exploited parallelism possibility on the hardware, it would be valuable to understand if the potential for increased parallelism does translate to practical benefits in training time in scenarios where GPU utilization is already saturated.\n\n**(3)** Regarding **\"the mixed dataset,\"** a mixup-like mechanism is proposed for combining samples for the supervised and self-supervised objectives. Since the current dataset has labels for all samples, the unique aspects of this mixing strategy are actually not explored. An experiment where a portion of the data is unlabeled would be the more natural approach to demonstrate the effectiveness of this mechanism.\n\nTo further elaborate on the connection between motivation and implementation, I have a couple of additional thoughts:\n\n- A key insight of the paper is that \"the rigid separation between SSL and SL phases in conventional pipelines is neither necessary nor optimal.\" However, the proposed MixTraining is still a rigid distinct phase. It would be interesting to investigate on a more seamless transition. For example, the proposed framework readily allows for a continuously annealing of the $\\alpha$ weight between the SSL and SL losses. The authors suggest that a smooth transition avoids instability, hence a more gradual approach should offer even greater benefits in this regard.\n- The main direction for future work is stated as: *“An important next step is to evaluate MixTraining in larger-scale settings, using more powerful models such as ViT-L/16 (Dosovitskiy et al., 2020; Zhai et al., 2022) and larger datasets such as ImageNet-21K (Ridnik et al., 2021).”*  — While scaling data and model is typically considered a valuable research direction, it seems to be in particularly stark contrast with the initial motivation.\n\nFinally, a few minor suggestions for clarity:\n\n- The paper's focus on reconstruction-based SSL, such as masked autoencoders, could be stated more explicitly early on. While MixTraining is presented in the context of general self-supervised learning, the experiments are specific to reconstruction tasks. Clarifying this focus would help manage reader expectations. If the authors envision MixTraining being applicable to other SSL paradigms like contrastive learning, including some initial experiments or a discussion on this would be a valuable addition.\n- When introducing the procedure for mixing SSL and SL datasets, the phrase: *“In the simple case where self-supervised learning and supervised learning use the same feature representation (i.e., xi ),[…]”*  was a little confusing. I believe \"feature representation\" here refers to the input modality, and rephrasing might prevent any ambiguity.", "claims_and_evidence_yn": "No", "claims_explanation": "While the paper presents promising initial results, the claims are not consistently supported by clear and convincing evidence, primarily due to a significant mismatch between the stated motivation and the experimental validation.\n\n**Claim: Improved Accuracy.** The paper reports accuracy gains, but the evidence is not entirely convincing. It is unclear if the use of mixup data-augmentation, was also applied to the baseline models for a fair comparison. The intuition for the accuracy gains (optimizing in \"different directions\" to create a smoother transition) is not clearly defined or substantiated. The provided data (Figure 3) showing positive cosine similarity between gradients seems to contradict the initial intuition that reconstruction and discrimination tasks are \"distinct,\" which weakens the explanation for why the method works.\n\n**Claim: Computational Gains.** The evidence for improved efficiency is underdeveloped. The paper states that merging passes reduces computation, but it's not clarified if this translates to fewer necessary gradient steps for convergence. Likewise, the evidence for improved GPU utilization is presented on small-scale datasets where utilization might naturally be low. This makes it difficult to assess if these gains would translate to practical, scenarios where GPU resources are already saturated.\n\n**Claim: A novel mechanism for mixed (labeled/unlabeled) datasets.** This claim lacks any direct empirical support. The mixup-like mechanism is a core part of the proposed method, but it is evaluated on a fully labeled dataset (TinyImageNet), meaning the unique aspects and benefits of this approach in a semi-supervised context are not demonstrated at all.", "audience_yn": "Yes", "audience_explanation": "Improving accuracy and computational requirements by a simple and broadly applicable mechanism in the realm of (reconstruction-based) self-supervised learning will be interesting for a wide audience.", "requested_changes": "I believe this paper has a promising core idea, but significant revisions are needed to strengthen claims and improve clarity. The requested changes are outlined below, from major to minor.\n\n**Major Revisions:**\n\n1. **Alignment of Experimental Validation with Motivation:** This is the most critical point. The paper is motivated by scenarios where large-scale pre-trained models are unavailable (due to privacy, domain-specificity, etc.), yet evaluation is conducted on a standard benchmarks. To be clear, my concern lies not with using a benchmark that is often used, but with this benchmark(s) being not fully aligned with the objectives of the presented work. On the one hand this choice partially leaves the power of the SSL objective under-explored, but it also weakens the impact of the results. I recommend experiments that more closely reflect the stated motivation. An ideal experiment would involve a semi-supervised setup where a large portion of the dataset is unlabeled. Such a setting then also demonstrates the utility of the proposed mixed-data mechanism. \n2. **Clarify Fairness of Comparisons:** It needs to be stated explicitly whether the mixup data augmentation was also applied to the SL and SSL+SL baselines. If not, the comparison should be updated to ensure all methods are compared under the same data augmentation scheme.\n\n**Minor Revisions:**\n1. **Bolster Claims of Computational Gains:** The \"reduction in computation\" should be further substantiated, e.g. by providing a direct comparison in an ablation study. Does it translate to fewer gradient steps for convergence? And, does the increased potential for parallelism (GPU utilization) translate to practical scenarios, e.g. larger resolution data?\n2. **Further discussion of the \"Rigid Separation\" of training phases:** The paper claims the \"rigid separation\" of phases is suboptimal, yet MixTraining appears as a distinct, rigid phase. I would encourage the authors to discuss this choice and perhaps explore a more seamless transition, for example, by continuously annealing the loss weight α over time.\n3. **Overall Clarity and Scoping:**\n    - **Scope:** To manage the reader’s expectations, I recommend to state explicitly in the introduction that the paper's experiments are focused on *reconstruction-based* SSL methods. If the method is intended to be more general, a discussion or preliminary experiment on other SSL families (e.g., contrastive, redundancy-reduction, etc.) would be a valuable addition.\n    - **Future Work:** I would urge to reconsider the future work paragraph. The suggestion to scale up data and model seems to contradict the primary motivation of the paper.\n    - **Typo?:** Please rephrase the sentence *“In the simple case where self-supervised learning and supervised learning use the same feature representation (i.e., xi )”* to avoid confusion, as \"feature representation\" appears to refer to the input modality.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)