BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimization
Authors: Gustav Wagner Zakarias
Lars Kai Hansen
Zheng-Hua Tan
Venue: Accepted by TMLR ()

Abstract: Models initialized from self-supervised pretraining may suffer from poor alignment with downstream tasks, limiting the extent to which subsequent fine-tuning can adapt relevant representations acquired during the pretraining phase. To mitigate this, we introduce BiSSL, a novel bilevel training framework that enhances the alignment of self-supervised pretrained models with downstream tasks by explicitly incorporating both the pretext and downstream tasks into a preparatory training stage prior to fine-tuning. BiSSL solves a bilevel optimization problem in which the lower-level adheres to the self-supervised pretext task, while the upper-level encourages the lower-level backbone to align with the downstream objective. The bilevel structure facilitates enhanced information sharing between the tasks, ultimately yielding a backbone model that is more aligned with the downstream task, providing a better initialization for subsequent fine-tuning. We propose a general training algorithm for BiSSL that is compatible with a broad range of pretext and downstream tasks. We demonstrate that our proposed framework significantly improves accuracy on the vast majority of a broad selection of image-domain downstream tasks, and that these gains are consistently retained across a wide range of experimental settings. In addition, exploratory alignment analyses further underpin that BiSSL enhances downstream alignment of pretrained representations.

Published in Transactions on Machine Learning Research (02/2026)
BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine- Tuning via Bilevel Optimization
Gustav Wagner Zakarias Aalborg University
guz aou, dk
Pioneer Centre for AI
Lars Kai Hansen
lkai@ dk
Technical University of Denmark
Pioneer Centre for AI
Zheng-Hua Tan
zt dk
Aalborg Universiby
Pioneer Centre for AI
Reviewed on OpenReview: https: //
// openreview. net/ forum? id= qQpA
Abstract
Models initialized from self-supervised pretraining may suffer from poor alignment with downstream tasks, limiting the extent to which subsequent fine-tuning can adapt relevant representations acquired during the pretraining phase. To mitigate this, we introduce BiSSL, a novel bilevel training framework that enhances the alignment self-supervised pretrained models with downstream tasks by explicitly incorporating both the pretext and downstream lasks into a preparatory training stage prior to fine-tuning. BiSST solves bilevel optimiza- tion problem in which the lower-level adheres to the self-supervised pretext task, while the upper-level encourages the lower-level backbone to align with the downstream objective. The bilevel structure facilitates enhanced information sharing between the tasks, ultimately yielding a backbone model that is more aligned with the downstream task, providing a better initialization for subsequent fine-tuning, We propose general training amgorithm for BiSSL that compatible with a broad range of pretext and downstream tasks. We demonstrate that our proposed framework significantly improves accuracy on the vast majority of broad selection of image-domain downstream tasks, and that these gains are consistently retained across wide range of experimental settings. In addition, exploratory alignment analyses further underpin that BiSSL enhances downstream alignment of pretrained representations.
1 Introduction
Self-supervised learning (SSL) leverages large amounts of heterogeneous unlabeled data to solve pretext tasks that encourage learning of general-purpose representations, which can be subsequently repurposed to solve downstream tasks. This has lead to strong performance across domains such vision (Chen et al.
2020
Bardes et al. 2022 He et al. 2020 Grill et al. 2020 Caron et al.) 2020 2021
He et al.
2022 Oquab
ct al.
2024h, language Devlin ct al. 2019
Lewis et al. 2019: Brown et al. 2020 He et al. 2021 Touvron
el al.
2023). and audio [Schneider et al. 2019 Baevski et. al. 2020 Hsu et al. 2021 Niizumi et al. 2021
Chung Glass 2018 Chung et al. 2019 Yadav et al. 2024). However, downstream tasks often involve
narrower domains that demand more specialized features, which can result in the self-supervised pretrained representations being misaligned with these tasks. Pretrained features that benefit the downstream task may therefore initially appear irrelevant. and thus risk being degraded overwritten during fine-tuning, thereby

---

Published in Transactions on Machine Learning Research (02/2026)
SSL Pipeline With BISSL
Conventional SSL Pipeline
Pretext Pretraining min C (0, Opp Op; D 0,07
Inil 0p, (000) -
Pretext Pretraining min (0, dz; D)) 0,07
0,
Upper-Level (Downstream) min (0t)(00)) do: )o; + L 6o; DC)
Downstream Head Warm-up min (e, Φn; DD) 0b
05.00
Φ
(p(0p)
0p
BISSL
Downstream Fine-Tuning min (0, uo; DO)
(8))
0,00
Lower-Lovel (Pretext) min (0r, Ar(0 0p)
Downstream Fine-Tuning ( DD:
0,8g
Figure 1: BiSSL introduces an intermediate training stage conducted after pretext pretraining. The sym- bols and & represent backbone and task-specific attached head parameters, respectively. When they are transmitted to the respective subsequent training stages, they are used as initializations. The objectives CP, _ represent the respective pretext pretraining and downstream fine-tuning objectives and DP, D the respective unlabeled pretext and labeled ream dalasets. We refer to Section for further details.
constraining the achievable performance (Kumar et al. 2022 Zaiem et al. 2024 Liao et al. 2025 Wang
et al. 2024 Boschini et al.) 2022 Hinton & Salakhutdinov 2006). Enhancing the alignment between the pretext and downstream tasks could address this issue, allowing the pretrained representations to be more optimally utilized during fine-tuning.
We suggest utilizing bilevel optimization (BLO) as novel and effective approach for enhancing this align- ment. BLO entails a main optimization problem constrained by the solution to secondary problem that depends on the main problem's parameters. This hierarchical setup captures and leverages interdependencies between problems and has shown strong potential in deep learning scttings that involve joint optimization of multiple coupled objectives (Zhang et al. 2023a) such parameter pruning (Zhang et al. 2022b) invariant
risk minimization (Arjovsky et al. 2019 Zhang et al. 2023b), meta-learning (Rajeswaran et al. 2019 Finn
et al. 2017), adversarial robustness (Zhang et al. 2021). hyper-parameter optimization (Franceschi et al. 2018), and corcsct selection [Borsos et al. 2020).
In this study, we propose BiSST,, novel bilevel training framework that improves the alignment of self- supervised pretrained backbone models with downstream tasks. BiSSL introduces a preparatory training stage prior to fine-tuning that solves bilevel optimization problem in which the pretext and downstream task objectives serve as the lower- and upper-level objectives, respectively. This bilevel setup facilitates enhanced information sharing, enabling the upper-level to guide the pretext task optimization in refining its representations to better align with the downstream objective, ultimately providing more effective backbone initialization for fine-tume finettunng. Figure depicts how BiSSL integrates into the standard SSL pipeline. We introduce a training algorithm that alternates between optimizing the two objectives within BiSSL, which is agnostic to the choice of pretext and downstream task. Our method makes no additional assumptions about how models are pretrained, allowing compatibility with standard off-the-shelf pretrained backbones. Through experiments using both the SimCLR (Chen et al.) 2020) and Bootstrap Your Own Latent (Grill et al, 2020) pretext. tasks to pretrain ResNet-50 backbones (He et al.) 2016) on the ImageNet-1K dataset (Deng et al. 2009); we demons(rate that BiSSL significantly improves downstream performance across most of 12 downstream image classification datasets, while also significantly improving accuracy within object detection and semantic segmentation. A suite of extended evaluations further shows that the gains remain consistent, and exploratory alignment studies of backbones highlight that BiSSL improves downstream alignment. Code and pretrained model weights are publicly available at
2

---

Published in Transactions on Machine Learning Research (02/2026)
2 Related Work
Bilevel Optimization in Self-Supervised Learning Bilevel optimization (BLO) refers to a constrained optimization problem, where the constraint itself is a solution to another optimization problem that depends on the parameters of the "main" optimization problem. The general BLO problem formulated as
f(, 20 (E)) s.t. (E) argmin (igmiig(EEv)) g(E,
(1)
+
where f and g are referred to as the upper-level and lower-level objectives, respectively. While the lower objective has knowledge of the parameters & from the upper-level objective, the upper-level objective f possesses full information of the lower objective itself through its dependence on the lower-level solution (E). Some works have incorporated bilevel optimization within self-supervised learning. Gupta et al. (002)) suggest formulating the contrastive self-supervised pretext task a bilevel optimization problem, dedicating the upper-level and lower-level objectives for updating the backbone and projection head pa- rameters respectively. Other frameworks such the Local and Global (LoGo) (Zhang et al. 2022a) and Only Self-Supervised Learning (OSSL.) (Boonlia et al.] 2022) utilize auxiliary models, wherein the lower- level objective optimizes the parameters of the auxiliary model, while the upper-level objective is dedicated to training the feature extraction model. MetaMask (Li et al.) 2022b) introduces a meta-learning based approach, where the upper-level learns masks that filter out irrelevant information from inputs that are provided to a lower-level self-supervisod contrastive pretext task. BLO-SAM (Zhang al. 2024) is tailorod towards fine-tuning the segment. anything model (SAM) (Kirillov al.) 2023) by interchangeably alternal- ing between learning (upper-level) prompt embeddings and fine-tuning the (lower-level) segmentation model. The aforementioned frameworks integrate bilevel optimization into either the pretraining or fine-tuning stage exclusively and are tailored lowards specific pretext or downstream lasks. In contrast, our proposed BiSSL employs a BLO problem that comprehensively incorporates both training stages of pretext pretraining and downstream fine-tuning, while not being confined to any specific type of pretext or downstream task.
Priming Pretrained Backbones Prior To Fine- Fine-Tuning Previous works have also demonstrated that downstream performance be enhanced by introducing techniques that modify the backbone between the pretraining and fine-tuning stages. Contrastive Initialization (COIN) (Pan et al. 2022) introduces a supervised contrastive loss to be utilized on backbones pretrained with contrastive SSL techniques. Noisy-
Tune (Wu et al., 2022) perturbs the pretrained backbone with tailored noise before fine-tuning. Speaker- invariant clustering (Spin) (Chang et al. 2023 utilizes speaker disentanglement and vector quantization for improving speech representations for speech signal specific downstream lasks. RIFLE (Li et al. 2020) conducts multiple fine-tuning sessions sequentially, where the attached downstream specific layers are re- initialized in between every session. Unlike BiSSL, these techniques either do not incorporate knowlodge of both the pretext task and downstream task objectives and their relationship, or do only implicitly.
3 Proposed Method
3.1 Notation
We denote the unlabeled pretext dataset DD = [zx |081 and labeled downstream dataset DD = yx vi)/i |DD respectively, where Zk: Xi E R. Let fo : RN + M denote a feature extracting backbone with trainable parameters and hT : R.M + RQ a task specific head with trainable parameters Φ. Given pretext and downstream models h for and h2, foo with Op, Op E RL, we denote the pretext and downstream training objectives (P(Ap, Φp: DP) and (P &D: DDD,, respectively. To simplify notation, we omit the dataset specification from the training objectives, c.g. CP ΦD) = <L"(DD, ΦD:
3.2 Optimization Problem Formulation
In standard SSL, a single backbone with parameters is first trained to minimize the pretext objective CP Φp), yielding a parameter configuration 0* that subsequently is used as an initialization for fine- tuning the downstream objective [ (0, To better align the self-supervised pretrained initialization
3

---

Published in Transactions on Machine Learning Research (02/2026)
with the downstream task, employ BiSSL preparatory training stage fine-tuning that incorporates the pretext and downstream task objectives into joint bilevel optimization problem. BiSST, uses two distinct sets of backbone parameters 0p and DD corresponding to the pretext and downstream tasks, that are treated as separate but strongly correlated parameter vectors. The bilevel optimization problem formulated as
min _ (0' (0p). 2D (00. 0D.0D
(2)
s.t. p (0p) E argmin min L (0p, 0p) + Ar(0p,0p),
(3)
&p
with being some convex regularization objective weighted by A R+ enforcing similarity between 0p and Op. The upper-level training objective in (2) is tasked with minimizing the downstream task objective _ and the lower-level objective in (3) aims to minimize the pretext task objective L while also ensuring its backbone remains similar to the upper-level backbone. As seen in the left. term of (2), the backbone parameters (0 are substituted into the downstream training objective, resembling how backbones conventionally are transferred after self-supervised pretraining. The lower-level's dependence on the regularization objective enables the upper-level to observe how changes in its own backbone parameters Op influence the lower- level solution (0))) thereby allowing the upper-level to indirectly guide the prctext task optimization toward lower-level backbone that better complies with the downstream task. The second term of (2) aids
convergence of the upper-level objective during training by acting as regularizer that constrains in to itself also align with the downstream task. Without this term, 0D would function only as an indirect controller of the lower-level, potentially leading to degenerate solutions.
3.3 Upper-level Derivative
Given the upper-level objective :DD (P00))))))) (o from 2 its derivative with respect to OD is given by
dF
(0)) T
VoL (B. + (0C
d0
d0p
Due to the dependence of the lover-level solution on the upper-level parameters, the first term of 4 includes the Jacobian of the implicit function (0)D)) referred to as the implicit Jacobian (IJ). To simplify notation, we let Veh(E)le=u == Vgh(tb) when clear from context which variables are differentiated with respect to. Following an approach similar to Rajeswaran el. al. (2019), with details on the derivations and underlying assumptions outlined in Section A.1 the in can be explicitly expressed as
d0p) d0D
2 0000 r(p 0D) Vor (00(0)(00)),)))))00000
(5)
A common convex regularization objoctive, which will also be the choice in the subscquent experiments of this work, is r(E, +b = 1 || 11 Using this regularization objective simplifies (5) down to
d0p) T V (0)))
d0p
where I the L x L-dimensional identity matrix. Hence the upper-level derivative can be expressed
de 1 co
dF 1
Tz Vo, Co. (0, (o,
(7)
As shown in 7 the pretext objective is explicitly incorporated into the upper-level gradient, causing the upper-level updates to be influenced by how its changes affect the lower-level.
4

---

Published in Transactions on Machine Learning Research (02/2026)
The inverse Hessian-vector product in is computationalyy infeasible to calculate directly, so it is approxi- mated using the conjugate gradient (CG) method Nazaaetth 2009; Shewchuk, 1994). We employ layer-wise implementation of the CG method based on that of [ajeswaran et al. (2019) and refer to their work for more details on applying CG in a deep learning setup with BLO. While CG has proven to be an effective approach for approximating the inverse Hessian-vector products in previous works &Pedregosa) 2016 Zhang et al., 2021 Rajeswaran et al., 2019), still introducos some computation and memory overhead due to its need for evaluations of multiple Hessian vector products. Future work could explore alternative established
methods for upper-level gradient approximation (Zhang et al. 2023a Choe et al. 2023 Liu et al. 2024 that
offer greater computational efficiency. We report the computation times of BiSSL alongside along pretraining
and fine-tuning and discuss the Hessian vector product calculation overhead in Section C.1
With an explicit expression of the IJ in (6, we can interpret the impact of the scaling factor À from (3) and 77 When À is very large, the dependence of the lower-level objective on the upper-level parameters 0 correspondingly very large. This effectively drives the lower-level backbone parameters toward the trivial solution (p(0p) Op. Meanwhile, the IJ in (6] approximately equals Tt., thereby diminishing the influence of the lower-level objective the upper-level gradient in (7). This roughly makes the task of the upper-level equivalent to conventional fine-tuning. Conversely, if À is very small, the lower-level objective in 3 effectively defaults to conventional pretext task training. Additionally, the implicit Jacobian in (6) would consist of numerically tiny entries, making the optimization the first term in the upper-level objective in
2 equivalent to linear probing the downstream head on the frozen pretext backbone (0)). Thus, an intermediate value of À is expected to yield an optimization that balances the contribution from both levels.
3.4 Training Algorithm and Pipeline
Algorithm BiSSL Training Algorithm
Backbone and head initializations 0, &p. Φn. Training objectives C., L. Optimizers opt p Regularization Weight À IR.. Number of training stage alternations T E N with upper and lower-level iterations No, Nu E N.
Input:
2: Initialize 0p + O and 0D + O.
3: for t= 1, do for n = 1,..., , Nr. do Compute gøp V (0p. ()),z)))=por
b. Lower-level
Compute ge, VoL (0, op)lo=0 X(0p -pD)
7:
Updale op + opl p Bdp) and Op opl p ).
8:
for = I....,,,,,, No do
b Upper-level
9:
Compute gøp = V (0p, (0, + V (0D, $10==
Compute v = voL (0, @
10:
11:
Approximate s | V (0, 1
11]
Use CG to approximate
12:
Compute g0, ViJ - V vo O 0p 0-0
0r.
13:
Update ΦD + p {d, and 0D + opt (0 ( ge,)
14:
Return: Backbone Parameters 0p.
Our proposed training algorithm itcratively alternates between solving the lower-level (3) and upper-level 2 optimization problems in BiSSL. The lower-level training optimizes the pretext. task objective, while
additionally including the gradient of the regularization term r, complying with (3) The upper-level stage computes gradients with respect to the backbone, corresponding to the left-hand term in (7), which are approximated using the CG method. Algorithm details the proposed training procedure. We remark that allowing No 1 deviates from conventional BLO training setups. However, document. in Sec-
5

---

Published in Transactions on Machine Learning Research (02/2026)
tion C.3 this modification proved more efficient and beneficial for downstream performance. The algorithm is applicable to any common pretext and downstream tasks.
Figure T illustrates the proposed training pipeline with BiSSL alongside the conventional SSL pipeline. Pretext pretraining on the unlabeled dataset D' provides initializations of and Φp, after which the downstream head ΦD is fitted on top of the frozen backbone 0 using the downstream dataset DD. BiSST. training is then conducted as outlined in Algorithm I as the initialization for final supervised fine-tuning the downstream task. We emphasize that the objective L in the upper-level of BiSSL is identical to the downstream fine-tuning objective, and similarly that the objective L in the lower-level is identical to the pretext pretraining objective.
yielding an updated backbone (p)()), which serves
Sontion A.1 notes that (0p) must satisfy the stationary condition Vo (L (0, + Xr (=)))l==0
O to justity the explicit expression of the IJ in G). This requirement naturally aligus with using a backbone that has already converged from sclf-supervised pretraining, as such an initialization roughly satisfics the sta- tionary condition at the outset and also makes the smoothness assumption on L., mentioned in Section AA.1| more reasonable. A similar consideration applies to @D. since randomly initialized downstream head can lead to rapid early updates to OD during fine-tuning, potentially violating the stationary assumption due to its coupling with 0p through the rogularization objective r.
4 Experiments and Results
4.1 Datasets
The ImageNet-Ile 1K Deng et al. 2009) dataset, devoid of labels, is used for self-supervised pretraining through- out the main experiments. For downstream fine-tuning and evaluation, we leverage a varied set of natural image classification datasets that encompass a wide array of tasks, including general image classification, fine-grained recognition across species and objects, scene understanding, and texture categorization. The datasets include Food 101 (Bossard et al.) 2014),, CIFAR1O (Krizhevsky) (0012), CIFAR100 (Krizhevsky 2012), Caltoch-UCSD Birds-200-2011 (CUB200) (Wah et al.) 2011),
SUN397 scene datasct (iia et al, 2013), PASCAL VOC 2007 (Evering- 2014h. Oxford-IIIT Pets (Parkhi et al.
2010 StanfordCars (Yang et al. 2015), FGVC Aireralt (Maji et al.,
ham et al., 2010). Describable Textures Dataset (DTD) (Cimpoi et al.
2012), Caltech-101 (Li et al., 2022a) and Oxford 102 Flowers (Nilsback & Zisserman 2008). Additionally, we use the combined VOCO7+12 (Everingham et al. 2010) dataset for both object detection and semantic segmentation, and the fine-grained partition of Cityscapes (Cordts et al. 2016) for semantic segmentation. All downstream datasets are split into training, validation, and test partitions, with details on how these
assignments are made in Section B.1
4.2
Implementation Details
4.2.1
Baseline Setup
We first outline each stage of the baseline setup following the standard SSL pipeline on the left of Figure 1
Pretext Task Training We conduct experiments using two different types of pretext tasks: Sim- CLR (Chen et al. 2020) and Bootstrap Your Own Latent (BYOL) (Grill et al., 2020). For SimCLR. we use a temperature of 0.5 and a ResNet-50 (He et al. 2016) backbone. On top of the backbone, a projection head is appliod, consisting of two fully connected layers with batch normalization (Toffe Szegedy 2015) and ReLU (Agarap) 2018) followed by single linear layer. Each layer consists of 256 neurons. For BYOL, we use constant target decay rate of 0.9995, with the backbone and projection head architectures identical to those of SimCLR. The additional BYOL-specific prediction head uses an architecture identical to the projection head. The remaining details in this paragraph apply to both SimCLR and BYOL.
The image augmentation scheme follows the approach used in Bardes et al. 2022)). with minor modifications: The image size is set to 96 x 96 instcad of 224 x 224, and the minimal ratio of the random crop is adjusted accordingly to 0.5 instead of 0.08. The implementation of the LARS optimizer YYou et al. (017]) from Bardes et al. 2022 is employed, with "trust" coefficient of 0.001, weight decay of 10-6 and momentum of
6

---

Published in Transactions on Machine Learning Research (02/2026)
0.9. The learning rate increases linearly during the first 10 epochs, reaching peak base learning rate of 4.8, followed by cosine decay with no restarts (Loshchilov & Hutter 2017) for the remaining epochs. A batch size of 1024 is used and, unless otherwise specified, pretraining is conducted for 500 epochs.
Fine-Tuning on the Downstream Task For downstream fine-tuning, single linear layer is attached to the output of the pretrained backbone. The training procedure utilizes the cross-entropy loss, the SGD optimizer with a momentum of 0.9, and a cosine decaying learning rate scheduler without restarts (Loshchilov & Hutter 2017). Fine-tuming is conducted for 400 epochs with a batch size of 256. An augmentation scheme similar to the fine-tuning augmentation scheme in [ardes et al. (2022) is employed during training, where images are center cropped and resized to 96 96 pixels with minimal crop ratio of 0.5, followed by random horizontal flips. Learning rates and weight decays for each downstream datasets are selected via random iyper-parameter grid search. After hyperparancters are determined, we train 10 models with different random seeds, and report the mean and standard deviation of the top-1 test accuracies. Details on the hyperparameter search and evaluation protocol are provided in Section B.2
4.2.2 BiSSL Setup
This section detail each stage of the training pipeline involving BiSSL, as illustrated on the right of Figure
Pretext Task Training The backbone and projection head Φp are initializod by self-supervised pre- training using setup identical to the baseline pretexi task training setup detailed in Section 4.2.1 This enables reusing the pretrained backbones from the baseline experiments.
Downstream Head Warm-up The training setup for the downstream head warm-up closely mirrors the fine-tuning setup of Section 4.2.1 The main difference is that only the linear downstream head is fitted on top of the now frozen backbone obtained from the pretext warm-up. Learning rates and weight decays are initially selected based on those listed in Table with adjustments made as needed when preliminary testing indicated a potential for improved convergence. These values are provided in Table G of Appendix [ The authors recognize that more optimal peper-parameter values may exist and leave further exploration of this for future refinement. The downstream head warm-up up is run for 20 epochs with a constant learning rate.
Lower-level of BiSSL The training configuration for the lower-level primarily follows the setup described for pretext pretraining in Section 1.2.1 with the modifications outlined here. As specified in (3) the lower- level loss function is the sum of the pretext task objective L (e.g. the NT-Xent loss for CLR (Chen 2110p 0D112. aligning with Algorithm 1 Based
et al. 2020g) and the regularization term r(0p,, Op) early experiments, the regularization weight À = 0.001 was selected, as it appeared to strike well-balanced compromise between the convergence rates of both the lower- and upper-level objectives. The lower-level is trained for the equivalent of approximately 8 conventional pretraining epochs, with further details provided in the composite configuration details paragraph. Accordingly the linear learning rate scheduler warm-up is adjusted to range over 10- Nr training steps, where N₂ is the number of lower-level iterations in Algorithm
Upper-level of BiSSL The upper-level training stage largely mirrors the downstream training setup described in Section 4.2.1 and we again only list the differences. The weight decays and base learning rates are set to match those obtained from the downstream head warm-up detailed in Table 6 of Appendix [
To approximate the upper-level gradient in 7 the conjugate gradient method Nazareth 1994h is employed, with implementation details covered in Section B.3
2009 Shewchuk
Composite Configuration Details of BiSSL As outlined in Algorithm 1 both lower- and upper-level backbone parameters 0p and 0D are initialized with the backbone parameters obtained during the pretext warm-up, and the training procedure alternates between solving the lower- and upper-level optimization problems. In this experimental setup, the lower-level performs Nr = 20 gradient steps before alternating to the upper-level, which then conducts Ny = gradient steps. A total of = 500 training stage alternations are executed. With the ImageNet dataset and the current batch size of 1024, there are a total of 1251 training batches without replacement. Consequently, the T = 500 training stage alternations correspond
7

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Comparison of classification accuracies obtained through fine-tuning without and with preparatory BiSSL training. Statistically significant improvements over the counterpart are highlighted in bold.
Food
CIFARIO
CIFAR100
CUB200
SUN397
Caus
Airezafts
VOCU7
DTD
Pets
Caltech101
Flowers
SimCL::
7.
9621102
19.5
03
193022
77.903
220110
710101
735=
= 82.6 033
BISSLFPT 76. = 0.1
96.3
0.1
79.8
5..7+022
50.9
0.1
77.0003
56.9
71.
0.1
64.2404
78340.3
87.2
84.1+0.2
Ang DaT
+1.2
40.2
40.4
46.1
+1.6
+3.8
BYOL:
75.3 1 0.3
0.1
80.1 +
0.2
227004
47.9
03
76.9
0.2
516
69.3
0.1
59.5
77.9
86.8
04
82.1 0.5
BISL+PT
76.2 0.1
1..
80.4
0.1
57.1
1.3
49.6
1.1.
77.5
2.5..2
57.2
272.1.0..
70.8
0.1
61.7
80.1
0.2
85.4
Aug DAy
+0.9
0.0
+0.3
+14
+1.7
-06
-5.6
+1.5
+22
+22
+0.6
+3.3
to roughly 8 conventional pretext epochs, negligible additional training load compared to the 500 pretext epochs used for the full pretraining process. Section B.4 outlines further details how data batches are handled during training. Lastly, gradient normalization is employed on gradients exceeding fn-norms of 10.
Fine-Tuning on the Downstream Task Subsequent downstream fine-tuning is conducted in a manner identical to that described the (Fine-Tuning on the Downstream Task" paragraph of section 4.2.1 Table in Appendix lists the considered optimal syper-paramend configurations for each dataset.
4.3 Downstream Task Performance
Table reports means and standard deviations of top-1 classification accuracies (or 11-point mAP the VOCO7 dataset), comparing results obtained from direct fine-tuning the pretrained backbone with those achieved by applying BiSSL prior to fine-tuning. We mark statistically significantly different accuracies with bold-font, with details on how the statistical significance lests are execuled in Section [B.6] Table [] of Section outlines the corresponding top-5 accuracies. For both pretext tasks, BiSSL significantly improves downstream performance on 11 out of 12 datasets, with no single result showing a decline in performance. Moreover, accuracies from BiSSL exhibit lower similar variances than the baseline. Smaller gains tasks like CIFAR10, VOCO7, and Caltech101 may reflect their coarse-grained nature and input distributions being more similar to the pretext task, whereas fine-grained tasks with more distinct input distributions that require more specialized representations like CUB200, Aircrafts, and Pets, exhibit greater gains from BiSSL.
4.3.1 Object Detection and Semantic Segmentation
We further assess performance the context object detection and semantic segmentation. Similar to the setup of [Bardes et al.] (2022); ; He et al. (2020) we
Table Comparison of APso scores for object detection and mIoU scores for semantic segmen- tation obtained by fine-tuning without and with preparatory BiSSL training. Scores statistically significantly higher from their counterparts are marked in bold.
for object detection Faster-RCNN style archilecture with a ResNet-50 C4 backbone Ren et al. 2015 on the VOC07+12 Everingham et al. 2010 dataset and report the standard AP50 metric. For semantic segmentation tasks, we follow the setup He et al. (2020); Grill et al.
Detection (APs)) VOC07+12
Semantic Seg. (mloU)
(2020) and use FCN-16 (Long al. 2015b based strue- ture top of the pretrained ResNet-50 backbone and evaluate the VOC07+12 [Everingham et al. 2010) and CityScapes (Cordts et al., 2016) datasets, reporting the mean intersection over union (mloU) scores. We refer to Section B.7 for complete implementation details. The re- sults Table support the previous conclusions, indicat- ing that BiSSL overall significantly improves downstream performance.
VOC07+12
CityScapes
SimCLR: FT
53.1 1.1100 0.3
72.7 - 0.6
47.8 0..
FISSL+FT
544 0.4
74.4 0.6
5.4
Aug Duf
+1.
+1.7
+2.9
FT
52.6
71.7 0.6
48.6
BISSL+FT
5.3
73.2
4.4
Aug Duj
+0.9
+1.5
+11
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Comparison of downstream classification accuracies across baseline methods detailed in Section B.9 For each dataset, the highest average accuracy is marked in bold, and the sccond-highest underlined.
Pets
DTD
VOC07
Flowers
CUB200
Random Initialization B.9.1
53.9 = 0.8
0.5
43.7 0.3
43.7 0.5
25.7 0.4
Linear Probing [.9.1
38.4 = 0.2
0.4
6.1
6.1
18.5 0.1
FT
73.5 = 0.7
0.9
7.10
0.3
49.6 0.3
FT (Zero Head Init) B.9.1
75.6 = 0.6
0.5
0.2
0.6
4.
NoisyTune B.9.2
73.4 = 0.3
0.70
0.2
0.2
0.5
Pretraining on Data Mix
B.9.3
+ FT
74.6 = 0.6
0.6
0.2
0.
0.7
Sebeext-Dowsttram Sum
B.9.4
+ FT
76.4
0.3
6.11
83.8
0.5
BiSSL (IJ Discarded)
B.9.5
+ FT
76.2 0.3
6.3
71.7
81.5 0.3
03.1
BiSSL + FT
78.3 + 0.2
6.42
715
84.1 = 0.2
55.7
4.4
Extended Evaluation
In this section, we present a series of experiments to examine how the performance of BiSSL compares with other baselines, and to explore how well BiSSI. adapts to different configurations and experimental settings.
4.4.1 Comparative Baseline Evaluations
We assess BiSSL relative to several other baseline methods in this section. Due to computational constraints, we restrict evaluation to five downstream datasets: Pets, DTD, VOCO7, Flowers, and CUB200. These cover both coarse- and fine-gra greiggra classification tasks across diverse and mostly disjoint visual domains. All base- lines involving self-supervised pretraining use SimCLR as the pretext task, as described in Section 4.2.1 We briefly summarize the conceptual design of each baseline and defer full implementation details to Section B.9
We include baselines consisting of randomly initialized backbone, linear probing, and fine-tuning with gero-initializer head, all described in Section [..9.] Then, we further compare BiSSL to methods that similarly introduce an intermediate stage between pretraining and fine-tuning. These include Noisy'Tune (Wul iet al. 2022). which perturbs backbone paramcters with noise (dctails in Scction B.9.2). continued pretraining on a mixture roamsttremm and pretexl. data (detailed Section B.9.3 and joint minimization a weighted sum of the downstream and pretext objectives (detailed in B.9.4). In addition, we also evaluate a BiSSL variant with the first term of the upper-level objective (2) discarded, eliminating the gradient coupling from the lower to upper-level, which effectively reduces to an alternating optimization scheme (details in [B.95
This variant can itself also be considered an ablation, as it tests whether the increased information sharing imposed by the bilevel structure provides any tangible benefit for the downstream tasks.
Table reports the results for all mentioned baselines, along with the FT and BiSSL+FT results from Table for ease of comparison. As expected, pretrained backbones consistently outperform initializing them randomly, and fine-tuning yields further gains over linear probing. Notably, DTD and VOCO7 exhibit the smallest relative drop in performance under linear probing compared to fine-tuning, suggesting that their tasks benefit more directly from the general-purposs representations captured during SSL pretraining. This could stem from the fact that VOC's labels are relatively coarso-grained, while DTD relics on general texture pailerns rather than fine-grained distinctions, in contrast. to the fine-grained Pets, Flowers, and CUB200 datasets. BiSSL overall achieves the strongest results, ranking first on four of the five datasets and second on the remaining one. In contrast, NoisyTunee fails to improve downstream accuracy this setting, and the remaining baselines provide gains over conventional fine-tuning less consistently, and when improvements do occur, they are generally smaller than those achieved by BISSI.

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Downstream classification performance when using ViT-backbone with MAE as the pretext task. BiSSL improves performance with statistical significance and reduces variance across all five datasets.
MAE
Pets
DTD
VOCO7
Flowers
CUB200
FT
81.3 0.5
61.3 + 0.7
69.5 + 0.3
81.1 + 1.1
59.0 + 0.7
+
82.9 + 0.1
62.9 0.4
72.0 + 0.2
87.3 0.3
65.1 0.4
Aug Diff
+1.6
+1.6
+2.5
+6.2
+6.1
4.4.2 Transformer-Basee Backbone Architecture
Our main experiments establish the effectivencss of BiSSL using ResNet-based architectures. To evalu- ale whether these benefits extend to other architectures, we test BiSSL with the more recent ViT back- bone (Dosovitskiy et al., 2021), trained using the Masked Autoencoder (MAE) pretext task (He et al. 2022) Experimental details are provided in Section B.S] The evaluation uses the same five downstream datasets as in Section 4.4.1 As shown in Table 4 BiSSL delivers consistent and statistically significant improvements
across all datasets and reduces variance in downstream accuracy compared to standard fine-tuning.
4.4.3 Varying the Pretraining Duration
To further assess the robustness of BiSSL, we vary the duration of self-supervised pretraining. Due computational resource constraints, we adopt smaller-scale version of the SimCLR sctup, using a ResNet-18 backbone and the unlabeled partition of the smaller-scale STL10 (Coates et al., 2011) dataset for pretraining, Reusing the parameters from Section 4.2.2 BiSSL training corresponds to 100 conventional pretext epochs using the STL10 dataset. To ensure fair comparison, the pretext-only baselines are accordingly pretrained for 100 additional epochs. The rest of the setup remains identical to as described in Section 4.2. We evaluate on the flowers datasel, where BiSSL previously showed significant gains. Figure depicts the final top-1 test accuracies achieved by separate models pretrained for varying durations. BiSSL consistently outperforms the baseline once sufficient pretraining is reached, consistent with the remarks in Section 3.4
Additional Results We provide additional results Appendix Section C.1 reports training durations for self-supervised pretraining, BiSSL and fino-tuning. Section C.3 presents ablations on BiSSL by lowering the number of upper-level iterations to N2 1, underlining the benefits of increasing this value in practice. A sensitivity analysis of À is provided in Section C.4 which reports how classification accuracy is affected by varying this parameter. Section C.5 presents ImageNet fine-tuning results, and Section C.6 demonstrates that BiSSL also achieves significant performance improvements using the official SinCI-R-pretrainee weights from Chen et al. (2020) with the original 224 X 244 image resolution.
5 Exploratory Backbone Alignment Analyses
To assess the extent of which BiSSL improves backbone alignment for fine-tuning, we compare downstream alignment of self-supervised with BiSST-ttrained backbones using quantitative and qualitative approaches.
5.1 Measuring Backbone Similarity
We first aim to quantitatively measure the backbone alignment. This is approached by calculating the cosine similarity (CS), firstly between backbone parameters and secondly between backbone oulput features. Specifically, we compare the CS between SimCLR- MennCLLrretrrieed backbones and their fine-tuned versions, with the CS of BiSSI-trained backbones and their fino-tuned counterparts. CS values are computed for models trained on each of the 12 datasets listed in Table D with the average CS across these datasets reported in Figure Further experimental setup details are provided in Section B.10 where we also provide individual
10

---

Published in Transactions on Machine Learning Research (02/2026)
78
1.0
0.94
BISSL+FT
0.91
77
0.8
0.76
78
Avg. Cosine Similarity
Top-1 Accuracy (%)
75
74
0.44
04
73
72
0.2
71
0.0 Parameters
Features
70
200
300
600
700
800
Total Number of Pretext Epochs
Figure 3: Paramcter-wise and fcaturo-wise sine similarities between SinCLRR-peetraing and BiSSL-trained backbones with their respec- tive fine-tuned counterparts. Results are av- eraged over the 12 downstream classification datasets listed in Table [ and values for each individual dataset are listed in Table g of Sec- tion B. 10.1 increases downstream alignment of backbones.
Figure 2: Top-1 test accuracies on the flowers dataset for models pretrained with SimCLR across different train- ing durations. BiSSL consistently achieves significantly higher accuracies when the pretraining duration is suffi- ciently high.
The results indicate that BiSSL
dataset CS values in Table g The results imply that BiSSL backbones are consistently more aligned with their fine-tuned versions than standard SSI-pretrained backbones, implying that BiSSL-trained backbones impose smaller parameter and feature shift during fine-tuning.
5.2 Visual Inspection of Latent Features
To better assess whether BiSSL nudges the latent features toward being more semantically meaningful for downstream tasks, qualitatively inspect latent spaces using the t-Distributed Stochastic Neighbor Em- compare features from SimCLR--petrainee backbones to those derived from BiSSL-trained lower-level backbones. Further implementation details are provided in Section 10.2 Figure 4 illustrates the results on the flowers dataset, indicating that BiSST, im- proves downstream feature alignment. Additional plots on a selection of downstream datasets in Section C.7 reinforce this finding, demonstrating that this trend consistently persists.
bedding (t-SNE) (Cieslak et al. 2020) method. Specifically,
6 Conclusion
This study introduces BiSSL, a novel training framework that integrates the self-supervised pretext and downstream fine-tuning objectives into a unified bilevel optimization problem. The bilevel structure enables the downstream objective to guide the pretext task optimization towards refining its representations to better align with the downstream task. We present task-agnostic training algorithm and pipeline that incorporates BiSSL as preparatory stage prior to fine-tuning, Experiments across multiple pretext and downstream tasks demonstrate that BiSSL consistently enhances fino-tuning performance, while additional alignment analyses indicate that il produces pretrained representations that belter conform lo downstream objectives. Overall, BiSSL demonstrates that explicitly addressing the misalignment between self-supervised pretrained backbones and downstream tasks prior to fine-tuning can yield substantial accuracy gains. This highlights the potential of bilevel optimization in this setting and motivates future research on training algorithms that explicitly optimize such alignment.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 4: Visualizations of backbone output features on the Flowers dataset for the self-supervise pretrained backbone (left) and the MISSL-traines backbone (right). Colors indicate different classes. Further details are provided in Section B.10.2 and analogous visualizations for other datasets are found in Section C.7 The BiSSL backbone output features exhibit improved discriminative alignment with the downstream classes.
Broader Impact Statement
Our work focuses improving downstream performance of self-supervised pretrained models without intro- ducing new socictal or ethical risks beyond those already associated with standard self-supervised learning and fine-tuning.
Acknowledgments
This project is supported by the Pioneer Centre for Artificial Intelligence, Denmark - We acknowledge the Danish e-Infrastructure Consortium (DeiC) for awarding this project access to the LUMI supercompuler. Lastly, the authors would like thank Sijia Liu and Yihua Zhang (Michigan State University) for provid- ing valuable foedback in discussion, which helped to refine and solidify perspective the topic of integrating bilevel optimization in deep learning.
References
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.00 08377, 2018.
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. ArXiu, 0///19977009993 2019.
Alexei Bacvski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli wav2vec 2.0:
framework for self-supervised learning of speech representations. Advonces neural information processing systems, 33: 12449-12460, 2020.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: cooiaace-narraaccccovvaaeee regularization for self-supervised learning. In International Conference on Learning Represenlations, 2022.
Harshita Boonlia, Tammoy Dam, Md Meftahul Ferdaus, Sreenatha G Anavatti, and Ankan Mullick. Improv- ing self-supervised learning for out-of-distribation task via auxiliary classifier. In 2022 IEEE International Conference Image Processing (ICIP), pp. 3036 3040. IEEE, 2022.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33:14879-1488 2020.
https: Aww. aicentre.dt
12

---

Published in Transactions on Machine Learning Research (02/2026)
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In European Conference Computer Vision, pp. 692 709. Springer, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1577-1901 Curran Associates, Inc., 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEFE/CVE International Conference Computer Vision (ICCV), pp. 9630 9630-9640, 2021. doi: N0.1109/ICCC48990 20200000009.....005051.
Heng-Jui Chang, Alexander H. Liu, and James Glass. Self-supervised Fine-tuning for Improved Content Representations by Opeaker-invarian Clustering, In INTERSPEECH 2023, pp. 2023. doi: 10.21437/ / 200p:e338p....2228887...
ISCA, Angust
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020.
Sang Keun Choe, Willie Neiswanger, Pengtao Xie, and Eric Xing. Betty: An automatic differentiation library for multilevel optimization. In The Eleventh International Conference on Learning Representations, 2023.
URL http://ppneevieeeeeeee/rrrrrrrre
Yu-An Chung and James Glass. Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Em- beddings from Speech. In Proc. Interspeech 2018, pp. 811-815, 2018. doi: 10.21437 111443//lnterspecch2011. 2011-2221.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An Unsupervised Autoregressive Model for Speech Representation Learning. In Proc. Interspeech 2019, pp. 146-150, 2019. doi: 10.21437/Interspeee 2479-1473.
Matthew C. Cieslak, Ann M. Castelfranco, Vittoria Roncalli, Petra H. Lenz, and Daniel K. Hartline. t- distributed stochastic neighbor embedding (t-sne): A tool for eco-physiological transcriptomic analysis. Marine Genomics, 51:100723, 2020. ISSN 1874-7787. doi: https:/ htlps://doi.org///////// 10. mtw.:/...a.....219..
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition. (CVPR), 2014.
Adam Coatos, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised fcature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215 223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.
Marius Cordts, Mohamed Omran, Scbastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proc. of the IEF. Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Mathicu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. How to compute hessian-vector products? In TCLR Blogposts 2024, 2024. URL https://iclr-bllogosss wwt.eri.ttttt.oooeoeeoe 1o/2024/blog/
bench-hvp/
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: M0.119/CVVR.2209.00
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics, 2019.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. De- caf: A deep convolutional activation feature for generic visual recognition. In Eric P. Xing and Tony Jebara (eds.): Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 647-655, Bejing, China, 22-24 Jun 2014. PMLR.
A.L. Dontchev and R.T. Rockafellar. Implicil Punctions and Solution Moppings: A View from Variational Analysis. Springer Series in Operations Research and Financial Engineering. Springer New York, 2014.
ISBN 9781493910373.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Inl. J. Compul. Vis., 88(2):303 338, 2010.
Chen Fan, Parikshit Ram, and Sijia Liu. Sign-MAML: Efficient model-agnostic meta-learning by signSGD. In Fifth Workshop on Learning at the Conference on Neural Information Processing Systems, 2021.
Chelsea Finn, P. Abbeel, and Sergey Levine. Model-agnostie meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel pro- gramming for hyperparameter optimization and meta-learning. In International conference on machine learning, pp. 1568 1577. PMLR, 2018.
Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. In Neural Information Processing Systems, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena. Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 21284, 2020.
Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning. ArXie, 221222212.1 2022.
Jie Hao, Xiaochuan Gong, and Mingrui Liu. Bilevel optimization under unbounded smoothness: A new algorithm and convergence analysis. ArXio, abs/2440.09553 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervisod visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 97389,977,1188 2020.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVI /CV conference compuler vision and pattern recognition, pp. 16000- 16009,, 2022.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhannn enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11916-11925, 2021. doi: LLL.///CCCV492222221...000
G. F. Hinton and R. R. Salakhutdinov. Reducing the dimensionality data with neural networks. Science, 313 507, 2006. doi: 10.1126//cience..12270 1127647. URL https:// science..e...c crrienceerrg/eeeeee. arg//di/abs//0.
1126/science 1127647
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Ab- delrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Edward J. Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In JCLR, 2022.
Feihu Huang, Optimal (lessinn/Jcoobaan-iin nonconvex-PP PL bilevel optimization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.); Proceedings of the 11s International Conference on Machine Learning, volume 235 of Proceedingss of Machine Learning Research, pp. 19598-19621. PMLR, 21-27 Jul 2024.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. pmir, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3992 4003, 2023.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Ananya Kunar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang, Fine-tuning can distort pretrained features and underperform dit-of-distribution. In International Conference on Learning Representations, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising soquence-ta-seqqeece pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics, 2019.
Fei-Fei Li, Marco Androcto, Aurclio Ranzato, and Pictro Perona. Caltech 101, Apr 2022a.
Jiangmeng Li, Wenwen Qiang, Yanan Zhang, Wenyi Mo, Changwen Zheng, Bing Su, and Hui Xiong. Meta- mask: Revisiting dimensional confounder for self-supervised learning. Advances in Newral Information Processing Systems, 35:38501-38515, 2022b.
Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhome Xu, and Dejing Dou. Rifle: Backpropagation in depth for deep transfer learning through re-initializing the fully-connected layer. In International Conference Machine Learning, pp. 6010-6019. PMLR, 2020.
Chonghua Liao, Ruobing Xie, Xingwu Sun, Haowen Sun, and Zhanhui Kang. Exploring forgetting in large language model pre-training. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2112-2127, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 1111998-9976--5--10 brg/2025..m acl acl-long. 105/
doi: 10.18653/ /v1
105. URL https:://aclaaatioooog//
15

---

Published in Transactions on Machine Learning Research (02/2026)
Risheng Liu, Zhu Liu, Wei Yao, Shangzhi Zeng, and Jin Zhang. Moreau envelope for nonconvex bi-level optimization: A single-loop and hessian-free solution strategy. ICML, 2024.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431 3440, 2015a.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In 2015 IEEE Conference Computer Vision and Pattern Recognátion (CVPR), pp. 3431-3440, 2015b.
llya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
J. L. Nazareth. Conjugate gradient method. WIREs Computational Statistics, 1(3):348 (0))3333-33,, 2009. doi: https://doi.orr/IU0 wtt../rrr/...0//ii......ooo
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for general-purpont audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN) pp- 1-8. TEEE, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wa Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. DINOv2: Learning robust visual features without supervision. Thansactions Machine Learn- ing Research, 2024. ISSN 2835-8856.
Haolin Pan, Yong Guo, Qinyi Deng, Hao-Fan Yang, Yiqun Chen, and Jian Chen. Improving fine-tuning self-supervised models with contrastive initialization. Neural networks : the official journal of the International Newral Network Sociely, 159:198 207, 2022.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Pattern Recognition, 2012.
Fabian Pedrogosa. Hyperparameter optimization with approximate gradient. In Maria Florina Balcan and Kilian Q. Weinberger (eds.),, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 237-746, New York, New York, USA, 20-22 22 Jun 2016. PMLR.
Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Neurol Information Processing Systems, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Steffen Schneider, Alexci Bacvski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised Pre- Pro-Training for Speech Recognition. In Proc. Interspeech 2019, pp. 3465-3469, 2019. doi: 10.21437 IIterspeechh 2019-1873.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Jonathan R Shewchuk. An introduction to the conjugate gradient method without the agonizing pain. Technical report, Carnegie Mellon University, 1994.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arxie:14083 1556, 2014.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Adoonces in Newral Information Processing Systems, volume 33, pp. 18098 18109. Curran Associates, Inc., 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers distillation through attention. In Marina Meila and Tong Zhang (eds.); Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 18-24 Jul 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, el al. Llama: Open and efficient foundation language models. orXiv preprint XrXi::2302. 139711 2023.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report 1-S-TRR0011-00,, California Institute of Technology, 2011.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. Noisytune: A little noise can help you finetune pretrained language models better. In Annual Meeting of the Association for Computational Linguistics, 2022.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to 20o. In 2010 IEEE Computer Society Conference on Computer Vision and Pallern Recognilion, pp. 3485 3492, 2010. doi: 10.1109/CVPH 2000//CC2..000553090700
Sarthak Yadav, Sergios Theodoridis, Lars Kai Hansen, and Zheng-Hus Tan. Masked autoencoders with multi-window local-global attention are better audio learners. In The Twelfth International Conference on Learning Represenlations, 2024.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car datasel for fine-grained categorization and verification. In 2015 IEEE Conference on Computer Vision and Pattern Recogrition (CVPR), pp. 3973-3981, 2015. doi: 141109/CVPR.215...22
Yan Yang, Bin Gao, and Ya-xiang Yuan. Bilevel reinforcement lcarning via the development of hyper- without lower-level convexity. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan (eds.). Proceedings of The 28th International Conference Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pp. 4780 4788. PMLR, 03 05 May 2025.
Zhuoyi Yang, Ming Ding, Yanhui Guo, Qingsong Lv, and Jie Tang. Parameter-efficienl tuning makes good classification head. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv: Computer Vision and Pattern Recognátion, 2017.
Salah Zaiem, Titouan Parcollet, and Slim Essid. Less forgetting for better generalization: Explor- ing continual-learning fine-tuning methods for speech self-supervised representations. arXiv preprint arXiv:2407.00756, 2024.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Li Zhang, Youwei Liang, Ruiyi Zhang, Amirhoscin Javadi, and Pengtao Xic. BLO-SAM: Bi-level optimization based finetuming of the segment anything model for werfitting-preventing semantic segmentation. Tn Ruslan Salakhutcinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (ods.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Reseavch, pp. 59289 59309. PMLR, 21 21-27 Jul 2024.
Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Leverage your local and global representations: A new self-supervised learning strategy. In Proceedings of the OREE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16580-16589, 2022a.
Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Min-Fong Hong, Shiyu Chang, and Sijia Liu. Revis- iting and advancing fast adversarial training through the lens of bi-level optimization. In International Conference on Machine Learning. 2021.
Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 18309-18326. Curran Associates, Inc., 2022b.
Yihua Zhang, Prashant Khanduri, Ioannis C. Tsaknakis, Yuguang Yao, Min-Fong Hong, and Sijia Liu. An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning. ArXiv,
00788, 2023a.
Yihua Zhang, Pranay Sharma, Parikshit Ram, Mingyi Hong, Kush Varshney, and Sijia Liu. What is missing in IRM training and evaluation? challenges and solutions. In The Eleventh International Conference Learning Representations, 2023b.
Nicolas Zucchet and João Sacramento. Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation. Newral Compulation, 34(12):2309 2346, 2022.
18

---

Published in Transactions on Machine Learning Research (02/2026)
A Theoretical Insights and Framework Comparisons in BiSSL
A.1 Derivation of the Implicit Jacobian
Assume the sctup of the BiSSL optimization problem described in 2 and (3). In the following derivations, we will assume that &p is fixed, allowing us to simplify the expressions involved. streamline the notation Vgh(v), when it is clear from context which
further, we continue use the convention Veh(E)l<= variables are differentiated with respect to. Under these cireumstances, we then define the lower-level objective from 33
G(On, 0r) :- p hp) Ar(0p,0p)
Now, furthermore assume that the regularization objective is strongly convex, and that L is twice- differentiable and L-smooth (i.c. it has Lipschitz-continmon gradients). As established and similarly ex- ploited in prior works (Rajeswaran et al. 2019 Zhang et al.) 2022b 2023a), these properties imply that an adequate scaling of À renders the lower-level objective G effectively convex. This is advantageous be- cause assuming convexity of ensures that for any OD E IRL, there exists a corresponding Op E Rf that In other words, we are assured that a potential min- imizer of -.) exists for all 0D E Rt. Now, further assume that the Hessian matrix G(O, 0p) is invertible for all 0p E R'. Under these conditions, the implicit function theorem (Dontchev & Rockafel- lar 2014 Zucchet. & Sacramento 2022) guarantees the existence of an implicit unique and differentiable function 0p (() + RL, with N(Op) being neighborhood of Pp, that satisfies (0 Op and Va, D, (0n)) = o for all 0n (On).
satisfics the stationary condition Go,,,00 0p)
As we then conclude that the lower-level solution (7(0p) is indeed a differentiable function under these conditions, this justifics that the expression
=
d0
is well-posed for all 0D E Rt. By applying the chain rule, the expression becomes
(00) T
0p(0))
d0p G(e. 0.
Recalling that V2, G(OD. (0p)) is assumed invertible, the implicit Jacobian (LJ) d0n
d00(0p)
can be isolated
d0'(0p)² T
d0p
and by substituting the expression for G from
S
the expression becomes
d0p(0p)" = AV 1o ((0,
0p) (C (r03,)))))))
d0p
VĀ ( ( (0000000000))))))
v
To summarize, given the following assumptions:
The lower-level pretext head parameters op are fixed.
The regularization objective r strongly convex.
The objective C is twice-differentiable and L-smooth, with accordingly chosen that G is convex.
The Hcssian matrix V22 G(OD. (0p))) is invertible for all 0p E L
Then, the LJ (0p) T d9
can be explicitly expressed by 99 The authors further acknowledge that an explicit expression for the IJ without fixing Φp achievable, though this is left for future exploration. The assump- tions in the final two bullets above may not strictly hold in practice with deep neural networks, yet the resulting gradient expressions have nonetheless proven practically beneficial both in this and prior work.
19

---

Published in Transactions on Machine Learning Research (02/2026)
A.2 Distinction from Bilevel Optimization in Meta-Learning
While bilevel optimization (BLO) has been applied in meta-learning frameworks such as MAML (Finn et al, 2017), Sign-MAML (Fan et al. 2021 and iMAML (Rajeswaran et al. 2019). BiSSL represents a distinct application and implementation of BI.O, tailored for the challenges of sel sell-sueeriieed learning (SSL.). In the aforementionod works, BLO is primarily utilized to address few-shot lcarning scenarios, focusing on efficiontly adapting models to new lasks with minimal labeled data. Conversely, BiSSL applies BLO to concurrently manage the more complex task of sulf-supe pretext pretraining on unlabeled data with downstream fine-tuning on labeled data. Another key distinction is that in meta-learning, the upper- and lower-level objectives are closely related, with the upper-level objective formulated a summation of the lower-level tasks. In contrast, BiSSL, involves fundamentally distinct objectives at each level, utilizing separate dalasets and tasks for pretraining and fine-tuning. This design allows BiSSL to better align the pretrained model with the roquirements of a specific downstream task. Conversely, the BLO in meta-lcarning aims to broadly generalize across a wide range of lasks, prioritizing adaptability rather than task-specific optimization.
B Experimental Details
B.1 Dataset Partitions
The Caltech-101 (Li et al. 2022a) dataset does not come with a pre-defined train/test split, so the same convention as previous works is followed (Chen et al.) 2020 Donahue et al.) 2014 Simonyan & Zisserman 2014), where 30 random images per class are selected for the training partition, and the remaining images are assigned for the test partition. For the DTD (Cimpoi et al., 2014) and SUN397 (Xiao et al., 2010) datasets, which offer multiple proposed train/lest, partitions, the first splits are used, consistent with the approach in (Chen et al. )020).
For downstream hyperparamcter optimization, portions of the training partitions from cach respective la- beled dataset are designated as validation datasets. The FGVC Aireraft (Maji et al. 2013), Oxford 102 Flowers (Nilsback & Zisserman 2008), DTD, and Pascal VOC 2007 (Everingham et al. 2010) datasets al-
ready have designated validation partitions. For the remaining labeled datasets, we reserve roughly 20% of the training data for validation. For CityScapes (Cordts et al.) 20166). this split is applied separately within each city, while for the other datasets, validation partitions randomly sampled while ensuring that the class proportions are preserved.
B.2 Downstream Task Fine-Tuning Tuning of the Baseline Setup
A random grid search of 100 hyper-parameter configurations for the base learning rates and weight decays is conducted, where one model is fine-tuned for each configuration. Base learning rates and weight decays are log-uniformly sampled over the ranges of 10-4 to 1 and 10-5 to 10-2, respectively. Validation data accuracy is evaluated after each epoch. The hyper-parameter configuration yielding the best balance betweon high validation accuracy and low validation loss is considered the optimal hyper-paraneter configuration The corresponding optimal Syper-parametery for each downstream dataset are documented in Table For subsequent evaluation on the test data, train 10 models with different random sceds, cach using the considered optimal hyper-parameter configurations. During the training of each respective model, the model parameters are stored after each epoch if the top-1 validation accuracy (or 11-point mAP for the VOCO7 dataset) has increased compared to the previous highest top-1 validation accuracy achieved during training. Top-1 and top-5 test data accuracies (or 11-point mAP for the VOCO7 dataset) are evaluated for each of the models, from which the calculated means and standard deviations of these accuracies are documented.
PIn certain scenarios during the experiments, the configuration that achieved the highest validation accuracy also yielded notably higher relative validation loss. To ensure better generalizability, an alternative configuration with more favorable trade-off was selected in these cases.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: Optimal hyper-parameter configurations used for downstream fine-tuiing after conventional
pretext pretraining.
SimCLR
BYOL
Dataset
Learning Rate
Weight Decay
Learning Rate
Weight Decay
Food
0.0167
0.00613
0.0513
0.00147
CIFARI0
0.0033
0.00158
0.0014
0.00106
CIFAR100
0.0027
0.00012
0.0023
0.00012
CUB200
0.0409
0.0084
0.0095
0.00594
SUN397
0.0069
0.00003
0.004
0.00003
Cars
0.0377
0.00454
0.115
0.00257
Aircrafts
0.0269
0.0038
0.0119
0.00333
VOCO7
0.0054
0.0089
0.0032
0.00616
DTD
0.0514
0.0011
0.0114
0.00011
Pets 0.0378
0.001 14
0.0044
0.00779
Caltoch101
0.0131
0.00005
0.0069
0.00027
Flowers
0.2178
0.00046
0.035
0.00262
B.3 Downstream Head Warmup and Upper-level of BiSSL
Table outlines the learning rates and weight decays used for the downstream head warm-up and upper- level of BiSSL of each respective downstream dataset, described in the BiSSL experimental setup of Scction 4.2.2
The first term of the upper-level gradient is approximated using the Conjugate Gradient
Table 6: dlyperparameee used for the Downstream Head Warm-up and Upper-level of BiSSL.
SimCLR
BYOL
Dataset
Learning Rate
Weight Decay
Learning Rate
Weight Decay
Food
0.03
0.001
0.035
0.002
CIFARIO
0.015
0.001
0.01
0.001
CIFARI00
0.01
0.0001
0.01
0.001
CUB200
0.03
0.001
0.015
0.0001
SUN397
0.015
0.00005
0.01
0.00005
Cars
0.035
0.001
0.04
0.002
Aircrafts
0.03
0.005
0.015
0.003
VOCO7
0.015
0.005
0.005
0.006
DTD
0.03
0.001
0.015
0.0001
Pets
0.03
0.001
0.02
0.002
Caltech101
0.03
0.0001
0.015
0.0002
Flowers
0.05
0.0005
0.035
0.002
21

---

Published in Transactions on Machine Learning Research (02/2026)
(CG) method [Nazareth, 2009 Shewchuk 1994 Our implementation follows similar structure to that used in Rajeswaran et al. (2019) employing No = 5 iterations and a dampening term Adamp 10. Given matrix A and vector v, the CG method iteratively approximates A='v, which requires evaluation of mul- tiple matrix-vector products Ad1, (0pp0((pp00)))) pp)) in its full form is often infeasible. Instead, a function that efficiently computes the re- quired matrix-vectan products instead of explicitly storing the matrix is typically utilized. For transparency, the function employed in our setup is detailed in Algorithm 2 This approach ensures that the output of the CG algorithm is an approximation of the inverse Hessian-vector product in the first term of Equation 7 intended.
Adx. In practice, storing the matrix A (in our case, the Hessian
Algorithm 2 Hessian Vector Product Calculation fn (To use in the CG Algorithm)
1: Input: Imput vector v. Model parameters 8p. Φp. Training objective LP. Lower-level data batch x. Regularization weight À and dampening Adamp-
2: x(0p) + (VoL* (0. l0
3: g - (oT 0-
D. Memory efficient calculation of V2 0
y + A+hdamp y
Return: fn(v): y
B.4
Composite Configuration of BiSSL
To avoid data being reshuffled between every training stage alternation, the respective batched lower- and upper-level training datasets are stored in separate stacks from which data is drawn. The stacks are only "reset" when the number remaining batches smaller than the number of gradient steps required before alternating to the other level. For example, the lower-level stack reshuffled every fourth training stage alternation. If the downstream dataset does not provide enough data for making Nr batches with non-overlapping data points, the data simply reshufiled every time the remaining number of data points is smaller than the upper-level batch size (c.g. 256 images in the classification experiments).
B.5
Downstream Fine- Tune-Tunng after BiSSL
The learning rates and weight decays used for downstream fine-tuning after BiSSL for each respective down- stream datasct are outlined in Table 7 Scction 4.2.2 outlines the experimental setup.
B.6 Statistical Significance Tests
As described in Section B.2 each documented accuracy is obtained by fine-tuning 10 models with different random seeds. When comparing two sets of observations, such as FT and BiSSL+FT in Table [ this yields two groups of 10 values each, for a total of 20 observations. Under the standard assumption that the values within each group are independent and identically distributed, we use two-sample permutation test assess whether the average accuracy of each group differs significantly.
Under the null hypothesis that all 20 values come from the same distribution, the test pools the values, randomly assigns 10 to each group, and computes the resulting difference between group averages for each permutation. Since there are (20) 184, 756 unique permutations, we perform the exact test by evaluating all possible assignments.
The resulting p-value is the proportion of permutations where the difference of averages at least as large as the observed difference. We consider p-values below 0.01 to indicate a statistically significant difference in average accuracy between the two groups
22

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Optimal parre-errretter configurations used for downstream fine-tuning after BiSSL.
SimCLR
BYOL
Dataset
Learning Rate
Weight Decay
Learning Rate
Weight Decay
Food
0.00059
0.00008
0.00018
0.00019
CIFAR10
0.00097
0.00258
0.00098
0.00039
CIFAR100
0.00252
0.00046
0.00042
0.00292
CUB2200
0.00043
0.00251
0.00066
0.00075
SUN397
0.00079
0.00001
0.00063
0.00029
Cars
0.05706
0.00412
0.06221
0.00317
Aircrafts
0.00015
0.00003
0.00025
0.00056
VOCO7
0.00022
0.00592
0.00029
0.004
DTD
0.00062
0.00002
0.00028
0.00008
Pets
0.00153
0.00314 4
0.00042
0.00002
Caltech101
0.00079
0.00005
0.00194
0.00003
Flowers 0.01768
0.0009
0.00058
0.00006
B.7 Object Detection and Semantic Segmentation
The experimental setup primarily follows the main implementation specified in Section 4.2 with the specific modifications made specified here.
Object Detection We utilize a Faster R-CNN with a ResNet-50 C4 backbone (Ren et al. 2015) as the downstream model architecture. In line with Grill et al. (2020): He et al. 2020); Bardes et al. (022 we assign the box prediction head as the conv, stage of the ResNet-50 followed by global pooling. We use anchors of size 32, 64, 128 and 256 with aspect ratios 0.5, 1.0 and 2.0. The downstream data augmentations involve rescaling the images so that. their longest. edge is between 196 and 320 pixels, followed by normalization. Downstream fine-tuning is conducted for 50 epochs with batch size of 16. The random hyperparameter grid scarch is performed across 50 distinct configurations of learning rates and weight decays, stochastically sampled log-uniformly within the intervals 10-4 to 10-1 and 10 to 10-2, respectively. The linear head warm-up, conducted prior to BiSSL, is executed for epochs.
We note that, since the downstream head inserted between intermediate stages of the backbone, this setup does not fully comply with the notation in Section 3.1 where the downstream head is assumed operate solely on the backbone's final output layer. Nonetheless, the derived gradient expression 7 retains its validity in this setting, as it does not depend on any part of the downstream head but only requires a one-to-one correspondence between the parameter wise components between the levels. In particular, the conv; stage parameters of the lower-level backbone can still be directly substituted into the corresponding convs stage of the upper-level backbone.
Semantic Segmentation For semantic segmentation tasks, we adopt an FCN-based architecture Long ct al. 2015a) similar to the setup used in Grill et al. (2020): He et al. (2020). We use the ResNet-50 backbone oulput prior to global pooling, and the 3 x 3 convolut ions in the convs block are reassigned dilation 2 and stride 1. The downstream head is composed of two 3 x 3 convolutions, each with dilation 6 and 256 channels, with batch normalization and ReLU activation applied in between. final X convolution performs the per-pixel classification required for segmentation, resulting in a total stride of 16 (corresponding the FCN-16 setup (Long et al. 2015a)).
23

---

Published in Transactions on Machine Learning Research (02/2026)
Data augmentation includes resizing such that the shorter image side is 192 pixels, followed by random horizontal flipping with probability 0.5 and normalization. The fine-tuning hyperparameter search is identical to the object detection setup, and the linear head warm-up is similarly conducted for 5 epochs. Although some structural modifications are applied to the backbone, similar to the object detection configuration, these changes do not. roduce any compalibility issues with BiSSL
B.8 Masked Autoencoder with ViT Backbone
Due to computational constraints, we adopt relatively lightweight configuration with a ViT-S back- bone (Touvron et al.
2021 Heo et al. 2021 and an 8 x patch size. The decoder matches the original MAE implementation (He et al. 2022). Pretext pretraining was run for 500 epochs using the AdamW (Loshchilov & Hutter 2019 optimizer with a cosine-decayed learning rate starting at 0.0005, weight decay of 0.05, and momentum coefficients of 31 0.9 and B2 = 0.95. The lower-level BiSSL stage used the same optimizer but with an initial learning rate of 0.0001. For both upper-level optimization and downstream fine-tuning, we applied 10% drop path rate and layer-wise wise learning rate decay with factor of 0.75. We conducted the random hyperparameter grid scarch over 50 different configurations. All other experimental scttings match those described in Section 1.22
B.9 Additional Baseline Comparisons
This section outlines the complete implementation details for each newly added baseline according the experiments summarized in Table 3 of Section 4.4.1 In all experiments within this section involving self- supervised pretraining, we use SimCLR as the pretext task.
B.9.1 Random Initialization, Linear Probing and lnitaaanti Classification Head
The first. baseline uses randomly initialized backbone weights instead of the SimCLLR-pretraineed backbone. The linear probing baseline freezes the backbone weights, and the pero-initialized head baseline the classification head parameters to zero rather than random values. As prior work suggests that randomly initialized heads may cause distortion of pretrained features during fine-tuning (Kumar al., 2022 Yang et al., 2022), we hypothesize that zero-initializing the downstream head may mitigate such loss. For each result, the hyperparameter grid search is conducted over 25 different combinations. All other aspects of the experimental sctup follow Section 4.2.1 Results are included in the respective 'Random Initialization, 'Linear Probing" and "FT (Zero Head Init)" rows of Table
B.9.2 NoisyTune
NoisyTune Wu et al. 2022 applies layer-wise Gaussian noise to the backbone weights prior fine-tuning, with the noise magnitude determined by each layer's statistics. We use noise scale of 0.015, and con- duct the hyperparameter grid search over 50 different combinations. All other aspects of fine-tuning follow Section 4.2.1 Results are reported in the "NoisyTune' row of Table B
B.9.3 Continued Pre- Training Downstream and Pretext Dataset Mixture
For each downstream dataset, we resumed self-supervised pretraining after adding the downstream data to the original pretraining data pool. This baseline thus continues pretraining on mixture of pretext and downstream data before fine-tuning.
Given the current BiSSL setting conducts what is roughly equivalent to 8 pretext epochs, we here continued SSL pretraining for 10 epochs using a oesine-decayed learning rate starting at 1.0. All other pretraining settings match those described in Scction 4.2.1 The subsoquent fino-tuning follows the standard sctup, but with a hyperparameter grid search over 25 combinations, Results for this baseline are reported in the "Pretraining on Data Mix FT row of Table
24

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8: Top-1 and Top-5 accuracies on the DTD dataset using different values of for the weighted objective baseline in (10), compared with conventional fino-tuning and BiSSL. Performance peaks w 0.25, but still lags behind BiSSL.
Accuracy
0
0.05
0.1
0.25
0.5
0.75
FT
BiSSL+FTT
Top-1
60.1 + 0.3
61.6 + 0.2
61.7 + 0.2
62.2 + 0.3
60.8 + 0.3
60.3 + 0.5
60.3 + 0.9
64.2 + 0.4
Top-5
86.3 0.3
86.2 + 0.3
86.2 0.3
86.1 0.3
86.2 0.3
85.7 0.3
85.8 0.6
87.8 0.3
B.9.4
Weighted Sum of Pretext and Downstream Objectives
This baseline is trained by solving the single-level optimization problem:
min 0.0P.DD
(10)
where w [0, 1] controls the relative weighting of the pretext and reamstream objectives. While this method offers a more straightforward alternative to BiSSL, it lacks the principled bilevel optimization structure, and as our results will show, its performance improvements are less elfective.
We use the lower-level optimizer configuration from BiSSL (see Section 4.2.2 and train for 4000 steps, matching the total number of upper-level iterations in the default BiSSL configuration. Fino-tuning follows the main setup but with a grid search over 50 hyperparameler combinations Backbone gradients for the two loss terms are computed separately their respective distinct mini-batches, scaled by and w respectively, and then summed prior to the update step.
To determine the suitable size of w. we conducted a sweep on the DTD datasel over different values of to in the range between 0 and 1, evaluating both top-1 and top-5 classification performance. For each value w, we conduct the subsequent fino-tuning hyperparamcter grid scarch over 25 different combinations, otherwise the mne-tuning setup remains identical as described in Section 4.2.1 The results of the sweep are shown in Table B.9.3
The results imply that = 0.25 provides the best top-1 classification accuracy. In contrast to BiSSL, the top- 5 accuracy does generally not show significant increase compared to conventional fine-tuning, regardless of the value of w. Setting w = 0 essentially reduces the training to pretext-only learning, reflected by comparable performance to conventional fine-tuning (the "FT' column), with slightly lower top-1 accuracy and slightly higher top-5 accuracy. Based on the results, we consider w 0.25 as the optimal value for this baseline.
Using 0.25, summarize the baseline experiments in the "Pretext-Downssrremm Sum FT row in Table 3
B.9.5 Discarded First Term of Upper-Level Objective BiSSL
For this bascline, we discard the first term of the upper-level objective in 2 which contains the IJ. This blocks the propagation of lower-level gradient information to the upper-level, hence instead resembling a more conventional alternating optimization scheme where only parameter values are shared. The rest of the sctup follows the default configuration Scction 4.2.2. excopt that the finc-tuning hyperparameter grid search was conducted for 50 different combinations. The results are provided in the "BiSSL (IJ Discarded) + FT''row of Table
25

---

Published in Transactions on Machine Learning Research (02/2026)
Table Parameter-wise and feature-wise cosine similarities (multiplied by 100 for better readability) between MLnCLR-protrainng and BiSSL-trained backbones with their respective fino-tuned counterparts for each dataset. In every case, the BiSSL/Fine-Tunin cosine similarities statistically significantly higher, as indicated by bold font.
Cosine Similarity (Parameters)
Cosine Similarity (Features)
Dataset
Pretoxt/FineeTTumdd
BiSSL/Finn-Thuee
Pretext/Fine-Tuned
BiSSL/Fine-Tuned
Food
15.88 = 0.22
99.79 0.03
16.77 0.43
93.50 = 0.37
CIFAR10
99.01 0.46
99.84 + 0.01
40.02 2.65
80.66 + 0.42
CIFAR100
99.50 = 0.03
99.69 - 0.07
56.00 + 0.46
89.75 = 0.97
CUB200
17.01 = 0.20
99.99 0.00
26.74 + 0.26
96.28 + 0.02
SUN397
99.45 + 0.09
99.97 + 0.00
58.33 + 0.28
97.11 + 0.08
Cars
24.42 + 0.11
26.64 = 0.10
28.78 + 0.34
65.01 = 0.14
Aircrafts
65.74 + 0.15
99.95 + 0.01
31.42 + 0.27
90.97 = 0.24
Pets
98.87 = 0.27
99.99 0.00
45.28 + 1.64
95.68 = 0.26
DTD
99.26 + 0.01
99.99 0.00
47.83 + 0.23
94.40 + 0.27
VOCO7
99.91 0.01
99.99 0.00
58.27 = 0.93
94.04 = 0.07
Caltech101
99.86 0.01
99.99 0.00
67.97 0.37
97.64 + 0.08
Flowers
98.61 = 0.02
99.98 - 0.01
48.22 - 0.11
92.46 = 0.43
Average (Fig. 3
76.46
93.82
43.80
90.63
B.10
Backbone Alignment Analysis
B.10.1
Cosine Similarities
Recall the notation in Section 3.1 where we let A denote the backbone parameters obtained after pretext task training, and (0) the backbone after applying BiSSL. We now further define 0." and (0))) for i- 1...., 10 as the fine-tuned backbone parameters of the pretext and BiSSL models, respectively, which we achieve of for each downstream dataset. (recall Section B.2.. Lastly, let
(xN(x,y)
x,y x.yER\/10)
(11)
1
denote the cosine similarity of N-dimensional vectors.
The purpose of those evaluations is then to mcasure how aligned the pretext pretrained backbones and their fine-Luned counterparts (0p) and 0.' are compared the BiSSI-trained backbones and their fine-tuned counterparts (0 and (7)())) Specifically, we first compute the parameter-we wise cosine similarities
(0, and
Cs (0 (7
(12)
for i 1,..., 10. For each downstream datasct, the mean and standard deviation of these values reported in the left. column of Table g
We also compute the average cosine similarity between output features of the respective backbones for each Prest
downstream dataset on its test partition Dr (Xx, ys) [D
given by
1
Σ
cSM(Jo(x); So:T and
Σ
CsM
(x), (x), (13)
DDei
Droon
x.yeDPvat
26

---

Published in Transactions on Machine Learning Research (02/2026)
Table 10: Total GPU hours required for pretext pretraining and BiSSL training with the SimCLR pretext task. BiSSL introduces some overhead relative to fino-tuning, while its computational cost is only fraction of that required for pretext pretraining.
Computation Time
Pretext Training
1464 GPU Hours
BiSSL Training
83 GPU Hours
Fine-Tuning Tuning
1-9 GPU Hours
for i 1.... ...., 10. During the evaluation, it is important to note that the batch normalization layers (loffe & Szegody 2015) of the backbones utilize the running means and variances inferred during training. Since the pretext and BiSSL backbones 0, (0p)) have not been exposed to the rean datasets during training, their batch normalization statistics may likely be suboptimal for the downstream datasets. To address this, the downstream training dataset is divided into batches of 256 samples, and roughly 100 batches are then forward-passer through these backbones, prior to calculating the output features in (13 This procedure updates the BN statistics to better reflect the downstream data, enabling fairer comparison of the learned representations. The resulting feature-wise cosine similarities for each downstream dataset are reported in the right column of Table
B.10.2 Visual inspection of Latent Features
Test data features of the downstream test data processed by backbones trained through conventional pretext pretraining with SimCLR are compared against those additionally trained with BiSSL. This allows for an inspection of the learned representations prior to the final fine-tuning stage.
In same manner in Section 1.10.1] we first calibrate the batch normalization running statistics by forward- passing roughly 100 downstream training batches through the backbones. We then extract the backbone output features for all samples in the test partition, and reduce their dimensionality to two using L-Distributed Stochastic Neighbor Embedding (t-SNE) (Cieslak et al. 2020). Additional plots on other downstream datasots are provided in Scction C.7
C Additional Results
C.1 Computation Times
We report the training times required for both pretext pretraining and BiSSL under the experimental setup described in Section [.2.1 and [.2.2] SimCLR pretraining was conducted on 8x A100 GPUs, and BiSSL training was measured on the DTD dataset using 4x A40 GPUs. Since the number of gradient steps is kept constant across all datasets in the main experiments, this measurement is representative of BiSSL's computational cost across other datasets as well. For transparency, we also report fine-tuning training times. The duration naturally varies with dataset size, and we present the range of computation times across the datasets listed in Table []] measured on a single A40 GPU.
The reported training times reflects the total training time required on a single GPU, therefore we adjust for multi-GPU usage (e.g., for BiSSL, we multiply the total time taken by 4, we used 4 GPUs). The results are documented in Table TO BiSSL incurs some additional overhead over fine-tuning, but requires only small fraction of the computational cost of pretraining.
C.1.1 Remarks on Hessian Vector Products
A significant portion of the computational overhead in BiSSL arises from calculating Hessian-vecton products (HVPs) during each upper-level gradient update. For each upper-level gradient update step, the upper-level
27

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Comparison of top-5 classification accuracies between the conventional SSL pipeline and our proposed BiSSL pipeline. Accuracies that are significantly higher from their counterparts are marked in bold font. Table outlines the top-1 accuracies (and 11-point mAP for the VOCO7 dataset).
Food
CIEARIO CTEAR100
CUB200
SUN:97
Cars
Airerafts
DTD
Pets
Calterh101
Flowess
SimCLR:
FT
0.1
100.0
0.0
0.14
75
7.
9.
0.
88806
91. 0.2 96.1
9.11 9.
BISS FT 93.1
0.0
1.0
96.6
0.2
8.
812
38.7
843
8..8
98.6
0.16
9.2
Avg Dif
+2.1
0.0
+0.2
-9.0
+1.7
0.0
24
+2.0
1.9
+0.5
0.8
BYOL:
FT
0.1
1.0
0..3
7..5.0.4
77.3
7.8
0.3
93.9
0.2
8.8
819
9..
9..
03.6 0.3
BISSL FT
0.1
100.0:0.000
0.0
9..40.11
845+02
70.7
940
862
56.7
96.6
98.5
0.1
941
Avg Diff
113
0.0
10.1
7.0
11.
10.1
44
118
1.0
10.5
0.6
compute No HVPs (five in the experiments of this paper) using the CG algorithm. Each HVP requires an additional backward pass, effectively adding 2N, backward passes. Memory usage is also increased, as activations must be stored for the second backward pass.
The memory and runtime costs depend on the HVP implementation, typically requiring two to three times the memory of standard gradient computations and taking between two and four times longer to com- pute (Dagréou et al. 2024). Further multiplying the runtime values by Ne highlights the substantial addi- tional compute required by upper-level optimization relative to standard gradient computation. In principle, memory and runtime are expected to scale roughly linearly with backbone size. Under heavy memory load, however, additional runtime overhead can occur, potentially resulting in super-linea scaling for very large models. The latter could however be mitigated by combining BiSSL with parametor-offcient fino-tuning (PEFT) techniques, such LoRA Hu et al. (0022), on both the upper and lower-levels, which we leave for future work.
C.2
Top-5 Classification Accuracies
Table [1 outlines the corresponding top-5 accuracies of the experiments described in Section 4.3 which re-emphasizes the performance improvements imposed by BiSSL as initially implied by Table !
C.3 Impact of Varying Upper-Level Iterations
BiSSL deviates from conventional bilevel optimization implereentations by allowing multiple upper-level gradient updates before alternating back to the lower level, controlled via the hyperparameter Ny in Algo- rithm |
We claim that conducting additional upper-level iterations improves convergence efficiency during training.
Experiments were conducted using the same five downstream datasets used in Section 4.4.1 namely Pets. DTD, VOCO7, Flowers and CUB200. As discussed there, these datasets span both coarse- and fine-grained classification tasks across diverse and largely disjoint domains. Also consistent with that setup, we use SimCLR as the pretext task.
We conduct experiments with Nv 1 match conventional bilevel update schedules, while keeping all other aspects identical to the default BiSSL setup in Section 4.2.2. aside from performing the subsequent fine-tuning hyperparameter grid search 50 different hyperparameter combinations. Since N2 = results in fewer total upper-level updates, we also experiment with longer training duration by incrcasing the number of training stage alternations from T = 500 to T = 4000, matching the total number of gradient updates conducted in the default setup. Notice however that this results in significantly more mumber of total lower-level steps (20 4000 80, 000) conducted compared to the default setting (40 500 = 20, 000).
Despite this, the results in Table I2 show that neither of the No variants provide any clear general advantage over the default configuration in the bottom row with No 8 and T = 500.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Table 12: Impact on downstream classification accuracy from setting Nu All experiments use SimCLR the pretext task. The bottom row (Nu 8, 500) corresponds to the default BiSSL configuration
described in Section 4.2.2
Pets
DTD
VOCO7
Flowers
CUB200
No 1, 500
76.1 0.5
61.8 + 0.4
71.0 + 0.1
82.4 + 0.5
50.0 + 0.9
No = 1, 4000
78.0 + 0.2
63.2 + 0.3
71.4 0.1
84.4 0.2
56.0 + 0.4
NO 8, 500
78.3 0.2
64.2 + 0.4
71.5 + 0.1
84.1 + 0.2
55.7 0.2
65
BISSL +
Tap-1 Accuracy (%)
61
60
Aw10e
Aw
Aw1gs
A=10-1
-=10-2
Aw10-a
1=10-4
1w10-s
1w10
Figure Top-1 test accuracies on the DTD dataset for models pretrained with SimCLR across different values used the BiSSL training stage. When is sufficiently large, BiSSL consistently yields substantial performance gains, with additional improvements possible through tuning around A 0.001.
C.4 Impact of Varying the Nower-Lever Regularization Weight
As discussed in Section 3.3 the magnitude of the lower-level regularization weight À controls the extent to which the upper-level downstream objective influences the lower-level pretext optimization. All main experiments in this paper use constant value of À = 0.001. To examine the effect of this parameter more systematically, we repeat the default experimental setup described in Section 4.2 while varying X. For each value, we run the full fine-tuning hyperparameter search spanning configurations. We use SimCLR as the pretext task and report results on the DTD dataset in Figure
The results show that BiSSL provides notable improvements whenever À is large enough for the upper-level signal to meaningfully shape the lover-level optimization. More precise tuning yields further gains, with À 0.001 achieving the highest accuracy among the tested values. This outcome aligns with the interpretation in Section 3.3 When is too small, the lower-level elfectively behaves like continued pretext pretraining, providing little to no enhanced alignment with the downstream task. Furthermore, the theoretical setup in Scction A.I requires X be sufficiently large in order preserve lower-level convexity. Values of À that are too small can violate these assumptions, which may also help explain the negligible or absent performance gains observed in that regime.
29

---

Published in Transactions on Machine Learning Research (02/2026)
Table 13: Top-1 validation classification accuracy on ImageNet using 1%, 10%, and 100% of the labeled training data. SimCLR used as the pretext task. Accuracies statistically significantly higher from their counterparts are marked in the bold.
ImageNet 1%
ImageNet 10%
ImageNet 100%
FT
29.7 + 0.1
50.9 + 0.1
68.2 + 0.1
+FS
31.5 0.1
51.1 + 0.1
68.3 + 0.1
Aug Diff
+1.8
+0.2
+0.1
Table 14: Downstream classification performance using 224px images with the official pretrained backbone via SimCLR. BiSSL improves accuracy with statistical significance across four of the five datasels.
SimCLR 224px
Pets
DTD
VOCU7
Flowers
CUB200
FT
87.0
+ 0.3
69.5 0.4
75.1 0.1
89.7 0.3
66.0 + 0.2
BISSL
87.7
0.2
70.7
77.2 +
89.8 0.3
71.6 0.3
Aug Diff
+0.7
+1.2
+2.1
+0.1
+5.6
C.5
Fine- Tuning on ImageNet
We expect the gains from BiSSL on Image.Net classification (Deng et al., 2009) as a downstream task to be limited in this specific setting, as pretraining is also conducted on ImageNet, which minimizes the distribution discrepancy between the pretext and downstream tasks. Nonetheless, for the sake of transparency, we conducted fine-tuning experiments on ImageNet within the bounds of our available computational resources.
Our setup consists of 50 fine-tuning epochs, using hyperparameter grid search over 25 configurations. For BiSSL, use T = 100 training stage alternations, while keeping the rest of the experimental setup consistent with the description in Section 4.2.2 We use SimCLR as the pretext task. Following standard practice in SSL literature for ImageNet fine-tuning (Chen et al. 2020 Grill el al. 2020 Bardes el al.) 2022)
we report the top-1 validation accuracy on models trained with the 1% and 10% subsets using the official splits from Chen al. (2020). Additionally, we also include results on the full ImageNet dataset (100%).
Table shows the results of these experiments. The most notable gain in the 1% setting may be attributed to greater distribution mismatch between the small labeled subset and the pretraining dataset, representing the type of scenario where BiSSL is particularly effective at improving aligument. In contrast, the 10% subset is more representative of the full dataset, and the relative advantage of BiSSL correspondingly decrease.
C.6 Larger Image Size with Official SimCLR Backbone Weights
To assess how BiSSL scales higher image resolutions and to verify its compatibility with off-the-shelf pretrained models, benchmark BiSSL using the official SimCLR ResNet-50 backbone weights released by Chen et al. (2020). trained with an input size of 224 X 244.
Because the official SimCLR checkpoints do not include their original projection head, we trained a replace- ment projection head with similar architecture on the frozen backbone for 20 epochs using the SimCLR objective. To accommodate computational limits, we used batch size of 1024 instead of the original 4096. For BiSSL, we further reduced total training time by setting T 200, Nu 4, and N2. 10 (see Algorithm for details). The subsequent fine-tuning hyperparameter grid search is conducted 25 different combina-
30

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 6: Features from backbones after applying self-supervised pretraining (left) and BiSST. (right) on the Pets (Parkhi et al. 2012) datasct.
airplane
autemobile
bind
cAt
dear
deg
frog
horse
shio
truck
Figure 7: Features from backbones after applying self-supervised pretraining (left) and BiSSL (right) on the CIFARIO (Krizhevsky 2012 dataset.
tions. We used the larger input resolution of 244 X 244, with all other settings identical to those described in Sections 4.2.1 and 4.2.2 Experiments were performed on the same five downnstream classification dataset used in Section 4.4.1 namely Pets, DTD, VOCO7, Flowers and CUB200. As argued in that section, these datascts cover both coarse- and fino-grained classification tasks across diverse and mostly disjoint domains.
While the setup does not replicate the full-scale SimCLR configuration (lacking the original projection, batch size, and employing shorter BiSSL training duration), it nonetheless provides meaningful indicator of BiSSL's robustness. As shown in Table [4 BiSSL sustains its ability to significantly enhance downstream performance, even under these conditions.
C.7 Visual inspection of Latent Features
Figure illustrated features on the flowers dataset from SinCLR-pretrained backbones to BiSSL-trained backbones. Figures 6 to [ illustrate similar backbone features on another selection the downstream datascts described Soction
The trend, where BiSSL appear to better align the backbone with the downstream task, persist even for datasets where the accuracy gains from applying BiSSL are minimal absent.
31

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 8: Features from backbones after applying self-supervised pretraining (Ieft) and BiSSI (right) on the Aircrafts (Maji et al. 2013 dataset.
Figure 9: Features from backbones after applying self-supervised pretraining (left) and BiSSL (right) on the CUB200 Wah et al. 2011) dataset.
Figure 10: Features from backbones after applying self-supervised pretraining (left) and BiSSL (right) on the Caltech-101 (Li et al. 2022a dataset.
32

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 11: Features from backbones after applying self-supervised pretraining (left) and BiSST. (right) on the Cars YYngg al., 2015) datasct.
D Discussion of Limitations and Future Work
The bilevel optimization procedure of this study employs a conjugate gradient (CG) solver, selected for its relative simplicity and prior success related settings (Rajeswaran et al. 2019 Pedregosa 2016). While this solver proved effective in our experiments, may not be the most optimal terms computational efficiency or compatibility with the non-convex landscape inherent in deep neural networks. Future work
could explore alternative solvers (Huang 2024 Hao et al. 2024 Yang al., 2025 approximation methods
for the inverse Hessian vector producls (Singh & Alistarh 2020 Frantar el al. 2021 reformulations of
the optimization problem to further improve both accuracy and efficiency.
Access to the pretext task, pretraining data, and associated pretext head parameters are current requirements for applying BiSST. This may limit applicability some constrained environments, although future work could investigate strategies to partially or completely decouple BiSSL from these dependencics.
BiSSL also requires the backbone architectures used in the pretext and downstream stages to be identical. While this is standard in many transfer learning pipelines, it may limit flexibility in scenarios involving architectural modifications, such as the use of parancter-effeient fine-tuning methods (Hu et al., 2022). However, this constraint could be mitigated by applying the same architectural modifications consistently across both stages of the BiSSL framework, another avenue worth exploring in future work.
Finally, the benefits from BiSSL are expected to be most pronounced when there meaningful distribu- tional discrepancy between the pretraining and downstream tasks. As demonstrated in Section C.5 when the downstream data distribution closely aligns with the pretext task, BiSST, may offer limited improve- ment. In practice some degree of distributional shift is often present, but users should be mindful that the effectiveness of BiSSL may depend on the extent of that mismatch.
33


Reviews:
[{"review_id": "GgjQj6S6rK", "reviewer": "Reviewer_6YHH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a framework for aligning self-supervised pretraining and downstream tasks through a bilevel optimization (BLO) formulation. The main motivation for using BLO is that the pretext task is explicitly made downstream-task-dependent: the downstream objective is optimized at the upper level, while the pretext task is optimized at the lower level within the BLO framework.\n\nAlthough BLO has previously been used in self-supervised learning, the formulation in this paper is pretext-task agnostic, i.e., it can be combined with existing SSL methods such as SimCLR. This modularity is a strong point of the work and could make the approach broadly applicable and impactful within the SSL community.\n\nThe formulation of the BLO problem and the proposed algorithm is clearly presented. The paper provides explicit derivations of the upper- and lower-level gradients required for the algorithmic design. Building upon prior work on BLO, the paper also states assumptions on the convexity of the lower-level optimization problem in order to guarantee stationarity conditions.\n\nAlthough the method demonstrates lower runtime than the baseline SSL methods in the reported experiments, a notable weakness is its scalability to larger backbone models. As mentioned in the appendix, the experiments rely on lightweight ViT architectures (for which the parameter count is not reported), and the implicit Jacobian computation and storage scale approximately linearly with the number of parameters. This raises concerns about the practicality of the approach for large-scale models commonly used in modern SSL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The method is comprehensively evaluated across several datasets (Pets, DTD, VOC07, Flowers, CUB200, CIFAR10, CIFAR100, among others), compared with different SSL frameworks such as SimCLR, MAE, and BYOL, and across different tasks. The results show consistent performance improvements in all reported scenarios. The paper also clearly documents the hyperparameter settings for each experiment. Finally, an interpretable feature analysis of the pretext task is presented via 2D t-SNE visualizations, showing that the proposed pretext process produces feature representations that cluster image classes more distinctly than those obtained from baseline self-supervised pretraining.\n\nHowever, one important issue regarding the evaluation is that it is unclear whether the BLO framework requires the downstream task used in the upper-level objective to be the same as the downstream task used during fine-tuning. This is critical because SSL methods are typically designed to be general-purpose and transferable across downstream tasks, whereas the proposed approach appears to be task-specific, potentially requiring separate pretraining stages for each distinct downstream task considered at fine-tuning.", "audience_yn": "Yes", "audience_explanation": "Yes. The paper should be of broad interest, as SSL is a central topic in the current machine learning community. In addition, the work presents interesting mathematical derivations of the BLO formulation, and the resulting algorithmic framework may also appeal to researchers in optimization and bilevel learning.", "requested_changes": "The paper mentions that the lower-level optimization problem is convex since $r(\\dot)$ is convex, but $L^P$ is clearly non-convex. The paper should clearly state under which conditions of $L^P$ the problem is convex (e.g., weak convexity, PL condition, or other structural properties) under which the overall lower-level problem becomes convex, or otherwise satisfies the conditions required for the theoretical guarantees and hypergradient derivations.\n\n\nThe paper should more explicitly characterize how memory consumption and computational time scale as the backbone size increases. This could include, for example, empirical measurements or complexity estimates that quantify the growth in runtime and memory when moving from lightweight ViTs to larger modern architectures \n\nThe paper should clearly state whether the downstream task used in the upper-level objective during the BLO stage is the same as the downstream task used for fine-tuning. If this is the case, the authors should discuss how the approach could be generalized to avoid requiring a distinct BLO-based SSL stage for every downstream task, and to what extent the learned pretext task transfers across different downstream tasks.", "broader_impact_concerns": "My main concern regarding the broader impact of the paper is the scalability of the method as the backbone model size increases. The paper should explicitly discuss this limitation and its implications for deploying the method in realistic large-scale settings. Additionally, the authors should elaborate on whether and how the learned pretext task generalizes to downstream tasks that differ from those used in the BLO stage, as this has direct consequences for the practicality and impact of the proposed framework in typical SSL use cases.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CesKab5t8R", "reviewer": "Reviewer_HMne", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes BiSSL, a bilevel optimization framework that couples self-supervised pretext objective and downstream supervised objective during an intermediate stage between pretraining and fine-tuning. The method uses implicit differentiation to let the downstream objective shape the pretext backbone via a similarity regularizer and the implicit Jacobian, and then initializes final fine-tuning from the lower-level solution. Across SimCLR or BYOL pretraining on ImageNet and multiple downstream tasks (12 classification datasets, VOC/Cityscapes detection/segmentation, and ViT+MAE), BiSSL yields consistent, significant improvements over standard fine-tuning and several strong baselines, with analyses indicating improved downstream alignment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Novelty:\n1. The bilevel setup that connects the SSL pretext stage with downstream fine-tuning is clearly motivated. It is different from earlier BLO approaches that operate only at pretraining or only at fine-tuning. \n2. The upper-level gradient derivation follows the general idea of iMAML-style implicit differentiation, but here it is adapted to the SSL plus fine-tuning pipeline, and the analysis helps explain how the similarity regularizer and its scaling parameter influence the interaction between the two stages. \n3. The  training procedure is task-agnostic and works with different SSL methods, backbones (ResNet and ViT), and downstream heads.\nExperiments:\n1. The experiments cover a wide range of settings: 12 classification datasets, detection, and segmentation, using both SimCLR and BYOL. The improvements are consistent, and the variance is reduced.\n 2. The comparisons include several baselines such as continued pretraining on mixed data, joint loss (pretext+downstream), NoisyTune, fine-tuning variants, and a BiSSL variant that removes the IJ coupling.\n3. Additional results on ViT with MAE support the method’s generality. The authors also provide useful ablations (for example on the number of upper-level steps) and include statistical significance tests.", "audience_yn": "Yes", "audience_explanation": "1. The work is at the intersection of self-supervised learning, transfer learning, and bilevel optimization, all of which are ongoing topics of interest for the TMLR audience.\n2. The work is important, for many works use off-the-shelf SSL backbones and care about how to adapt them to downstream tasks without changing pretraining.", "requested_changes": "The below proposed adjustments are for strengthening the work:\n1. Section 3.3, I would suggest to move the derivation to the appendix and keep the focus of the section on the high-level intuition.\n2. State more clearly that the theoretical assumptions  might not strictly hold in deep networks.\n3. Provide the sensitivity curve of regularization weight on atleast one or two datasets.", "broader_impact_concerns": "Ethical risks are not present, as BiSSL does not introduce new data sources or label types. Its is a training procedure applied on existing datasets commonly used in vision.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Av6LJtMkzT", "reviewer": "Reviewer_5F7N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces BiSSL, a bilevel optimization framework intended to better align self-supervised pretrained backbones with downstream fine-tuning tasks. Instead of relying purely on pretext-only pretraining, the authors insert a preparatory training stage where the pretext objective (lower level) and the downstream objective (upper level) jointly influence the backbone parameters. The lower level is regularized to stay close to the upper-level backbone, and the upper level uses an implicit-gradient approximation via conjugate gradients to guide the pretext optimization toward representations that are ultimately more compatible with the downstream task.\n\nEmpirically, the paper evaluates BiSSL very extensively—on SimCLR, BYOL, MAE; on ResNet and ViT architectures; and across many downstream classification, detection, and segmentation datasets. The method consistently improves fine-tuning performance and reduces variance. Additional analyses measuring cosine similarity and t-SNE visualizations also support the claim that BiSSL produces backbones that are more stable under fine-tuning.\n\nStrengths\n- Very comprehensive experimental evaluation across many datasets and architectures.\n- The bilevel formulation is conceptually clean and methodologically interesting.\n- Consistent and statistically significant performance gains.\n- The alignment analyses are simple but effective and help motivate the method.\n\nWeaknesses\n- The method adds a fairly heavy training stage (multiple alternations, CG steps), and computational overhead is not fully addressed.\n- Several training choices (e.g., warm-up, λ selection, NL/NU schedule) feel somewhat ad-hoc and might require expertise to tune.\n- The theoretical section mainly formalizes the optimization but does not provide much deeper insight into why BiSSL should work beyond empirical evidence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims about improved downstream performance are well supported: the paper shows results across 12+ classification datasets, two detection/segmentation tasks, three different pretext tasks, and two backbone families. The gains are generally consistent and statistically significant. The ablations and extended experiments lend credibility to the robustness of the approach.\n\nThe “better alignment” narrative is necessarily more qualitative, but the cosine-similarity results and feature visualizations provide reasonable evidence that the method makes the pretrained model more stable with respect to fine-tuning.\n\nMy only hesitation is that the method’s sensitivity to various hyperparameters is not extensively analyzed, but this does not substantially detract from the strength of the empirical results.", "audience_yn": "Yes", "audience_explanation": "TMLR readers working on self-supervised learning, meta-learning, bilevel optimization, and transfer learning will all find the idea relevant. The problem of pretraining–downstream misalignment is widely recognized, and a general-purpose framework that can sit between pretraining and fine-tuning is timely. The empirical improvements alone should be of interest to practitioners.", "requested_changes": "Critical\n\nDiscuss computational cost more explicitly. The extra bilevel stage and CG computations likely add nontrivial overhead. A clearer breakdown and comparison with baseline pretraining cost would help readers judge practicality.\n\nNon-critical (would strengthen the work)\n\nClarify the intuition for the intermediate λ regime beyond the formal derivation.\n\nProvide more detail on failure cases or datasets where BiSSL provides only marginal improvement.\n\nConsider adding a lighter-weight variant of BiSSL or discussing how one might reduce CG overhead.\n\nImprove the readability of Algorithm 1 with a higher-level explanation before diving into the full pseudocode.", "broader_impact_concerns": "The paper does not appear to introduce new societal or ethical risks beyond standard self-supervised learning. It may reduce reliance on large labeled datasets, which is usually positive. There is no application domain discussed that would raise special concerns. A brief Broader Impact section acknowledging this would be sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)