SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data Preparation We conduct experiments on four popular datasets. CIFAR-10-LT/CIFAR-100-LT are long-tail subsets sampled from the original CIFAR10/CIFAR100 [10]. We set the imbalance ratio to 100 in default. ImageNet-100-LT is proposed by [27] with 12K images sampled from ImageNet-100 [57] with Pareto distribution. Places-LT [40] contains about 62.5K images sampled from the large-scale scene-centric Places dataset [80] with Pareto distribution. For each dataset, we first sub-sample  $|\mathcal{Y}_k|$  classes from the whole long-tail datasets to stimulate known classes and treat the rest as novel classes, then sub-sample  $50\%$  instances in each known class as  $\mathcal{D}^l$  and concentrate others with all samples in novel classes to form  $\mathcal{D}^u$ . The default  $|\mathcal{Y}_k|$  for CIFAR-10-LT, CIFAR-100-LT, ImageNet-100-LT, Places-LT are 5, 80, 50, 182 respectively. We also evaluate BaCon with different ratios of  $|\mathcal{Y}_k|$  and  $|\mathcal{Y}_n|$ , set up  $\mathcal{D}^l$  with few annotated samples,  $\rho_l \neq \rho_u$  in E.
Evaluation Protocols and Metrics Following previous works [60, 4], we calculate the accuracy by assigning the model prediction (cluster result)  $\hat{\pmb{y}}$  to ground truth  $\pmb{y}$  via the Hungarian assignment algorithm [32]. With the assignment fixed, we report the overall classification accuracy on  $\mathcal{V}_k$  (Old),  $\mathcal{V}_n$  (New), and  $\mathcal{V}_k \cup \mathcal{V}_n$  (All). To observe the impact of long tail distribution on performance, we evenly divide  $\mathcal{V}_k$  and  $\mathcal{V}_n$  into three disjoint groups  $\{Many, Median, Few\}$  respectively in terms of the number of instances of each class. We also calculate the standard deviation (Std) of the accuracy of the three groups, which quantitatively analyze the balancedness of a feature space [27].
Table 2: Test accuracy (%) and balancedness (Std↓) of existing methods on CIFAR-100-LT.
Method	Subset	Known	Novel	Overall
Many↑	Med↑	Few↑	Std↓	All↑	Many↑	Med↑	Few↑	Std↓	All↑
GCD	Db	66.2	75.4	73.6	4.0	71.8	60.3	54.1	51.5	3.7	55.2	68.5
Di	75.9	69.4	52.9	9.7	65.5	41.9	54.2	49.8	5.1	49.0	62.2
SimGCD	Db	75.1	75.3	72.2	1.4	74.2	60.2	58.8	59.7	0.6	59.5	71.3
Di	72.8	71.1	34.6	17.6	59.8	32.9	25.8	15.5	7.1	24.8	52.8
ABC	Di	77.4	51.6	16.3	25.0	48.5	20.8	32.0	5.9	10.7	20.8	43.0
DARP	Di	74.8	55.2	33.4	16.9	54.5	29.9	30.1	10.0	9.4	24.0	48.4
Table 3: Test accuracy (%) on four generic long-tailed image recognition datasets. (bold: best performance among all methods, underline: best performance among the baseline methods.)
Methods	CIFAR-10-LT	CIFAR-100-LT	ImageNet-100-LT	Places-LT
Old	New	All	Old	New	All	Old	New	All	Old	New	All
ABC†	77.7	20.2	48.9	48.5	20.8	43.0	-	-	-	-	-	-
DARP†	77.6	35.2	56.4	54.5	24.0	48.4	-	-	-	-	-	-
TRSSL†	78.4	66.8	72.6	58.7	35.8	54.1	-	-	-	-	-	-
ORCA†	77.5	55.6	66.6	55.0	30.8	50.1	69.3	29.0	49.2	21.5	6.9	14.2
SimGCD	75.1	41.5	58.3	59.8	24.6	52.8	81.1	33.4	57.8	31.4	18.2	24.8
GCD	78.5	71.7	75.1	65.5	49.0	62.2	81.0	76.8	78.9	29.8	22.7	26.2
OpenCon†	87.2	47.2	67.2	64.2	40.9	59.6	83.3	42.8	63.0	30.6	12.4	21.6
BaCon-O	83.3	78.0	80.7	66.5	69.6	67.1	84.2	80.0	82.1	30.7	25.6	28.1
BaCon-S	94.2	88.1	91.1	67.4	66.5	67.2	84.6	82.8	83.7	31.1	28.4	29.9
Training Settings Following GCD [60], we load a DINO [6] pre-trained ViT-B/16 [13] as the backbone for both branches and only fine-tune the last block for all datasets. The classification head and projector are randomly initialized. To achieve a fair comparison, we replace the backbone model of all baseline methods with the ViT pre-trained by DINO (the same as ours), and  $\dagger$  denotes adapted methods. More training details can be found in Appendix C.
5.2 Migration of Existing Methods to DA-GCD
Here we first state that migrating existing methods from two similar scenarios (GCD [60] and imbalanced SSL [31]) to the proposed DA-GCD result in inferior performance due to the co-occurrence of data imbalance and open-set samples. Existing GCD methods, whether based on classifier [60] or non-parametric classification methods [70], are insufficient in addressing the issue of data imbalance. In Table 2, we report two SOTA methods (GCD [60] and SimGCD [70]) performances on the balanced set  $\mathcal{D}_b$  and its long-tailed counterpart  $\mathcal{D}_i$ . It's observed that they exhibit significant performance degradation (especially on minority classes) on  $\mathcal{D}_i$  compared to their performance on  $\mathcal{D}_b$ . While imbalanced SSL methods [33, 31] can barely classify novel class samples under the DA-GCD setting.
5.3 BaCon's Accuracy, Balancedness and Versatility
The results of the BaCon in various datasets are presented in Table 3. BaCon-O and BaCon-S refer to training the pseudo-labeling branch in ORCA-style [4] and SimGCD-style [70] respectively. We compare BaCon with two SOTA imbalanced-SSL methods ABC [33] and DARP [31], and five latest open-world recognition techniques including contrastive-based methods GCD [60] and OpenCon [56] and classifier-based methods TRSSL [52], ORCA [4] and SimGCD [70]. As shown in Table 3, BaCon significantly outperforms all these methods by a large margin under various datasets. Specifically, on CIFAR-100-LT, BaCon-S outperforms the best baseline by  $17.5\%$  and  $5.0\%$  on novel classes accuracy and overall accuracy respectively.
2We don't compare BaCon with OLTR [40] since they perform in supervised scenarios without the presence of novel classes (Table 1), resulting in inferior performance on novel class classification.
(a) DINO.
(b) GCD.
(c) Ours.
Figure 3: t-SNE visualization on the test set of CIFAR-10.
Table 4: Test accuracy (%) and balancedness (Std↓) on CIFAR-100-LT.
Methods	Known	Novel	Overall
Many↑	Med↑	Few↑	Std↓	All↑	Many↑	Med↑	Few↑	Std↓	All↑
ABC†	77.4	51.6	16.3	25.0	48.5	20.8	32.0	5.9	10.7	20.8	43.0
DARP†	74.8	55.2	33.4	16.9	54.5	29.9	30.1	10.0	9.4	24.0	48.4
TRSSL†	78.8	73.0	24.1	24.3	58.7	32.7	50.9	18.8	13.1	35.8	54.1
ORCA†	73.6	60.0	31.0	17.8	55.0	47.5	28.4	17.2	12.5	30.8	50.1
SimGCD	71.1	72.8	34.6	17.6	59.8	32.9	25.8	15.5	7.1	24.6	52.8
GCD	75.9	69.4	52.9	9.7	65.5	41.9	54.2	49.8	5.1	49.0	62.2
OpenCon†	86.8	69.7	35.8	21.2	64.2	55.8	48.9	15.3	17.7	40.9	59.6
BaCon-O	72.9	64.7	62.0	4.6	66.5	72.6	70.6	65.3	3.1	69.6	67.1
BaCon-S	74.4	67.1	60.8	5.6	67.4	73.7	66.1	59.8	5.7	66.5	67.2
Table 5: Test accuracy (%) on CIFAR-100-LT with different imbalance ratio  $\rho$  .
Methods	CIFAR-100-LT (ρl = ρu = ρ)
ρ = 20	ρ = 50	ρ = 100	ρ = 150
Old	New	All	Old	New	All	Old	New	All	Old	New	All
ABC†	64.0	23.0	55.8	53.7	21.5	47.3	48.5	20.8	43.0	41.9	18.5	37.2
DARP†	66.8	27.4	58.9	58.9	25.1	52.1	54.5	24.0	48.4	49.7	23.3	44.3
TRSSL†	71.8	43.6	66.2	62.9	49.0	60.1	58.7	35.8	54.1	55.2	33.3	50.8
ORCA†	66.9	37.7	61.0	61.7	30.0	55.3	55.0	30.8	50.1	51.0	36.5	48.1
SimGCD	71.8	23.1	62.1	63.7	23.4	55.7	59.8	24.2	52.8	57.5	19.9	50.0
GCD	69.8	58.7	67.6	66.6	59.7	65.2	65.5	49.0	62.2	65.4	50.2	62.4
OpenCon†	76.3	44.3	69.9	71.2	42.0	65.4	64.2	40.9	59.6	59.7	45.3	56.8
BaCon-O	71.5	67.9	70.8	70.0	63.7	68.7	66.5	69.6	67.1	67.4	64.2	66.8
BaCon-S	71.6	68.5	71.0	70.4	64.2	69.2	67.4	66.5	67.2	67.0	64.3	66.5
To better evaluate the impact of dataset imbalance, we make a fine-grained comparison in Table 4. The lower standard deviation of the performance of  $\{Many, Median, Few\}$  on both known and novel classes suggest BaCon yields a more balanced feature space. Besides, our method significantly improves the performance of the minority classes without sacrificing the accuracy of the majorities.
We also report the performance of BaCon when the training set has a different imbalance ratio  $\rho$  in Table 5. Note that we set the imbalance ratio of  $\mathcal{D}^l$  and  $\mathcal{D}^u$  to be the same, i.e.,  $\rho_l = \rho_u = \rho$ . Empirical results with  $\rho_l \neq \rho_u$  are in Appendix E. Notably, BaCon achieves the best overall accuracies on all settings, indicating the universal effectiveness across various scenarios, and the performance gain becomes more evident under extremely imbalanced data. Also, open-world recognition emphasizes novel class learning, where BaCon achieves a more notable performance gain.
Strategy	Old	New	All
reg with πDl	76.4	13.2	44.8
balance prior	75.6	31.7	58.3
cls k-means	76.3	36.4	56.4
con k-means	81.7	42.3	62.0
oracle	85.0	44.2	64.6
(a) Regularization Term.
Strategy	Old	New	All
baseline	65.1	52.0	62.5
vanilla	66.3	62.4	65.5
w/ debiasing	67.1	64.1	66.5
w/ sampling	66.8	65.6	66.6
w/ both	67.4	66.5	67.2
(b) Debiasing and Sampling.
Loss	Old	New	All
baseline	65.1	52.0	62.5
hard	66.0	60.7	64.9
soft	67.4	66.5	67.6
(c) Loss Design.
Interval r	Old	New	All
1	67.3	70.2	67.9
5	66.4	70.6	67.2
10	67.4	66.5	67.2
25	66.5	67.5	66.7
50	66.6	65.8	66.4
(d) Re-estimate Interval  $r$ .
Function	Old	New	All
L1	64.6	64.0	64.5
L2	63.2	60.9	62.7
cosine	67.8	64.1	67.0
dot	67.4	66.5	67.2
(e) Similarity Function.
k	Old	New	All
0	65.8	67.8	66.2
0.25	66.1	69.2	66.7
0.5	67.4	66.5	67.2
1.0	67.5	65.0	67.0
1.5	66.9	66.2	66.8
(f) Hyper-parameter  $k$
Table 6: BaCon ablation experiments. For each ablation, we report test accuracy (%) of known, novel and all classes, denote as 'Old', 'New', and 'All'. Our default settings are marked in orange.
5.4 Analysis and Ablation Study
The effectiveness of  $\mathcal{L}_{reg}$  Recall in Section 4.1, we propose to regularize the predictions of the pseudo-labeling branch by the estimated train-set distribution. In Table 6a, we show the performance of the pseudo-labeling branch on ImageNet-100-LT with different estimation strategies. 'Oracle' denotes we use the true distribution  $\pi$  of  $\mathcal{D}$  (unknown in practice) as the target distribution in Eq. 2, and it serves as an upper bound of the performance. Compared to previous works (ORCA and SimGCD) that use a balance prior, regularizing the predictions with oracle long-tailed distribution significantly improves the performance on both known and novel categories, showing the importance of the distribution estimation process. Meanwhile, the similar results achieved by our estimation strategy imply  $\pi_{e}$  could be a reliable proxy to  $\pi$ . Furthermore, we investigate whether two alternative estimation strategies could help the pseudo-labeling branch: 1), only regularize known classes prediction with  $\pi_{\mathcal{D}^l}$  2), perform k-means clustering on the feature of the pseudo-labeling branch, and they all result in inferior accuracy and could in turn deteriorate the contrastive learning process.
The effectiveness of sampling & debiasing In Section 4.2, we suggest adjusting the prediction logits according to the estimated distribution for debiasing and sampling unlabeled instances for re-balancing. We adopt step-by-step ablation experiments to the proposed approaches in Table 6b. 'Baseline' refers to not using the proposed pseudo-label-based contrastive loss, and it leads to inferior performance on novel classes since they only get supervision from the self-supervised contrastive loss. 'Vanilla' means incorporates the designed pseudo-label-based loss into the optimization objective with Eq. 7 without the debiasing and sampling module. It brings performance gain on novel classes due to the leverage of additional supervision in pseudo labels. Finally, we achieve higher test accuracy on both known and novel classes by combining the two techniques together.
Definition of the pseudo-label based contrastive loss In Section 4.3, we design a novel soft contrastive loss based on pseudo-labels to transfer the knowledge of  $f_{cls}$  into  $f_{con}$ . As an opposite, we could also construct the loss in a hard manner where we formulate the positive pairs on top of the prediction class with the largest logit and further perform the supervised contrastive loss. Intuitively, the hard design discards the probability distribution information and is more susceptible to the potential false pseudo-labels, while the soft contrastive loss utilized in our method could help alleviate the influence of erroneous pseudo-labels. The empirical results also support the intuition, as shown in Table 6c, the proposed  $\mathcal{L}_{CL}^{s}$  (soft) outperforms the supervised CL loss (hard) by a large margin. This phenomenon is also observed in knowledge distillation [25, 65] that transferring knowledge by using soft labels rather than one-hot predictions achieves better performance.
Changing of metrics and hyper-parameters The proposed soft CL loss introduces a pair-wise co-efficient to adjust the contribution of different samples to the anchor instance. We apply various similarity metrics on the pseudo-labels of  $\pmb{x}_i$  and  $\pmb{x}_p$  to calculate the positiveness score  $w_{ip}$ . The results are exhibited in Table 6e, we incorporate dot product in BaCon due to the superior performance
in practice. We also show the effect of hyper-parameters involved in BaCon. Table 6d shows the empirical results of changing estimate interval  $r$ . We observe similar accuracy when scaling  $r$ , hence we use a relatively large interval to reduce the computational overhead. Table 6f shows the classification accuracy when changing  $k$  in Eq. 4.
Feature Visualization We adopt the widely-used T-distributed stochastic neighbor embedding (t-SNE) [58] for feature space visualization. We show the feature space of BaCon (ours) with DINO [6] pre-trained model (without fine-tuning) and GCD [60] in Figure 3, different colors represent data belonging to distinct categories. Compare to vanilla DINO (Figure 3a) and GCD fine-tuned model (Figure 3b), BaCon has more compact features for samples within the same category, suggesting better intra-class consistency, and also a larger margin between different classes.
In this paper, we formulate a more realistic DA-GCD task that unifies long-tailed and open-world learning, and design a novel framework BaCon for DA-GCD to fight against data imbalance and classify open-set categories simultaneously. Extensive experiments on various datasets verify the effectiveness of the proposed method. There are nevertheless some limitations. First, our work focuses on the imbalanced open-world classification task, how to extend to object detection and instance segmentation is also worth exploring. Second, while the computation increment is not severe, how to further accelerate the proposed method will be explored in the future. We hope our work can promote the research on imbalanced open-set recognition in real-world applications.
We thank the anonymous NeurIPS reviewers for providing us with valuable feedback that greatly improved the quality of this paper!
This work is supported by the Zhejiang Provincial Key RD Program of China (Grant No. 2021C01119), the National Natural Science Foundation of China (Grant No. U21B2004, 62106222), and the Core Technology Research Project of Foshan, Guangdong Province, China (1920001000498), the Natural Science Foundation of Zhejiang Province, China (Grant No. LZ23F020008) and the Zhejiang University-Angelalign Inc. R&D Center for Intelligent Healthcare.
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
[2] Jianhong Bai, Zuozhu Liu, Hualiang Wang, Jin Hao, YANG FENG, Huanpeng Chu, and Haoji Hu. On the effectiveness of out-of-distribution data in self-supervised long-tail learning. In The Eleventh International Conference on Learning Representations, 2022.
[3] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
[4] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In International Conference on Learning Representations, 2022.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32, 2019.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[7] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33:12546-12558, 2020.
[76] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5704-5713, 2019.
[77] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 438-454. Springer, 2020.
[78] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Advances in Neural Information Processing Systems, 35:34077–34090, 2022.
[79] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596, 2021.
[80] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1464, 2017.
[81] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367-27377. PMLR, 2022.
[82] Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 915–922, 2014.
[83] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289-305, 2018.
A Proof for Claim 1
Proof. We start with recalling some related notations and definitions. In Section 4.3, we design a novel soft contrastive loss by incorporating a pair-wise positiveness score  $w$  in Eq. 1:
$$
\mathcal {L} _ {\mathrm {C L}} ^ {\text {s o f t}} (\mathcal {D}) = \frac {1}{| \mathcal {D} |} \sum_ {i \in \mathcal {D}} \left[ \frac {1}{\sum_ {j \in A (i)} w _ {i j}} \sum_ {j \in A (i)} - w _ {i j} \cdot \log \frac {\exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {j} / \tau\right)}{\sum_ {a \in A (i)} \exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {a} / \tau\right)} \right], \tag {8}
$$
where  $\mathcal{D}$  is the training set,  $A(i)$  is the index of the set of instances involved in the same batch,  $z_{i}$  is the contrastive feature for instance  $x_{i}$ , and  $\tau$  is the temperature hyper-parameter.
For each instance  $\pmb{x}_i$ , the soft contrastive loss  $\mathcal{L}_{\mathrm{CL}}^{\mathrm{soft}}(\pmb{x}_i)$  can be written as:
$$
\mathcal {L} _ {\mathrm {C L}} ^ {\text {s o f t}} \left(\boldsymbol {x} _ {i}\right) = C _ {i} \sum_ {j \in A (i)} \left[ - w _ {i j} \cdot \log \frac {\exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {j} / \tau\right)}{\sum_ {a \in A (i)} \exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {a} / \tau\right)} \right], \tag {9}
$$
where  $C_i = \frac{1}{\sum_{j \in A(i)} w_{ij}}$  is a constant value for fixed  $\mathbf{w}$ , then the contrastive logit  $p_{ij}$  is defined as:
$$
p _ {i j} = \frac {\exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {j} / \tau\right)}{\sum_ {a \in A (i)} \exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {a} / \tau\right)} \tag {10}
$$
Then, we omit the constant  $C$  in Eq. 9 and solve the optimization problem with the Lagrange multiplier. The problem is defined as follows:
$$
\left\{ \begin{array}{l l} \text {M i n i m i z e :} & f \left(p _ {i 1}, \dots , p _ {i n}\right) = - \sum_ {k = 1} ^ {n} \left(w _ {i k} \cdot \log p _ {i k}\right), \\ \text {S u b j e c t t o :} & g \left(p _ {i 1}, \dots , p _ {i n}\right) = \sum_ {k = 1} ^ {n} p _ {i k} - 1 = 0, \end{array} \right. \tag {11}
$$
where  $n$  refers to the number of instances in the training batch, i.e.,  $n = |B| = |A(i)|$ . The Lagrangian function of Eq. 11 and its corresponding partial derivatives are:
$$
\mathcal {L} \left(p _ {i 1}, \dots , p _ {i n}, \lambda\right) = - \sum_ {k = 1} ^ {n} \left(w _ {i k} \cdot \log p _ {i k}\right) + \lambda \left(\sum_ {k = 1} ^ {n} p _ {i k} - 1\right), \tag {12}
$$
$$
\left\{ \begin{array}{l l} \frac {\partial \mathcal {L}}{\partial p _ {i j}} & = - \frac {w _ {i j}}{p _ {i j}} + \lambda = 0 \\ \frac {\partial \mathcal {L}}{\partial \lambda} & = \sum_ {k = 1} ^ {n} p _ {i j} - 1 = 0 \end{array} \right. \tag {13}
$$
From Eq. 13, the optimal value of  $p_{ij}$  is  $p_{ij}^{*} = \frac{w_{ij}}{\sum_{k=1}^{n} w_{ik}}$ , which concludes the proof for Claim 1.
Claim 1 indicates that minimizing Eq. 6 encourages the similarity of features between two samples to be proportional to the corresponding positiveness score. In this way, we effectively transfer knowledge from the pseudo-labeling branch to contrastive learning.
We summarize the pipeline of self-Balanced Co-advice contrastive framework (BaCon) in Algorithm 1, and the source code can be found here.
Input: Train set  $\mathcal{D} = \{\mathcal{D}^l\cup \mathcal{D}^u\}$ , model parameter  $f_{con},f_{cls}$ , train epoch  $T$ , warm-up epochs  $w$ , estimation interval  $r$ , sampling rate  $\alpha, \beta$ , similarity metric  $\mathrm{Sim}(\cdot)$ , hyper-parameter  $p$  and  $k$ .
Output: Trained model parameter  $\theta_T$ .
Initialize: Load DINO [6] pre-trained parameters for the backbone in two branches and the classifier and projector is randomly initialized.
Algorithm 1 The overall pipeline of BaCon.
for epoch  $= 0,\dots ,T - 1$  do if epoch  $\% r = 0$  then Re-estimate the training set distribution and re-place  $\pi_{e}$  (Section 4.1);   
end if Regularize the predictions of the classifier with Eq.2; Train  $f_{cls}$  with Eq.3;   
Compute the self-supervised CL loss  $\mathcal{L}_{\mathrm{CL}}^u (\mathcal{D})$  and supervised CL loss  $\mathcal{L}_{\mathrm{CL}}^{s}(\mathcal{D}^{l})$  with Eq. 1; if epoch  $\geq w$  then Debiasing pseudo-labels with Eq.4; Obtain the sampled pseudo-labels  $M(\mathcal{D}^u)$  with sampling rate in Eq.5; Calculate the pair-wise positiveness score  $\pmb{w}$  with  $\pmb {w} = \operatorname {Sim}(\widetilde{\pmb{p}}_i,\widetilde{\pmb{p}}_j)$  Compute the soft CL loss  $\mathcal{L}_{\mathrm{CL}}^{\mathrm{soft}}(\mathcal{D}^l\cup M(\mathcal{D}^u))$  with Eq.6; Train  $f_{con}$  with Eq.7. else Train  $f_{con}$  with  $\mathcal{L}_{\mathrm{CL}}^u (\mathcal{D}) + \gamma_1\mathcal{L}_{\mathrm{CL}}^s (\mathcal{D}^l)$  end if   
end for
After training, we test the model via k-means clustering on the backbone feature of the contrastive-learning branch. The test strategy is summarized in Algorithm 2.
Input: The balanced test set  $\mathcal{D}^t = \{(x_i, y_i)\} \in \mathcal{X} \times \mathcal{Y}$ , fine-tuned backbone of the contrastive-learning branch  $f_b$ .
Output: Classification accuracy Acc.
Algorithm 2 The test stage strategy of BaCon.
for  $\pmb{x}_i\in \mathcal{D}^t$  do  
Obtain the feature of  $\pmb{x}_i$  via  $f_{b}(\pmb{x}_{i})$  end for
Perform k-means clustering on all the features of test samples with the cluster number  $C = |\mathcal{D}^t|$ ; Calculate the optimal assignment between clusters and classes by Hungarian algorithm [32]; Compute the test accuracy Acc based on the optimal assignment.
C Detailed Experimental Settings
In this section, we provide a comprehensive overview of the experimental setup, expanding upon the details presented in Section 5.1.
C.1 Benchmark Datasets