Towards Realistic Long-tailed Semi-supervised Learning in an Open World

## Abstract

Abstract Open-world long-tailed semi-supervised learning (OLSSL) has increasingly attracted attention. However, existing OLSSL algorithms generally assume that the distributions between known and novel categories are nearly identical. Against this backdrop, we construct a more Realistic Open-world Long-tailed Semi-supervised Learning ( ROLSSL ) setting where there is no premise on the distribution relationships between known and novel categories. Furthermore, even within the known categories, the number of labeled samples is significantly smaller than that of the unlabeled samples, as acquiring valid annotations is often prohibitively costly in the real world. Under the proposed ROLSSL setting, we propose a simple yet potentially effective solution called dual-stage post-hoc logit adjustments. The proposed approach revisits the logit adjustment strategy by considering the relationships among the frequency of samples, the total number of categories, and the overall size of data. Then, it estimates the distribution of unlabeled data for both known and novel categories to dynamically readjust the corresponding predictive probabilities, effectively mitigating category bias during the learning of known and novel classes with more selective utilization of imbalanced unlabeled data. Extensive experiments on datasets such as CIFAR100 and ImageNet100 have demonstrated performance improvements of up to 50.1%, validating the superiority of our proposed method and establishing a strong baseline for this task. For further researches, the anonymous link to the experimental code is at https://github.com/heyuanpengpku/ROLSSL .

## 1 Introduction

In recent years, due to the prohibitive cost of labeling large amounts of data, many researchers have shifted their focus to semi-supervised learning (SSL). This learning paradigm aims to compensate for the lack of labeled data by leveraging the information from a large amount of unlabeled data. However, most existing semi-supervised learning methods Ahmed et al. (); Berthelot et al. (); Oliver et al. (); Chen et al. () follow closed-set and class-balanced assumptions, which are unrealistic. The former assumption means that the labeled data, unlabeled data, and test data all contain samples of the same classes, but the unlabeled and test datasets often contain new classes that are not present in labeled dataset. For the latter assumption, it indicates that both labeled and unlabeled datasets are class-balanced, which conflicts the fact that the class distribution of real datasets is inevitably long-tailed. And long-tailed distribution causes a significant issue: there will be a large discrepancy in test accuracy between the head classes and the tail classes. To solve aforementioned problems, open-world semi-supervised learning (Open-world SSL) Cao et al. (); Sun and Li (); Mullappilly et al. (); Wang et al. () and long-tailed semi-supervised learning Kim et al. (); Lai et al. (); Lee et al. (); Wei and Gan (); Wei et al. () have been proposed. Moreover, to simultaneously address open-world and long-tailed recognition problems, open-world long-tailed SSL (OLSSL) Bai et al. (); Zhang et al. () is proposed to learn long-tailed and open-end data during training and test on a balanced test dataset containing samples from head, tail and open classes. Existing OLSSL methods follow a setting where the number of known classes is consistent with that of unknown classes in labeled data, which conflicts with real-world applications. The number of labeled data for known classes tends to be smaller than that of unlabeled data due to the expensive labeling cost. The realistic circumstance further increases the difficulty of recognizing known and novel classes.

**Table 1: Relationship between our novel ROLSSL and other machine learning settings.**
| Setting | Known classes | Novel classes | Data Distribution | S/N Consistency |
| --- | --- | --- | --- | --- |
| Semi-supervised learning (SSL) | Classify | Not present | Balanced | Reject |
| Robust SSL | Classify | Reject | Balanced | Reject |
| Open-set recognition | Classify | Reject | Balanced | Reject |
| Open-set SSL | Classify | Not present | Balanced | Reject |
| Long-tailed SSL | Classify | Not present | Long-tailed | Reject |
| Generalized zero-shot learning | Classify | Discover | Balanced | Yes |
| Novel class discovery | Not present | Discover | Balanced | Yes |
| Open-world recognition | Classify | Discover | Balanced | Yes |
| Open-world SSL (OSS) | Classify | Discover | Balanced | Yes |
| Open-world long-tailed SSL (OLSSL) | Classify | Discover | Long-tailed | Yes |
| Realistic open-world long-tailed SSL | Classify | Discover | Long-tailed | No |

To simulate the real-world tasks, we propose a novel SSL setting named *Realistic Open-world Long-tailed Semi-supervised Learning* (ROLSSL). Unlike the OLSSL setting, the OLSSL setting, the training set employed for model training consists of a small amount of labeled data and abundant unlabeled data for known classes in ROLSSL setting, which greatly increases the difficulty of model recognition and classification of known classes. Furthermore, the class distributions of unlabeled data are categorized into three representative forms: *Consistent*, *Uniform*, and *Reversed*. The model not only needs to extract knowledge relevant to novel classes from a large amount of long-tailed unlabeled data to identify novel classes and assign instances to them, but also utilize the extracted information to assist in training on long-tailed labeled dataset with a small number of samples for classifying known classes. It indicates that higher requirements are placed on the recognition algorithm.

Due to the poor performance of the OLSSL algorithm under the ROLSSL setting and its tendency to degrade the recognition of novel classes as training progresses, the original PLA only maintains good performance in datasets with few classes (detailed in Section 4.3). To address the ROLSSL problem, we initially apply post-hoc logit adjustment (PLA) Menon et al. () to the ROLSSL setting but find that the original PLA maintains good performance only in datasets with few classes, such as CIFAR-10 and SVHN. For datasets with more classes, it significantly reduces model performance (detailed in Ablation 4.4). Consequently, we revisit the design of PLA, incorporating sample frequency data, total class count, overall dataset size, and estimated sample frequency of unlabeled data to develop a dual-stage PLA (DPLA). By considering the relative context of current data, such as the total number of categories, the first-stage PLA adaptively modifies the relationship between the sample frequency of labeled data and the magnitude of logit adjustment, thereby encouraging a larger relative margin between the logits of rare and dominant labels in ROLSSL and preventing the degradation of novel class recognition during training. Furthermore, we aim to improve performance by making more effective use of unlabeled data. We apply the predicted sample frequency of the model to scale the logits for each class accordingly. In this process, we suppress the contribution of classes with higher frequency to the loss calculation while encouraging greater participation from classes with lower frequency. This approach, termed the second-stage PLA, helps the model achieve better recognition performance in the ROLSSL setting. Additionally, the first-stage PLA is utilized to adjust the generated pseudo-labels, further enhancing the model’s performance.

The main contributions are summarized as follows:

- •
We propose a ROLSSL setting where the number of labeled data is much smaller than that of unlabeled data for known classes, and the distribution of labeled and unlabeled data mismatches, which better simulates the requirements of real-world applications.
- •
A novel strategy named dual-stage post-hoc logit adjustments (DPLA) is designed consisting of the first stage logit adjustment that integrates factors about sample frequency and the number of classes to better utilize labeled and unlabeled data and the second stage that guides model to suppress the participation to categories with more samples and encourage to make better use of less frequent categories.
- •
The detailed experimental results and ablation experiments demonstrate that the proposed ROLSSL setting is more difficult to be solved. And the DPLA strategy achieves excellent performance compared with previous advanced methods on six benchmark datasets.

The rest of this paper is organized as follows. Section 2 introduces some relevant work in the field of Long-tailed SSL and OLSSL. The proposed method is illustrated in Section 3 and experimental results are given in Section 4. Besides, conclusions are provided in Section 5.

## 2 Related Work

Long-tailed Semi-supervised Learning: Long-tail semi-supervised learning (LTSSL) has garnered attention for its relevance in real-world applications. Various methods have been developed to tackle its challenges. Techniques such as DARP Kim et al. () and CReST Wei et al. () aim to correct biased pseudo-labels by aligning the distributions of labeled and unlabeled data. ABC Lee et al. () improves generalization by using an auxiliary classifier to adjust biases in predominant classes. CoSSL Fan et al. () employs a mixup strategy Zhang et al. () that focuses on minority classes to enhance performance. However, these methods often assume consistent distributions across labeled and unlabeled data, which may not hold true in practice. DASO Oh et al. () offers a dynamic method that adjusts pseudo-labels using linear and semantic approaches based on observed class distributions. Despite its effectiveness, the issue of skewed class distributions still affects the accuracy of learned representations and pseudo-label reliability. ACR Wei and Gan () addresses this by introducing an Adaptive Consistency Regularizer that estimates and adjusts to the true class distribution of unlabeled data, facilitating more accurate pseudo-label refinement.

Open-world Semi-supervised Learning (OSSL): ORCA Cao et al. () first proposed the OSSL task, recognizing that unlabeled test data may include classes not present in the labeled training set. It differs from novel class discovery Han et al. (, ); Zhao and Han (); Zhong et al. () in that it does not assume that unlabeled data consists solely of new class samples. Recent advancements have aimed to enhance OSSL performance. OpenLDN Rizve et al. () introduces a pairwise similarity loss to detect new classes, thereby converting the problem into a standard semi-supervised learning (SSL) task upon the discovery of new classes. OpenCon Sun and Li () employs contrastive prototype learning to create a compact representation space that promotes tight clustering by aligning representations within the same predicted category. Further studies explore solutions for scenarios where known and unknown classes share a long-tail distribution (OLSSL). Bacon Bai et al. () combines contrastive learning and pseudo-labeling to address imbalances in open-world recognition, while NCDLR Chuyu et al. () uses a relaxed optimal transport problem to infer high-quality pseudo-labels for new classes, mitigating bias in learning known and new categories. Specifically, Realistic long-tailed open-world SSL (ROLSSL) differs from existing tasks by not assuming relationships between known and unknown category distributions, and by stipulating that labeled data in known categories is significantly less than their unlabeled counterparts.

## 3 Method

To avoid the imbalance between known and novel category data, which biases model learning towards dominant labels, we have revisited strategies based on label frequency for post-hoc logit adjustment and threshold tuning for pseudo label masks. Due to the complete failure of the original post-hoc logit adjustment in open-set long-tail recognition, which suppressed model performance compared to an unmodified learning process, the former reconsidered the relationship between label frequency, category count, and dataset size to encourage a larger relative margin between the logits of rare and dominant labels. The latter, on the other hand, relies on estimates of the categories to which unlabeled data belong, making targeted adjustments to the probabilities of pseudo labels predicted to belong to different categories to promote training of less numerous classes. This also involves masking pseudo labels of more numerous classes, allowing for the use of high-quality pseudo labels to mitigate their dominance in loss computation. We retain the fundamental open-world recognition framework, which leverages pairwise similarity loss to implicitly cluster unlabeled data into known and novel categories and uses entropy regularization to prevent a single category from dominating the batch.

Figure: Figure 1: The overview of the ROLSSL setting and the Dual-stage Post-hoc Logit Adjustment method. On the left, the dataset composition within the ROLSSL framework is illustrated. On the right, the overall process of the Dual-stage Post-hoc Logit Adjustment is shown. In the first stage of logit adjustment, factors such as the number of classes, sample frequency, and overall dataset size are considered to encourage a larger relative margin between the logits of rare and dominant labels. In the second stage, the predicted class frequencies are used to adjust the logits for the unlabeled data, further guiding the model to focus on learning from predicted minority class samples and reducing the attention given to samples from the predicted majority classes.

### 3.1 Problem Formulation

In the ROLSSL scenario, we consider three kinds of datasets: a labeled known-class dataset $\mathcal{D}^{l}_{k}$, an unlabeled known-class dataset $\mathcal{D}^{u}_{k}$, and an unlabeled novel-class dataset $\mathcal{D}^{u}_{n}$. The known-class dataset $\mathcal{D}^{l}_{k}$ consists of $m^{l}_{k}$ labeled samples $\{(x_{i}^{l},y_{i}^{l})\}_{i=1}^{m^{l}_{k}}$ and the unlabeled known-class dataset $\mathcal{D}^{u}_{k}$ consists of $m^{u}_{k}$ unlabeled samples $\{x_{j}^{u}\}_{j=1}^{m^{u}_{k}}$, where $x_{i}^{l}$ is a labeled instance with label $y_{i}^{l}=[c_{k}]=\{1,2,\ldots,c_{k}\}$, and $x_{j}^{u}$ is an unlabeled instance from one of $c_{k}$ known classes, with $m^{l}_{k}\ll m^{u}_{k}$. Let $N_{c}$ represent the number of samples for class $c$ in the labeled known-class dataset, we have $N_{1}\geq N_{2}\geq...\geq N_{c_{k}}$, and the imbalance ratio of the labeled known-class dataset can be denoted as $\gamma_{k}^{l}=\frac{N_{1}}{N_{c_{k}}}$. The unlabeled known-class dataset remains the same setting of the labeled known-class dataset and the number of samples for class $c$ is denoted as $H_{c}$ with imbalance ratio $\gamma_{k}^{u}$. For the unlabeled novel-class dataset, let $M_{c}$ represent the number of samples for class $c$ and the imbalance ratio $\gamma_{n}^{u}=\frac{max_{c}M_{1}}{min_{c}M_{c_{k}}}$ because there is no assumption about the distributions on the unlabeled novel-class dataset. Three kinds of representative distributions are considered, i.e., consistent, uniform, and reversed. Specifically, 1) for *Consistent* setting, we have $M_{1}\geq M_{2}\geq...\geq M_{c_{k}}$ and $\gamma_{k}^{l}=\gamma_{n}^{u}$; 2) for *Uniform setting*, we have $M_{1}=M_{2}=...=M_{c_{k}}$ and $\gamma_{n}^{u}=1$; 3) for *Reversed* setting, we have $M_{1}\leq M_{2}\leq...\leq M_{c_{k}}$ and $\gamma_{k}^{l}=1/\gamma_{n}^{u}$. The unlabeled novel-class dataset $\mathcal{D}^{u}_{n}=\{x_{j}^{u}\}_{j=m^{u}_{k}+1}^{m^{u}_{k}+m^{u}_{n}}$ includes $m^{u}_{n}$ samples, each belonging to one of $c_{n}$ novel classes, where $c_{n}$ represents the total number of classes in $\mathcal{D}^{u}_{n}$, with $m^{l}_{k}+m^{u}_{k}\stackrel{{\scriptstyle\sim}}{{=}}m^{u}_{n}$. Under the ROSSL framework, the combined unlabeled dataset $\mathcal{D}^{u}=\{\mathcal{D}^{u}_{k},\mathcal{D}^{u}_{n}\}$ may contain samples from classes that are not present in the labeled dataset $\mathcal{D}^{l}=\{\mathcal{D}^{l}_{k}\}$, with the total class count $c_{t}$ in the open-world setting being $c_{t}=c_{k}+c_{n}$.

### 3.2 Foundational Techniques of OSSL

To identify new classes, previous work employs a neural network Rizve et al. (), denoted as $f_{\Psi}$, for feature extraction. This network projects an input image $x\sim\mathbb{P}^{\mathcal{Q}},\mathcal{Q}={m^{u}_{k}+m^{l}_{k}+m^{u}_{n}}$ for unknown distribution $\mathbb{P}$, into a high-dimensional embedding space $\mathbb{Z}$ by transforming $x$ into its embedded representation $\mathbf{z}\in\mathbb{R}^{d}$. The set of all embeddings is represented by $\mathbb{Z}$, and $\mathbb{X}$ denote the sets of input images, respectively. The system recognizes both known and novel class samples by employing a classifier, $f_{\Theta}$, which maps embeddings $\mathbf{z}$ to a structured output space $f_{\Theta}:\mathbb{Z}\rightarrow\mathbb{R}^{c_{k}+c_{n}}$, where the first $c_{k}$ logits are associated with known classes and the remaining $c_{n}$ logits correspond to novel classes. The classifier outputs are converted into softmax probabilities $\hat{\mathbf{y}}=\text{Softmax}(f_{\Theta}\circ f_{\Psi}(x))$ for further processing. The primary goal is to effectively discern novel classes while maintaining recognition of known classes, achieved through an objective function comprising three components: a pairwise similarity loss $\mathcal{L}_{pair}$, a cross-entropy loss $\mathcal{L}_{ce}$, and an entropy regularization term $\mathcal{L}_{reg}$. The pairwise similarity loss enhances class differentiation Hsu et al. (); Chang et al. (), the cross-entropy loss facilitates the classification of known and novel classes using true labels and generated pseudo-labels Han et al. (); Chapelle and Zien (), and the entropy regularization prevents the model from settling on overly simplistic solutions Arazo et al. ():

$$ $\mathcal{L}_{ossl}=\mathcal{L}_{pair}+\mathcal{L}_{ce}+\mathcal{L}_{reg}$ (1) $$

Following training with $\mathcal{L}_{ossl}$, samples corresponding to the $c_{n}$ logits in the output space are classified as belonging to novel classes. Eventually, novel class samples are added to the labeled set with the generated pseudo-labels, enabling the application of any standard closed-world semi-supervised learning (SSL) method, thereby leading to further performance improvements.

### 3.3 DPLA: Dual-Stage Post-hoc Logit Adjustment

We initially consider the post-hoc logit adjustment (PLA) for data where the frequency of samples corresponding to specific categories can be precisely obtained Menon et al. (); Tao et al. (); Wang et al. (). Given a labeled known-class sample $x_{i}^{l}$, suppose we learn a neural network with logits $f_{y}(x_{i}^{l})=w_{y}^{\top}\Phi(x_{i}^{l}),f_{y}=f_{\Theta}\circ f_{\Psi}$. We predict the label $\arg\max_{y\in[c_{k}]}f_{y}(x_{i}^{l})$. When trained with softmax cross-entropy, $p_{y}(x_{i}^{l})\propto\exp(f_{y}(x_{i}^{l}))$ can be viewed as an approximation of $\mathbb{P}(y|x_{i}^{l})$, predicting the label with the highest probability. In the first-stage post-hoc logit adjustment for known class, we propose a new prediction method for the known-class dataset with suitable $\tau_{1}>0$:

$$ $\operatorname{argmax}_{y_{i}^{l}\in[c_{k}]}\operatorname{exp}\left(w_{y}^{\top }\Phi(x_{i}^{l})\right)/\Omega_{y_{i}^{l}}^{\tau}=\operatorname{argmax}_{y_{i} ^{l}\in[c_{k}]}f_{y}(x_{i}^{l})-\tau_{1}\cdot\log\Omega_{y_{i}^{l}}\\ \\$ (2) $$

Specifically, $\Omega_{y_{i}^{l}}$ is a parameter synthesizing consideration of number of classes, the sample frequency and overall size of dataset, which can be defined as (detailed in Ablation 4.4):

$$ $\Omega_{y_{i}^{l}}=10\cdot(\lceil\mathcal{C}/\mathcal{C}_{base}\rceil)\cdot \sqrt{\mathcal{S}/\mathcal{S}_{base}}\cdot\mathcal{F}_{y_{i}^{l}}$ (3) $$

where $\mathcal{C}$, $\mathcal{S}$ and $\mathcal{F}$ are the total number of classes, overall size of the estimated dataset, $\mathcal{C}_{base}$ and $\mathcal{S}_{base}$ are the basic discounting parameter for total number of classes and overall size of the dataset, and $\mathcal{F}_{y_{i}^{l}}$ represents the sample frequency of the category to which the corresponding label belongs. For $\tau\neq 1$, we apply temperature scaling to the logits, formulated as $\overline{p}_{y_{i}^{l}}(x_{i}^{l})\propto\exp\left(\tau^{-1}\cdot w_{y_{i}^{l
}}^{\top}\Phi(x_{i}^{l})\right)$. This adjustment is based on having access to the true probabilities $\mathbb{P}(y_{i}^{l}|x_{i}^{l})$ and involves calibrating the probabilities through temperature scaling, commonly used in the context of distillation Hinton et al. (). These techniques help improve the model’s generalization ability across different class distributions. For the second stage, given the unknown categories of the unlabeled data, we cannot perform post-hoc logit adjustments as with known-category data where sample frequencies are accessible Van Engelen and Hoos (). However, the imbalance in the unlabeled data necessitates corresponding logit adjustments. We propose a simple logit adjustment approach for the unlabeled data, which involves scaling the logits for each category based on the predictions of neural network $f$ on the categories for the unlabeled samples and the scaling weight $w_{c}$ for class $c$ can be defined as:

$$ $w_{c}=\sigma\left(\frac{\exp\left(-\pi^{r}_{c}\right)}{\exp\left(-\pi^{r}_{ \max}\right)}\right)\cdot(\alpha-\beta)+\beta$ (4) $$

where $\pi^{r}_{c}$ represents the ratio of the number of samples in class $c$ to the total number of samples across all classes, $\sigma$ denotes the sigmoid activation function, $\alpha$ and $\beta$ are hyper-parameters for re-adjusting the scaling weight. Assume $w=[w_{1},w_{2},...,w_{c_{k}+c_{n}}]$ is a vector of length $|c_{k}+c_{n}|$, the scaled logit $\hat{f}(x_{j}^{u})$ for an unlabeled sample $x^{u}$ from known or novel class can be given as:

$$ $\hat{f}(x_{j}^{u})=w\cdot f(x_{j}^{u})$ (5) $$

Due to the uniform threshold applied to pseudo-label masking Cai et al. (); Zheng et al. (), we propose leveraging an estimation of the distribution of unlabeled data to scale the logits. This method facilitates the adjustment of the masking level for samples across different predicted categories. More specifically, it limits the participation in loss calculation of samples from categories with a higher number of predicted instances, while increasing the participation rate of samples from categories with less estimated amounts. This adjustment aids the model in focusing more on learning from samples that are biased towards the tail classes Ma et al. ().

### 3.4 Overall Optimization Objective

Inspired by Wei and Gan (), the logits of original pseudo-label ${q}(x_{j}^{u})$ corresponding to known classes in the part used for generating refined pseudo-labels $\widetilde{q}(x_{j}^{u})$ are adjusted:

$$ $\widetilde{q}(x_{j}^{u})=\arg\max\left(f(x_{j}^{u})_{[1,c_{k}]}-\tau_{2}\cdot \log\Omega_{{q}(x_{j}^{u})}\right),\tau_{2}>0$ (6) $$

where $f(x_{j}^{u})_{[1,c_{k}]}$ represents the operation of adjusting the logits for the known categories in the generated pseudo-labels, in the same manner as is done with labeled data. Therefore, for the loss calculation in adjusted branches of labeled and unlabeled data based on cross-entropy loss Ren et al. (); Sohn et al. () the loss function of adjusted branch can be defined as follows:

$$ $\mathcal{L}_{b_ce}=-\sum_{i=1}^{m^{l}_{k}}\log\left(\frac{e^{f_{y}(x_{i}^{l}) +\tau\log\Omega_{y_{i}^{l}}}}{\sum_{c=1}^{c_{k}}e^{f_{c}(x_{i}^{l})+\tau\log \Omega_{c}}}\right)+\sum_{j=1}^{m^{u}_{k}+m^{u}_{n}}\mathbb{M}(x_{j}^{u}) \mathcal{L}_{ce}\left(\hat{f}(x_{j}^{u}),\widetilde{q}(x_{j}^{u})\right)$ (7) $$

where $\mathcal{L}_{ce}$ represents standard Cross Entropy loss and $\mathbb{M}(x_{j}^{u}):=\mathbb{I}\left(\max(\delta(\hat{f}(x_{j}^{u}))\geq\rho
)\right)$ is the sample masks which selects unlabeled samples with predicted confidence levels exceeding a predefined threshold $\rho$. In detail, $\delta(\cdot)$ and $\mathbb{I}(\cdot)$ denote Softmax function and indicator function Wei and Gan (). Therefore, there are two types of losses in neural networks that need to be optimized. The first is the original Cross Entropy loss calculation for both labeled and unlabeled data; The second is the balanced Cross Entropy loss calculation after logit adjustment, which can be given as follows:

$$ $\mathcal{L}_{rolssl}=\mathcal{L}_{pair}+\lambda_{1}\mathcal{L}_{ce}+\lambda_{2 }\mathcal{L}_{b_ce}+\mathcal{L}_{reg}$ (8) $$

where $\lambda_{1}$ and $\lambda_{2}$ are trade-off parameters, and they are generally set to $\lambda_{1}=\lambda_{2}=0.5$ in order to keep the scale of the loss consistent with OLSSL design (detailed in Ablation 4.4).

## 4 Experiments

### 4.1 Implementations

Datasets: To evaluate the effectiveness of OpenLDN, we conduct experiments on five widely-used benchmark datasets: CIFAR-10 Krizhevsky et al. (), SVHN Netzer et al. (), CIFAR-100 Krizhevsky et al. (), ImageNet-100 Deng et al. (), Tiny ImageNet Le and Yang (), and the Oxford-IIIT Pet dataset Parkhi et al. (). The CIFAR-10 and CIFAR-100 datasets each contain 60,000 images (split into 50,000 for training and 10,000 for testing), with 10 and 100 categories, respectively. SVHN dataset contains 73257 digits for training, 26032 digits for testing, with 10 classes. The ImageNet-100 dataset consists of 100 categories selected from ImageNet. Tiny ImageNet includes 100,000 training images and 10,000 validation images across 200 classes. The Oxford-IIIT Pet dataset comprises images from 37 categories, divided into 3,718 training and 3,707 testing images.

Implementation Details: We employ ResNet-18 as our primary feature extractor across all experiments except in instances involving ImageNet-100, where ResNet-50 is utilized. Our pairwise similarity prediction network, utilizing an MLP with a single 100-dimensional hidden layer, and a linear classifier, forms the basis of our feature extraction architecture. We train the network to discover novel classes over 50 epochs with batch sizes of 200 and 480 for ImageNet-100. For CIFAR-10 dataset, SGD optimizer is employed and the Adam optimizer is used consistently throughout the training process for the remaining five datasets. The learning rates are set at 5e-4 for the feature extractor and 1e-2 for ImageNet-100. In order to boost performance, we incorporate Mixmatch, a well-regarded closed-world SSL methods, during the second stage of training to enhance data balance and pseudo-label accuracy for each class during iterative self-labeling sessions. More details on these implementation strategies and parameter settings, e.g. $N_{c}$, $\tau_{1}$, can be found in Appendix A.

Evaluation Metrics: We assess accuracy for known classes using standard measures. For novel classes, we evaluate clustering accuracy and employ the Hungarian algorithm for accurate prediction alignment and ground truth labels matching before final accuracy calculations. The effectiveness of the proposed method is further demonstrated by joint accuracy measurements on both known and novel classes utilizing the Hungarian algorithm and normalized mutual information (NMI).

OSSL Baselines: We employ FixMatch Sohn et al. (), DS^3L Guo et al. (), CGDL Sun et al. (), DTC Han et al. (), RankStats Han et al. (), UNO Fini et al. (), ORCA Cao et al. () and OpenLDN Rizve et al. () to compare OSSL baselines with ROLSSL methods.

**Table 2: Accuracy on the CIFAR-10, CIFAR-100, and ImageNet-100 datasets with 50% known and 50% novel classes under three different long-tailed conditions.**
|  | CIFAR-10 | CIFAR-100 | ImageNet100 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Known | Novel | All | Known | Novel | All | Known | Novel | All |
| Method | *Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| FixMatch (NIPS’20) | 71.5 | 50.4 | 49.5 | 39.6 | 23.5 | 20.3 | 65.8 | 36.7 | 34.9 |
| DS^3L (PMLR’20) | 77.6 | 45.3 | 40.2 | 55.1 | 23.7 | 24.0 | 71.2 | 32.5 | 30.8 |
| CGDL (CVPR’20) | 72.3 | 44.6 | 39.7 | 49.3 | 22.5 | 23.5 | 67.3 | 33.8 | 31.9 |
| DTC (CVPR’19) | 53.9 | 39.5 | 38.3 | 31.3 | 22.9 | 18.3 | 25.6 | 20.8 | 21.3 |
| RankStats (ICLR’20) | 86.6 | 81.0 | 82.9 | 36.4 | 28.4 | 23.1 | 47.3 | 28.7 | 40.3 |
| UNO (ICCV’21) | 91.6 | 69.3 | 80.5 | 68.3 | 36.5 | 51.5 | — | — | — |
| ORCA (ICLR’22) | 88.2 | 90.4 | 89.7 | 66.9 | 43.0 | 48.1 | 89.1 | 72.1 | 77.8 |
| OpenLDN (ECCV’22) | 95.2 | 92.7 | 94.0 | 73.3 | 46.8 | 60.1 | — | — | — |
| Method | *Long-tailed (Consistent) & Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| OpenLDN | 44.2 | 12.7 | 28.4 | 31.3 | 11.6 | 22.9 | 18.7 | 4.2 | 12.5 |
| Ours | 46.7 | 38.7 | 46.2 | 32.0 | 18.9 | 25.4 | 17.1 | 8.1 | 14.0 |
| NMI (OpenLDN) | - | 0.196 | 0.224 | - | 0.391 | 0.389 | - | 0.256 | 0.281 |
| NMI (Ours) | - | 0.564 | 0.464 | - | 0.427 | 0.424 | - | 0.285 | 0.305 |
| Method | *Long-tailed (Reversed) & Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| OpenLDN | 48.5 | 1.2 | 26.6 | 27.2 | 19.7 | 24.0 | 18.8 | 4.8 | 13.7 |
| Ours | 49.8 | 38.3 | 44.1 | 31.8 | 20.7 | 26.8 | 13.9 | 13.6 | 15.2 |
| NMI (OpenLDN) | - | 0.068 | 0.125 | - | 0.378 | 0.372 | - | 0.256 | 0.287 |
| NMI (Ours) | - | 0.394 | 0.405 | - | 0.457 | 0.438 | - | 0.302 | 0.323 |
| Method | *Long-tailed (Uniform) & Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| OpenLDN | 44.5 | 3.8 | 24.2 | 21.4 | 7.6 | 14.8 | 13.6 | 3.3 | 10.4 |
| Ours | 47.1 | 53.9 | 50.5 | 22.8 | 8.2 | 15.9 | 11.2 | 7.3 | 11.4 |
| NMI (OpenLDN) | - | 0.111 | 0.155 | - | 0.289 | 0.280 | - | 0.238 | 0.252 |
| NMI (Ours) | - | 0.429 | 0.399 | - | 0.351 | 0.323 | - | 0.271 | 0.273 |

### 4.2 Discussions on Experimental Results

In Tables 2 and 3, we compare the performance of OpenLDN and the proposed method across six experimental benchmark datasets under Long-tailed (*Consistent*), Long-tailed (*Reversed*), and Long-tailed (*Uniform*) conditions. The results demonstrate that the proposed method consistently outperforms OpenLDN across almost all datasets in terms of known class, novel class, and overall class recognition accuracy. Furthermore, the proposed method is able to achieve a more stable training process (detailed in Section 4.3). Specifically, under the Long-tailed (*Consistent*) condition, the proposed method shows significant improvements in CIFAR10, CIFAR100, SVHN, and Oxford-IIIT Pet datasets. Under the Long-tailed (Reversed) condition, the proposed method exhibits substantial enhancements in recognizing novel classes across all datasets, with particularly notable performance in CIFAR10, ImageNet100, and SVHN. Under the Long-tailed (*Uniform*) condition, the proposed method significantly surpasses OpenLDN in CIFAR10 and SVHN datasets. Moreover, the proposed method achieves higher normalized mutual information (NMI) scores in the recognition of both novel classes and overall samples, which further attests to the superior performance of the model presented in this paper. These results indicate that the proposed method not only excels in recognizing known classes but also demonstrates exceptional performance in novel and overall class recognition, highlighting its robustness and adaptability in handling complex, long-tailed distributions. Overall, the proposed method showcases superior accuracy and broad applicability in ROLSSL tasks, proving its effectiveness in addressing the challenges posed by diverse datasets and varying class distributions. The proposed method provides a potentially viable solution within the ROLSSL framework.

**Table 3: Accuracy on the Tiny ImageNet, Oxford-IIIT Pet, and SVHN datasets with 50% known and 50% novel classes under three different long-tailed conditions.**
|  | Tiny ImageNet | Oxford-IIIT Pet | SVHN |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Known | Novel | All | Known | Novel | All | Known | Novel | All |
| Method | *Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| DTC (CVPR’19) | 28.8 | 16.3 | 19.9 | 20.7 | 16.0 | 13.5 | 90.3 | 65.0 | 81.0 |
| RankStats (ICLR’20) | 5.7 | 5.4 | 3.4 | 12.6 | 11.9 | 11.1 | 96.3 | 96.1 | 96.2 |
| UNO (ICCV’21) | 46.5 | 15.7 | 30.3 | 49.8 | 22.7 | 34.9 | 85.4 | 74.3 | 79.0 |
| OpenLDN (ECCV’22) | 52.3 | 19.5 | 36.0 | 67.1 | 27.3 | 47.7 | 95.7 | 87.2 | 92.6 |
| Method | *Long-tailed (Consistent) & Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| OpenLDN | 13.7 | 7.7 | 12.1 | 12.7 | 2.1 | 10.5 | 59.9 | 0.5 | 37.9 |
| Ours | 15.6 | 10.0 | 13.8 | 14.9 | 6.7 | 12.0 | 67.5 | 35.4 | 56.2 |
| NMI (OpenLDN) | - | 0.421 | 0.418 | - | 0.146 | 0.134 | - | 0.099 | 0.236 |
| NMI (Ours) | - | 0.445 | 0.441 | - | 0.157 | 0.147 | - | 0.292 | 0.470 |
| Method | *Long-tailed (Reversed) & Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| OpenLDN | 13.8 | 9.5 | 13.0 | 13.2 | 1.8 | 10.0 | 31.4 | 15.3 | 25.9 |
| Ours | 17.3 | 10.3 | 14.9 | 13.0 | 8.5 | 12.5 | 77.1 | 20.2 | 48.6 |
| NMI (OpenLDN) | - | 0.410 | 0.422 | - | 0.127 | 0.119 | - | 0.064 | 0.121 |
| NMI (Ours) | - | 0.426 | 0.427 | - | 0.165 | 0.154 | - | 0.170 | 0.465 |
| Method | *Long-tailed (Uniform) & Semi-supervised & Open-world* |  |  |  |  |  |  |  |  |
| OpenLDN | 9.5 | 9.0 | 9.0 | 10.7 | 1.4 | 9.5 | 56.3 | 2.9 | 36.6 |
| Ours | 9.2 | 10.6 | 9.6 | 10.2 | 4.2 | 10.2 | 58.4 | 31.9 | 49.7 |
| NMI (OpenLDN) | - | 0.385 | 0.379 | - | 0.126 | 0.114 | - | 0.050 | 0.210 |
| NMI (Ours) | - | 0.401 | 0.390 | - | 0.123 | 0.113 | - | 0.297 | 0.451 |

Figure: (a) CIFAR-10 (OpenLDN)
Refer to caption: extracted/2405.14516v1/tsne_com.png

### 4.3 Discussion about OSSL method in ROLSSL settings

In the previous section, we mentioned that directly applying the OSSL scheme within the ROLSSL framework leads to a decline in the recognition ability for novel classes as the training progresses. Here, we illustrate this phenomenon by examining the recognition accuracy of known, novel, and overall samples during the training process on the SVHN dataset under a consistent setting using the OLSSL scheme. As shown in the Figure 5 of Appendix B, the OLSSL scheme, OpenLDN, consistently exhibits low recognition performance for novel classes, dropping to zero recognition accuracy for novel classes at around the 16th epoch and failing to recover this ability throughout the subsequent training. In contrast, the dual-stage post-hoc logit adjustment (DPLA) proposed in this paper effectively addresses this issue. DPLA maintains the ability to recognize novel class samples and can achieve recognition accuracy close to or even exceeding that of OpenLDN for all samples at certain stages, demonstrating the effectiveness of DPLA. Moreover, DPLA consistently achieves higher recognition accuracy for both known classes and overall samples compared to OpenLDN, without experiencing a gradual decline in accuracy. This indicates that DPLA is well-suited for the ROLSSL framework and significantly improves accuracy. It is also noteworthy that OpenLDN rarely regains the ability to recognize novel class samples as training progresses; however, due to random seed variations, OpenLDN has a slight chance of achieving very low recognition accuracy for novel class samples in the final few epochs, thereby transitioning into a close-world training phase. To highlight the performance differences between OpenLDN and DPLA under their optimal conditions, we selected the best performance of OpenLDN when it had a favorable initialization and could transition into the close-world training phase for comparison.

### 4.4 Ablation Study

**Table 4: Performance comparison of ablation experiments designed to explore the role of each stage of DPLA. Baseline is OpenLDN.**
| Method | Known | Novel | All |
| --- | --- | --- | --- |
| Baseline | 59.9 | 0.5 | 37.9 |
| + First Stage | 65.1 | 32.5 | 54.4 |
| + Second Stage | 67.2 | 35.3 | 55.7 |
| +PLR (DPLA) | 67.5 | 35.4 | 56.2 |

Method Design: We utilize the SVHN dataset to investigate the impact of each design within each DPLA on model performance. We employ OpenLDN as the performance baseline for model comparisons. As observed, the inclusion of logit adjustment in the first stage significantly improves the performance for known, novel, and overall categories, with accuracy for the novel category increasing by up to 30%. The introduction of the second stage and pseudo-label adjustment further enhances model performance, though the improvement is less pronounced and shows a diminishing trend.
First-stage Scaling Factor: The reason for designing the scaling factor $10\cdot(\lceil\mathcal{C}/\mathcal{C}_{base}\rceil)\cdot\sqrt{\mathcal{S}/
\mathcal{S}_{base}}$ in the first stage is that original post-hoc logit adjustment design only achieves expected performance in datasets with fewer categories, such as CIFAR-10 and SVHN. However, for datasets like CIFAR-100 and ImageNet-100, it suppresses model performance and fails to improve accuracy under data imbalance conditions. Therefore, we design the first-stage scaling factor based on sample frequency, total number of dataset categories, and data size. From Figures 4 and 4, it can be concluded that when PLA is applied directly to ROLSSL without any modifications, the model performance is even lower than the baseline performance obtained by directly applying OpenLDN to ROLSSL. As the scaling factor increases to the multiples set in this study, model accuracy gradually rises and eventually surpasses the baseline. However, when scaling factor continues to increase, model performance declines, demonstrating the rationality and effectiveness of our proposed method design.

Figure: Figure 3: Ablation study on the performance of CIFAR-100 for the Scaling Factor.

Trade-off Parameter: The design of the final loss optimization objective involves setting $\lambda_{1}$ and $\lambda_{2}$. We conduct an investigation based on the CIFAR-10 dataset, and it is observed that for this dataset, $\lambda_{1}=\lambda_{2}=0.5$ is the optimal setting. Under other settings, the model performance shows some degree of decline, and this performance pattern is consistent across most other datasets. In the experiments, $\lambda_{1}+\lambda_{2}=1$ should be satisfied, mainly to maintain the numerical scale of each loss similar to the original design in $\mathcal{L}_{OSSL}$, which is beneficial for recognizing novel category data.

## 5 Conclusion

Realistic open-world long-tailed semi-supervised learning (ROLSSL) provides a more realistic experimental setup for open-world semi-supervised learning by considering various data imbalance relationships among known and novel categories, as well as the high cost of obtaining labeled data in real-world scenarios. Building on the traditional post-hoc logit adjustment, this paper proposes dual-stage post-hoc logit adjustment (DPLA). By integrating factors such as sample frequency and the total number of categories, this approach better utilizes both labeled and unlabeled data. In simple terms, DPLA tends to suppress the proportion of samples from more frequent categories in loss computation while encouraging the greater use of samples from less frequent categories. The proposed method significantly improves model performance under the ROLSSL setting, outperforming other comparative approaches and providing a simple yet strong performance baseline for this task.

## Appendix A Appendix on Experimental Settings

Due to computational constraints, we evaluated the performance of the base stage only on the ImageNet-100 and Oxford-IIIT Pets datasets. For all datasets that underwent closed-world stage training, the total number of training epochs is 256, with a batch size of 64 and a learning rate of 0.002. For the experiments for all of the datasets, the masking threshold is set to $0.5$ uniformly. $\mathcal{C}_{base}$ is set to $10$ and $\mathcal{S}_{base}$ is set to $32\times 32$ which conforms to the resolution of single images of CIFAR-10. $\mathcal{C}$ and $\mathcal{S}$ are corresponding parameters of the estimated dataset.
CIFAR-10 dataset: In the context of the CIFAR-10 study, our methodology is evaluated using the setting: $N_{1}=500,H_{1}=4000,M_{1}=4500$. We establish the three kinds of imbalance ratios at $\gamma_{k}^{l}=\gamma_{k}^{u}=\gamma_{n}^{u}=100$. Moreover, maintaining a constant $\gamma_{k}^{l}=\gamma_{k}^{u}=100$, we further explore our approach under varying conditions $\gamma_{n}^{u}=1/100$ and $M_{1}=M_{2}=...=M_{c_{n}}=1500$, to simulate both reversed and uniform distributions of unlabeled novel-class data classes. Besides, for the remaining experimental parameters $\lambda_{1}=0.5$, $\lambda_{2}=0.5$, $\tau_{1}=2$, $\tau_{2}=2$, $\alpha=1.2$ and $\beta=0.8$.
CIFAR-100 dataset: In the context of the CIFAR-100 study, our methodology is evaluated using the setting: $N_{1}=50,H_{1}=400,M_{1}=450$. We establish the three kinds of imbalance ratios at $\gamma_{k}^{l}=\gamma_{k}^{u}=\gamma_{n}^{u}=100$. Moreover, maintaining a constant $\gamma_{k}^{l}=\gamma_{k}^{u}=100$, we further explore our approach under varying conditions $\gamma_{n}^{u}=1/100$ and $M_{1}=M_{2}=...=M_{c_{n}}=150$, to simulate both reversed and uniform distributions of unlabeled novel-class data classes. Besides, for the remaining experimental parameters $\lambda_{1}=0.5$, $\lambda_{2}=0.5$, $\tau_{1}=1$, $\tau_{2}=1$, $\alpha=1.05$ and $\beta=0.95$.
ImageNet-100 dataset: In the context of the ImageNet-100 study, our methodology is evaluated using the setting: $N_{1}=75,H_{1}=600,M_{1}=675$. We establish the three kinds of imbalance ratios at $\gamma_{k}^{l}=\gamma_{k}^{u}=\gamma_{n}^{u}=100$. Moreover, maintaining a constant $\gamma_{k}^{l}=\gamma_{k}^{u}=100$, we further explore our approach under varying conditions $\gamma_{n}^{u}=1/100$ and $M_{1}=M_{2}=...=M_{c_{n}}=225$, to simulate both reversed and uniform distributions of unlabeled novel-class data classes. Besides, for the remaining experimental parameters $\lambda_{1}=0.8$, $\lambda_{2}=0.2$, $\tau_{1}=1$, $\tau_{2}=1$, $\alpha=1.05$ and $\beta=0.95$.
Tiny ImageNet dataset: In the context of the Tiny ImageNet study, our methodology is evaluated using the setting: $N_{1}=50,H_{1}=400,M_{1}=450$. We establish the three kinds of imbalance ratios at $\gamma_{k}^{l}=\gamma_{k}^{u}=\gamma_{n}^{u}=10$. Moreover, maintaining a constant $\gamma_{k}^{l}=\gamma_{k}^{u}=100$, we further explore our approach under varying conditions $\gamma_{n}^{u}=1/100$ and $M_{1}=M_{2}=...=M_{c_{n}}=150$, to simulate both reversed and uniform distributions of unlabeled novel-class data classes. Besides, for the remaining experimental parameters $\lambda_{1}=0.5$, $\lambda_{2}=0.5$, $\tau_{1}=1$, $\tau_{2}=1$, $\alpha=1.2$ and $\beta=0.8$.
Oxford-IIIT Pet dataset: In the context of the Oxford-IIIT Pet study, our methodology is evaluated using the setting: $N_{1}=20,H_{1}=60,M_{1}=80$. We establish the three kinds of imbalance ratios at $\gamma_{k}^{l}=\gamma_{k}^{u}=\gamma_{n}^{u}=10$. Moreover, maintaining a constant $\gamma_{k}^{l}=\gamma_{k}^{u}=100$, we further explore our approach under varying conditions $\gamma_{n}^{u}=1/100$ and $M_{1}=M_{2}=...=M_{c_{n}}=20$, to simulate both reversed and uniform distributions of unlabeled novel-class data classes. Besides, for the remaining experimental parameters $\lambda_{1}=0.5$, $\lambda_{2}=0.5$, $\tau_{1}=1$, $\tau_{2}=1$, $\alpha=1.05$ and $\beta=0.95$.
SVHN dataset: In the context of the SVHN study, our methodology is evaluated using the setting: $N_{1}=500,H_{1}=4000,M_{1}=4500$. We establish the three kinds of imbalance ratios at $\gamma_{k}^{l}=\gamma_{k}^{u}=\gamma_{n}^{u}=100$. Moreover, maintaining a constant $\gamma_{k}^{l}=\gamma_{k}^{u}=100$, we further explore our approach under varying conditions $\gamma_{n}^{u}=1/100$ and $M_{1}=M_{2}=...=M_{c_{n}}=1500$, to simulate both reversed and uniform distributions of unlabeled novel-class data classes. Besides, for the remaining experimental parameters $\lambda_{1}=0.5$, $\lambda_{2}=0.5$, $\tau_{1}=2$, $\tau_{2}=2$, $\alpha=1.2$ and $\beta=0.8$.

## Appendix B OSSL method performance in ROLSSL settings.

To better observe the impact of the ROLSSL setting on previous OLSSL methods, we visualize the classification performance of the OpenLDN and our proposed DPLA methods on the SVHN dataset as epochs increase. In this Figure 5, circles and triangles represent DPLA and OpenLDN, respectively, while blue, yellow, and green represent the accuracy of known classes, novel classes, and all classes, respectively. It can be seen that for known classes, the accuracy of OpenLDN gradually decreases as epochs increase, whereas the accuracy of our proposed DPLA steadily rises. For the prediction of novel classes, OpenLDN’s performance approaches zero, while DPLA significantly outperforms OpenLDN, although it exhibits considerable fluctuations. Overall, OpenLDN performs very poorly under our proposed ROLSSL setting, especially in recognizing novel classes, while our proposed DPLA method far surpasses OpenLDN and effectively recognizes novel classes.

Figure: Figure 5: OSSL method performance in ROLSSL settings.

## Appendix C Appendix on Pseudo-Algorithm for ROLSSL

Here we provide pseudo-code for the method proposed in this paper to clarify the steps of it.

Figure: Algorithm 1 Dual-Stage Post-hoc Logit Adjustment for ROLSSL

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See Weaknesses.
soundness: 2
strengths: 1. The combination of long-tailed, semi-supervised, and open-world learning is interesting.
summary: The paper addresses the limitations of existing open-world long-tailed semi-supervised learning (OLSSL) algorithms, which assume identical distributions between known and novel categories. It introduces a more realistic open-world long-tailed semi-supervised learning (ROLSSL) setting, where no assumption is made about the distribution relationship between known and novel classes. The authors propose a novel approach called dual-stage post-hoc logit adjustments, which dynamically adjusts predictive probabilities to mitigate category bias during learning. The method takes into account the frequency of samples, the number of categories, and the size of the data, improving performance by more selectively utilizing imbalanced unlabeled data. Experiments on CIFAR100 and ImageNet100 show some improvements in performance, demonstrating the effectiveness of the proposed method and setting a strong baseline for future research.
weaknesses: 1. The writing of this paper is not very clear. For example, 'S/N Consistency' in Tab. 1, a key feature of the proposed setting, is never explained.
2. Open-world Long-tailed Semi-supervised Learning seems quite similar to [*1], which is not discussed in related works. Therefore, the novelty of this paper may be weak.
3. Experiments may be insufficient. Only some tiny datasets are considered. Large-scale benchmarks (e.g., ImageNet-LT or iNaturalist) should be included.


[*1] SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning. ICML'24

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How to handle the scenarios where the number of new classes is unknow? It seems that the proposed method requires the prior class number, which might not realistic.
2. Could the authors summarize the main novelty intuitively?
soundness: 2
strengths: 1. This paper is well-motivated and easy to follow.
2. The proposed problem setting is realistic and reflects the long-tailed nature of the real world.
3. This paper conducts comprehensive experiments to show the superiority of the method.
summary: This paper studies a realistic open-world setting, namely open-world long-tailed semi-supervised learning (ROLSSL), where the distribution relationships between known and novel categories might be different. To solve this problem, the paper proposes dual-stage post-hoc logit adjustments. Specifically, it estimates the distributions of the unlabeled data and utilizes the estimations to re-adjust the logits. Comprehensive experiments validate the superiority of the method.
weaknesses: 1. The comparative methods are outdated, and the most recent comparative method of this paper is ECCV 2022. In the literature, OSSL is also referred to as generalized category discovery (GCD). Some recent GCD methods, including GCD [R1] and SimGCD [R2], should be included in Table 2 and 3. There are also some works [R3, R4] in GCD considering long-tailed scenarios, which are highly related to this work and should be compared.
2. The novelty of this paper is limited. The basic loss functions $L_{pair}$ and $L_{reg}$ are well-known techniques in novel class discovery and GCD, while the logit adjustments are also prevalent methods in long-tailed classification. Simply applying model predictions in logit adjustments is naïve and might not provide great insights to the community.
3. The method still requires strong priors of the ground truth new class numbers, which is unrealistic.
4. Experiments with the ViT backbone should be conducted.
5. The notations should be concise and clear.

References:
[R1]. Generalized Category Discovery. CVPR 2022.
[R2]. Parametric Classification for Generalized Category Discovery: A Baseline Study. ICCV 2023.
[R3]. Novel Class Discovery for Long-tailed Recognition. TMLR 2023.
[R4]. Generalized Categories Discovery for Long-tailed Recognition. ICCV 2023, workshop.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Have you considered running experiments on iNaturalist? It has more classes than any of the benchmarks used in this paper and presents a more challenging fine-grained, long-tailed distribution.
soundness: 3
strengths: 1. The problem addressed by this paper is highly practical, as determining the distributional relationships between known and novel categories is indeed nearly impossible in real-world scenarios.

2. The paper is well-written and easy to follow.

3. Key experimental results demonstrate that the DPLA approach consistently outperforms the OpenLDN baseline in both known and novel class recognition, achieving up to a 50.1% improvement on datasets like CIFAR-100 and ImageNet-100. Authors further showcase the robustness and adaptability of the proposed method across varying levels of data imbalance.
summary: The paper introduces a approach called Dual-stage Post-hoc Logit Adjustments (DPLA) to tackle the challenges in Realistic Open-world Long-tailed Semi-supervised Learning (ROLSSL). DPLA is a two-stage logit adjustment technique designed to improve model performance on both frequent and infrequent (tail) classes.  In the first stage, logits are adjusted based on the sample frequency and class count, which helps balance predictions across known classes. In the second stage, adjustments are dynamically refined using the predicted class distribution from unlabeled data, with a particular focus on emphasizing tail classes to counter bias towards dominant classes. This dual-stage approach enables better recognition in long-tailed, semi-supervised, and open-world settings, especially where labeled data is sparse and imbalanced. Key experimental results highlight that the DPLA approach consistently surpasses the OpenLDN baseline in both known and novel class recognition, achieving up to 50.1% improvement experiments on datasets such as CIFAR100 and ImageNet100. They also show the robustness and adaptability of the proposed method across different data imbalance conditions.
weaknesses: 1. A key concern is the complexity of the proposed pipeline, which involves multiple stages and numerous hyper-parameters. With additional stages and hyper-parameters to tune, it becomes more challenging to generalize this pipeline to new problem settings and datasets. For instance, as shown in Tables 3 and 4, using non-optimal scaling factors can significantly impact performance on CIFAR-100 and ImageNet-100, despite both datasets having the same number of classes. Have you explored methods to automatically tune the hyperparameters or analyzed which parameters are most sensitive? 

2. Technical contributions. A primary contribution of this paper is the proposed logit adjustment method for pseudo-labels, which often exhibit class imbalance. However, the issue of naturally imbalanced pseudo-labels has been previously addressed by DebiasPL [1], which also applies a logit adjustment based technique to mitigate class imbalance in pseudo-labels, resulting in improved classification performance on long-tailed datasets. A discussion on the technical distinctions between this approach and DebiasPL would be beneficial.

3. While this paper presents experiments on several benchmarks to demonstrate performance gains, most datasets used are small to medium-scale, such as ImageNet-100 or CIFAR. I believe it is essential to include experiments on larger datasets, such as iNaturalist or ImageNet-1k or ImageNet-22k, to better assess the generalizability of the proposed method. It will be nice if you could provide any potential challenges or modifications needed to apply your method to larger datasets.

4. In addition to presenting results on novel and known subsets, I believe it is essential to include performance metrics for many-shot, medium-shot, and few-shot subsets. This would help clarify whether the current logit adjustment methods compromise performance on many-shot classes to improve results on few-shot classes—a common issue with most logit adjustment techniques. It would be valuable for the authors to discuss whether their proposed method faces this challenge and, if not, to explain why it avoids this problem. If it does, providing insights into potential strategies to address this limitation in future work would be beneficial. 

5. I may suggest authors to include a specific breakdown of performance metrics for these subsets in their results tables. Add a discussion section that explicitly addresses the trade-offs between performance on different shot categories and potential strategies to balance these trade-offs. In my opinion, while improving performance on few-shot classes is important, it is equally crucial to avoid jeopardizing performance on many-shot classes, as they are more prevalent in real-world applications. 

[1] Wang, Xudong, Zhirong Wu, Long Lian, and Stella X. Yu. "Debiased learning from naturally imbalanced pseudo-labels." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657. 2022.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: This work may need more effort to refine its writing and the methodology sections need more explanations for equations as mentioned above.
soundness: 3
strengths: SSL in a more realistic setting is proposed and explored in this work.
summary: This paper mainly focuses on the more challenging SSL setting - open-world long-tailed SSL (OLSSL). The authors propose a more realistic setting than the existing OLSSL: labeled samples are much less than the unlabeled ones. Based on the existing OLSSL method, this work proposes a dual-stage post-hoc logit adjustment method to calibrate and optimize the model. Several experiments are conducted for verification.
weaknesses: 1. Since readers may not be quite familiar with the development of open-world long-tail SSL, it is suggested that more background about the existing OLSSL algorithms be included in Section 3.2, especially for the definition of Eq(1).
2. The dual-stage post-hoc logit adjustment section is not easy to read. Specifically, 
- why the aremax is used in Eq(2)? According to the Fig 1, it is expected to be a distribution adjustment. 
- What is the definition of F_{y_i^l} in Eq(3)? If the labeled samples are limited, will this sample frequency be biased? How C_base and S_base are defined? 
- What is the relationship of the equation in L244 and the Eq(2)? 
- for unlabeled samples, how the sample ratio \pi_c^r in Eq(4) is obtained when the class of the unlabeled samples is unknown? How alpha and beta are defined? What is the meaning of (alpha-beta) in Eq(4)?
3. Does the \tau in Eq(7) denote the ones in Eq(2)? If so, why the operation become plus but not minus as in Eq(2)? 
4. The proposed method is argued to be a two-stage method, does it mean training labeled and unlabeled samples in different stages? However, according to Eq(8), the model seems optimized on labeled and unlabeled samples together.
5. In Figure 1, the right side of the diagram shows a question mark ("?") symbol, does it represent the unlabeled samples? If so, why does the labeled branch also include a depiction of unlabeled samples? Is there a particular significance or reason for this choice in the illustration?
6. The paper seems primarily focused on addressing the long-tailed problem while offering little innovation for the open-world problem.
7. In the defined Realistic Open-World Long-Tailed Semi-Supervised Learning (ROLSSL) framework, it is assumed that the number of novel classes is known (L200). The proposed method leverages this prior knowledge for model calibration (Eq(3)). However, this assumption seems somewhat unrealistic, as it is uncommon to have prior knowledge of the exact number of novel classes present in the unlabeled samples.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The analysis of hyperparameters $\tau_1$ and $\tau_2$ are missing.
soundness: 2
strengths: 1. The studied problem in this paper is practical and underexplored. Most previous studies focus on long-tailed or open-world semi-supervised learning, without considering both.
2. The proposed approach is simple and easy to implement. This paper improves previous methods by proposing a dual-stage post-hoc logit adjustment technique, which does not involve complex strategies or additional learnable parameters.
3. Experiments on six datasets show the advantage of the proposed approach against previous methods.
summary: This paper aims to solve a complex task in semi-supervised learning, i.e., the long-tailed label distribution and the presence of novel classes in unlabeled data. The paper presents a simple solution for this task based on techniques such as logit adjustment, and entropy maximization.
weaknesses: 1. The novelty of the proposed method seems limited. As stated in the *Strengths*, the proposed method is simple and easy to implement because it is pretty much built upon existing techniques. However, this also makes the contribution of the method minor.

2. The rationale behind the method is not well-explained. It is unclear why the logit adjustment strategies for labeled and unlabeled data are different. Also, the design of Eq. (3) is not straightforward and needs in-depth understanding and justification. In Eq. (4), the impact of hyperparameters $\alpha$ and $\beta$ are not studied.

3. The writing needs improvement. For instance, the paper should include the definition of $\mathcal{L}\_{pair}$ and $\mathcal{L}\_{reg}$ to be self-contained.

4. It would be better if the paper could include real-world semi-supervised datasets which follow long-tailed label distribution and also include novel classes in unlabeled data.

5. It seems that the logit adjustment strategy in open-world semi-supervised learning has been considered in [1].

[1] Bridging the Gap: Learning Pace Synchronization for Open-World Semi-Supervised Learning. IJCAI 2024.