RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Data
Authors: Sangwoo Mo
Jong-Chyi Su
Chih-Yao Ma
Mido Assran
Ishan Misra
Licheng Yu
Sean Bell
Venue: ICLR 2023 poster (2023)
Keywords: semi-supervised learning
representation learning
uncurated data

Abstract: Semi-supervised learning aims to train a model using limited labels. State-of-the-art semi-supervised methods for image classification such as PAWS rely on self-supervised representations learned with large-scale unlabeled but curated data. However, PAWS is often less effective when using real-world unlabeled data that is uncurated, e.g., contains out-of-class data. We propose RoPAWS, a robust extension of PAWS that can work with real-world unlabeled data. We first reinterpret PAWS as a generative classifier that models densities using kernel density estimation. From this probabilistic perspective, we calibrate its prediction based on the densities of labeled and unlabeled data, which leads to a simple closed-form solution from the Bayes' rule. We demonstrate that RoPAWS significantly improves PAWS for uncurated Semi-iNat by +5.3% and curated ImageNet by +0.4%.

Published as a conference paper at ICLR 2023
ROPAWS: ROBUST SEMI-SUPERVISED REPRESENTA- TION LEARNING FROM UNCURATED DATA
Sangwoo Mo
Jong-Chyi Su
Chih-Yao Ma
Mahmoud Assran
Ishan Misra
Licheng Yu²
Sean Bell
KAIST NEC Laboratories America
Meta Al
McGill University
Mila
ABSTRACT
Semi-supervised learning aims to train model using limited labels. State-of-the- art semi-supervised methods for image classification such as PAWS rely on self- supervised representations learned with large-scale unlabeled but curated data. However, PAWS often less effective when using real-world unlabeled data that is uncurated, e.g., contains out-of-class data. We propose RoPAWS, robust ex- tension of PAWS that can work with real-world unfabeled data. We first reinterpret PAWS generative classifier that models densities using kernel density estima- tion. From this probabilistic perspective, we calibrate its prediction based on the densities of labeled and unlabeled data, which leads a simple closed-form solu- tion from the Bayes' rule. We demonstrate that RoPAWS significantly improves PAWS for uncurated Semi-iNat by +5.3% and curated ImageNet by +0.4%.
INTRODUCTION
Semi-supervised learning aims to address the fundamental challenge of training models with limited labeled data by leveraging large-scale unlabeled data. Recent works exploit the success of self- supervised learning (He et al., 2020; Chen et al., 2020a) in learning representations from unlabeled data for training scage semi-supervised models (Chen al., 2020b; Cai et al., 2022). In- stead of self-supervised pre-training followed by cupisviee fine-tuning, PAWS (Assran et al., 2021) proposed single-stage approach that combines supervised and self-supervised learning and achieves state-of-the-art accuracy and convergence speed.
While PAWS can leverage curated unlabeled data, we empirically show that not robust to real- world uncurated data, which often contains out-of-class data. A common approach to tackle un- curated data in semi-supervised learning is to filter unlabeled data using out-of-distribttion (OOD) classification (Chen et al., 2020d; Saito al., 2021; Liu et al., 2022). However, OOD filtering meth- ods did not fully utilize OOD data, which could be beneficial to learn the representations especially on large-scale realistic datasets. Furthermore, filtering OOD data could be ineffective since in-class and out-of-class data are often hard to discriminate in practical scenarios.
To this end, propose RoPAWS, a robust semi-supervised learning method that can leverage curated unlabeled data. PAWS predicts out-of-class data overconfidently in the known classes since it assigns the pseudo-label nearby labeled data. To handle this, RoPAWS regularizes the pseudo- labels by measuring the similarities between labeled and unlabeled data. These pseudo-labels are further calibrated by label propagation between unlabeled data. Figure shows the conceptual illus- tration of RoPAWS and Figure 4 visualizes the learned representations.
More specifically, RoPAWS calibrates the prediction of PAWS from a probabilistic view. We first introduce a new interpretation of PAWS as generative classifier, modeling densities over represen- tation by kernel density estimation (KDE) (Rosenblatt, 1956). The calibrated prediction given by a closed-form solution from Bayes' rule, which implicitly computes the fixed point of an iterative propagation formula labels and priors of unlabeled data. In addition, RoPAWS explicitly con- trols out-of-class data by modeling prior distribution and computing reweighted loss, making the model robust to uncurated data. Unlike OOD filtering methods, RoPAWS leverages all of the unlabeled (and labeled) data for representation learning.
Work done during an internship at Meta AI. Correspondence to: swmo @ kak Code: https: / / www.hhc.....c.eteccooccoteeeeeee

---

Published as a conference paper at ICLR 2023
Labeled data
Labeled data
Unlabeled data
Prediction
Prediction
dog
uncertain
D|
(a) PAWS
(b) RoPAWS (ours)
Figure 1:
Conceptual illustration of the proposed RoPAWS. PAWS assigns the pseudo-label of unlabeled data by the nearby labeled data; however, this makes the prediction of out-of-class data overconfident. uncurated setting, unlabeled data contains out-of-class data, for which the model should have uncertain (not confident) predictions. Therefore, RoPAWS regularizes the pseudo-labels by comparing the similarities between unlabeled data and labeled data.
We demonstrate that RoPAWS outperforms PAWS and previous state-of-the-arr (SOTA) robust semi- supervised learning methods in various scenarios. RoPAWS improves PAWS in uncurated data with large margin. In addition, PAWS improves PAWS in curated data by calibrating the prediction of uncertain data. Figure 2 highlights our main results: RoPAWS improves (a) PAWS by +5.3% and previous SOTA by +1.7% on uncurated Semi-iNat (Su & Maji, 2021b), (b) PAWS (previous SOTA) by +0.4% on curated ImageNet (Deng et al., 2009) using 1% of labels, and (c,d) PAWSS by +6.9% and +22.7% on uncurated CIFAR-10 10 (Krizhevsky et al., 2009) using 25 labels for each class, using CIFAR- 100 and Tiny-ImageNe as an extra unlabeled data, respectively.
RELATED WORK
Semi-supervisse learning has been active area of research for decades (Chapelle al., 2006; Van Engelen & Hoos, 2020). Classic works directly regularize the classifier's prediction, which we call the prediction-buxed approach. Recent works focus the final classifier, which we call the representation- based approach. Leveraging the recent progress of self-supervised learning (He et al., 2020), the representation-basse approach has more promise in large-scale scenarios and can be more robust under uncurated data. In the following subsections, we will briefly review the general idea of each approach and how they handle uncurated data."
the representation that stems from
2.1 PREDICTION-BASED SEMI-SUPERVISHI LEARNING
General approach. Prediction- based approaches regularize the classifier's prediction of unlabeled data. Two objectives are popularly used: pseudo-labeling and consistency regularization. Pscudo- labeling (Lee et al., 2013; Yalniz al., 2019; Xie et al., 2020b; Rizve al., 2021; Cascante-Bonill et al., 2021; Hu et al., 2021) predicts labels of unlabeled data and retrains the classifier using the predicted labels, minimizing the prediction entropy of unlabeled data (Grandvalet & Bengio, 2004). Consistency regularization (Sajjadi et al., 2016; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Miyato et al., 2018; Xie al., 2020a) enforces the prediction that views of the same image are similar. Combining two objectives, prediction-based approaches has shown notable results (Berth- elot et al., 2019; 2020; Sohn et al., 2020; Kuo et al., 2020; Li al., 2021). However, they underper- form than representation- based approaches for large-scale scenarios (Chen et al., 2020b). Moreover, most prior works assume that unlabeled data are curated, i.e., follow the same distribution of labeled data, and often fail when unlabeled data are uncurated (Oliver et al., 2018; 8; Su et al., 2021).
Handling uncurated data. Numerous works have attempted make the prediction- based approach robust uncurated data. Most prior works assume the unlabeled data are composed of in- and out- of-domain (OOD) data and filter OOD data by training an OOD classifier (Chen et al., 2020d; Guo et al., 2020; Yu et al., 2020; Huang et al., 2021a;;; Saito al., 2021; Killamsetty et al., 2021; Nair et al., 2019; Augustin & Hein, 2020; Park et al., 10. From the perspective of the prediction approach, it is natural to filter OOD (particularly out-of-class) data since they are irrelevant in-
Bessdees the deep learning approaches, SSKDE (Wang et al., 2009) proposed supervsue kernel density estimation, which technically resembles RoPAWS. The detailed discussion and comparison are in Appendix H.
2

---

Published as a conference paper at ICLR 2023
+5.3
+3..7
43.0
+0.4
66.9
+.6.9
85.6
66.5
82.5
76.7
60.0
WWWS
CESSL
RoPAWS
PAWS
RoPWWS
PAWS
RoPWWS
PAWS
OpenCos
RoP
(a) Semi-iNat (uncurated)
(b) ImageNet
(c) CF-10 (+ CF-100)
(d) CF-10 (+ Tiny-IN)
Figure 2: Test accuracy (%) of PAWS, previous SOTA, and RoPAWS on various datasets. CF stands for CIFAR and Tiny-IN stands for Tmny-ImageNet. We use 1% of labels for ImageNet and 25 labels for each class for CIFAR. RoPAWS highly improves PAWS and outperforms previous SOTA.
class prediction. However, the filtering approach has two limitations: (a) it ignores the representation learned from OOD data, and (b) the detection OOD can artificial. Note that the model cannot distinguish near OOD and in-class data not covered by labels. Therefore, we do not filter the OOD samples but (a) learn representation from them and (b) consider soft confidence of being in-domain. Some recent works reveal that utilizing OOD data can be better than filtering them (Luo al., 2021; Han et al., 2021; Huang et al., 2022). RoPAWS is also in this line of work; however, leveraging OOD data more natural from the representation learning perspective.
2.2 REPRESENTATION
SOMI-SUPERVISED LEARNING
General approach. Self-supervised learning has shown remarkable success recently, learning uni- versal representation that can be transferred to various downstream tasks. Specifically, the siamese network approach, which makes the representation of two views from the same image invariant, has dramatically improved image classification (He et al., 2020; Chen et al., 2020a). Recent works reveal that fine-tuning (or training a lightweight classifier on top of) the self- seff-supervised model per- forms well for semi-supervised learning, especially for high-resolution and large-scale datasets (Zhai al., 2019; Rebuffi et al., 2020; Chen al., 2020b; Cai et al., 2022). However, self-supervised pre- training is inefficient when the target downstream task is given; one can directly train a task-specific model. From this motivation, PAWS (Assran al., 2021) proposed semi-supervised extension of the siamese network approach. enforcing the prediction (computed from representation) of two views invariant. Notably, PAWS has shown be more effective (i.e., performs better) yet ficient (i.e., converges faster) than the two-stage approach of self-supervise pre-training and then fine-tuning. However, PAWS is often not robust uncurated data, limiting its applicability to real- world scenarios. Thus, propose RoPAWS, robust extension of PAWS for uncurated data.
Handling uncurated data. Self-supervised learning robust uncurated data, learning generic representation of data, and does not suffer from the issues of labels such out-of-class, long-tail, or noisy labels (Kang et al., 2020; Tian et al., 2021; Goyal al., 2021; 2022). Thus, fine-tuning a self- supervised model is strong baseline for robust semi-supervised learning. However, claim that semi-supervised learning can maintain its efficacy (and efficiency) while enjoying the robustness of self- self-supervised learning by designing robust semi-supervised method called RoPAWS. On the other hand, some works combine self-supervised and supervised learning, where the supervised loss only applied to the labeled (or confident) data (Yang al., 2022; Khosla et al., 2020). It is safe but too conservative as it does not propagate labels over unlabeled data. This approach is effective for related problem called open-world semi-supert semi-supervised learning, which aims to cluster novel classes while predicting known classes (Cao et al., 2022; Vaze et al., 2022; Rizve et al., 2022; Sun & Li, 2022). In contrast, we focus on predicting the known classes, i.e., no extra burden for novel classes. Extending our work to open-world setups would be an interesting future direction.
3 ROPAWS: ROBUST PAWS FOR UNCURATED DATA
We propose RoPAWS, robust semi-supervised learning method for uncurated data. In the following subsections, we first explain the problem of PAWS (Assran et al., 2021) on uncurated data, then introduce novel probabilistic interpretation of PAWS. Finally, we deseribe our proposed RoPAWS, which calibrates the prediction of targets from the probabilistic view for robust training. Figure visualizes the overall framework of RoPAWS. Detailed derivations provided in Appendix A.

---

Published as a conference paper at ICLR 2023
view 1 (output)
x
fa
p(
labeled data
xP
yf
fo
zf
1
N n=1
CE loss
view 2 (target)
x
fg
X
p
unlabeled data
M 1=1,m=a
Same as PAWS
T
fo 7
New in RoPAWS
Figure 3: Overall framework for RoPAWS. RoPAWS computes the cross-entropy (CE) loss between the predictions of output and target views. Output prediction comes from the similarities to labeled data, same as PAWS. However, target prediction comes from both labeled and unlabeled data, which calibrates the prediction for uncertain data and makes training robust to uncurated data.
3.1 PROBLEM SETUP AND PRELIMINARIES
Problem setup. Semi-supervised learning aims to learn classifier from a large unlabeled dataset Du ;= {ru} and a small labeled dataset Dj == fæı, 31 for {m C. Du. Most prior works assume that Dj and D., are drawn from the same distribution. However, the unlabeled data are often uncu- rated, i.e., containing uncertain (far from labels, often distribution-shifee aim extend the applicability of semi- supi-supervised learning for these uncurated scenarios.
or out-of-class data. We
Preliminaries. In this paper, we focus on semi cumi-suuer for image classification. We start with introducing PAWSS (Assran et al., 2021), which the state-of-the-arr method. PAWS leverages the power of self-supervised learning, particularly the siamese network approach (Caron et al., 2020), to learn useful representation from unlabeled data. Specifically, PAWS enforces the predictions of two views from the same image to be invariant. Unlike self-supervised learning. PAWS defines the prediction using labeled samples. Formally, let B, = {xi, vi } and B, = {x LMd be the labeled and unlabeled batches of size N and M, sampled from Dj and Du,, respectively. Here, the labeled for By := {(r) yf) E Bi yt = y) for all the classes y E [C] = (1, , C). PAWS samples subset of classes for each mini-batch.
batch B3 is class- balanced, i.e., BY = с
N
Given the batches, PAWS first computes normalized embedding = = fo (=)=)/]l[( i.e., |= using encoder f-).) For simplicity, omit the encoder fo(-) in the future equations and denote the normalized embedding by substituting ı with z from the original data x, i.e., fo(x We assume that an embedding Z is X d vector, and z, and z, are N d and M d matrices represent the embeddings of labeled and unlabeled data. Here, PAWS predicts the label p(y|z) of data x by finding the nearest labeled sample 21 and softly assigning the label y1- Note that we compute the similarity on the representation space and p(y) 22) is function of z. Then,
super, 2
super
Peams (y)x) := o, z7] P.
where // E NCC are the corresponding one-hot (or soft) labels with a row-wise sum of 1, and o-(+) is the softmax function with temperature т. PAWS then enforces the prediction of two views pi = p(yltt(==") and ph p(ylt2(2*) from the same unlabeled data x² with augmentations t1, ~ T to be identical. Additionally, the sharpening function p(-) is applied to targets which detached from backpropagation. PAWS further regularizes the average sharpened prediction p: 2M (((p)) + p(p$)) to be uniform. To sum up, the training objective of PAWS is:
Leams = 2M
Σ
(H(PI. H(P p(ri))) H()))
2 E....
where H(..)) is cross-entropy and and H((-) is entropy. Recall that the prediction is computed from the labeled data, and thus the objective is a function of both B, and B,.
Problem with PAWS. The prediction of PAWS is overconfident to given [C] classes since it assigns the label of the nearby labeled data. However, it can be problematic for uncurated data containing oft-of-class data. Specifically, PAWS equally assigns high confidence for both in- and oft-of-class data, unlike RoPAWS assigns high confidence for in-class and low confidence for out-of-class data (see Appendix J). Thus, we aim calibrate the prediction make PAWS robust.
4

---

Published as a conference paper at ICLR 2023
3.2 PROBABILISTIC VIEW OF PAWS
Our key observation is that PAWS can be interpreted as a generative classifier, which estimates the densities the representation space. To this end, we consider the kernel density estimation (KDE) (Rosenblatt, 1956) using the kernel as the similarity on the representation space:"
1 Σ k(2,z') for k(2,2)) = Z exp((x22/ 2//T)
p(x) [D]
where Z is a normalizing constant make 1 fg. A(2, 2') = 1. From the definition, estimate the conditional p(zly) and marginal p(x) densities using the labeled batch of each class By and the entire labeled batch Br, respectively. Formally, the densities PAWS are:
Peaus (xly)
1
k(2,2") and PEANS (x) x'EBY
Σ
1
Σ
k(z,z).
(4)
B
B
z'EB.
Applying them to the Bayes' rule, we can compute the prediction p(y|æx):
Peaus (3/2x) PeANs (3 p(y) EBy h(z,²')
exp(z-2//T) o (- 2J] pp" EB, exp(x - k/TT
(5)
PeaNs (x)
Es, k(2,2))
BY
where p are one-hot labels. Remark that it recovers the original prediction of PAWS in Eq. (1).
We further extend the densities consider soft labels, e.g., PAWS applies label smoothing (Müller et al., 2019), using the weighted kernel (2, 2') = K p(ylæ') k(2,='):
Peans (xly) NK
Σ p(ulx') k(z,z') and E'EB
Peass (x)
1
d (6)
Σ
k(2,z')
NK
E'EB:
assuming that the normalizing constant K is identical for all classes (class-balanced), Then, this definition also recovers the original prediction of PAWS considering soft labels:
Σ exp(z-2//7) B Es, exp(2 //TT) p(yl=') = 0,(2 zT) 1/
Peans soft (y]x)
To summarize, PAWS a generative classifier, and one can estimate the density of a sample on the representation space learned by PAWS. This probabilistic view gives principled way extend PAWS for various scenarios, c.g., robust training on uncurated data, as stated below.
3.3 ROBUST PAWS VIA CALIBRATED PREDICTION
Recall that the prediction of PAWS is overconfident to the known classes [C] since propagates labels from nearby labeled data. Assuming that also know the labels of unlabeled data, which have less confident values for uncertain (or out-of-class) data, we can calibrate the prediction by considering both labels of nearby labeled and unlabeled samples. However, do not know the labels of unlabeled data, and using them for prediction raises chicken-and-egg problem.
Interestingly, the probabilistic view gives the solution: from the Bayes' rule, the prediction pa. (y/ar) of unlabeled data is function of densities p., (xly) and (z). which are again a function of labels pr(y) æ²) of labeled data and predictions Pu (y|æz) of unlabeled data. Solving the equation, we can get a closed-form solution of P., (3/2 ())z) as function of labels pi (v(yl
(y(zz) (z) and distances of embeddings zı, zu. To this end, we first extend the densities in Eq. (6) to consider both labeled and unlabeled data:"
Phopaus (xlw)
Σ BUBB
p(ul=') k(2,z") and Prespaces (Ir)
1 K
Σ BJUB.
k(z,z'). C
K
'Lee al. (2018b) suggested generative classifier view of deep representations using the Gaussian mixture model (GMM). However, GMM needs enough labeled samples, and KDE fits more few label setting.
N "Rccall that the labeled batch is class-halanced, hence | = N, |88| 2 and p(y) = for all y- We balance the scale of laheled and unlabeled data with ratio when the unlaheled hatch is large.
5

---

Published as a conference paper at ICLR 2023
where K' = (N M)K is the normalizing constant. Applying them to the Bayes' rule:
PRoPEES PRoPxes(u( (u]z) = (z [z] 2.])
[p]
|T C C(v)
(9)
= =8 p p(v) + 8u p = C p(y)
where [s||s=] ;= oz (z [ E R RIx(N+M) are similarities of a sample z to labeled and unlabeled data. Aggregating the predictions of unlabeled data, we can formulate the equation in a matrix form:
Pd = (st ② C pl))
p/ + (s. O C' P.) pM
(10)
where S/ S., are matrix versions of $1; Su. P is a matrix version of p(y). and @ is an element-wise product. By organizing the formula, we get the prediction ((y|2)) of RoPAWS:"
q == - -P) (s: O P E RRC
(11)
This formula can also be interpreted as label propagation (Zhou et al., 2003; Bengio et al., 2006; Iseen et al., 2019) of labels pf and priors P of unlabeled data to get the posterior pa 2/
In-domain prior. Considering both nearby labeled and unlabeled samples gives a better calibrated prediction but has a threat of amplifying the wrong prediction of unlabeled data. Indeed, propagating the labels from (certain) labels and (uncertain) predictions of unlabeled data equally can be too risky. Thus, we further improve the prediction formula by putting more weights on certain samples.
This modification can be naturally done under the proposed framework by controlling the prior p(y) to consider out-of-class data. Specifically, we give data-dependent prior of being in-domain (i.e., close to labeled data) for unlabeled data. Let p g4xx1 be the in-domain prior for each sample. Then, we set the prior belief p(y) of in-class as P /C (class-balanced). Here, we only adopt the prior belief of unlabeled data, and the densities p(æ) y) and p(z) remains the same as in Eq. (8). Thus, applying the adopted belief of the prior gives the regularized prediction formula:
q = (I- s. ④ p -1 (st p..) p.
(12)
The final remaining step to estimate the in-domain prior p Here, we assume that data is likely to be in-domain if it is close to some labeled data. To this end, we measure the similarity between data x and labeled data {=1} using the kernel defined in Eq. (3). However, we normalize the kernel to make the similarity between z and xf itself to be 1 (i.e., in-domain prior of xf is 1) and use the different temperature Tprior which controls the overall confidence of in-domain prior; use lower Tprioe for when data are more uncertain. Putting it all together, the in-domain prior given by:
2 (2.2i--)) E (0, 1]
p(in x) == max exp #
(13)
Tprior
where pi is matrix version of p(nn(z)). Note that p(in(x;)) exp(0) = 1.
We remark that setting p(in < implicitly assigns the remaining probability to the background class. However, unlike the background class enforcing the uncertain samples to be collapsed into a few regions (Dhamija et al., 2018), RoPAWS implicitly considers them and all computations are done in the label space of [C]. Here, the prediction q(y(x) E RC from Eq. (12) has the sum 1, where this sum gives a posterior probability of being in-domain ginn/z). We convert this prediction (y)2)) to the normalized C-class probability by adding a uniform multiplied by ((n))r).. i.e., uncertain samples are close to uniform (Lee et al., 201 8a), gives the final prediction values.
Training objective. Recall that the training objective of PAWS is to enforce the prediction of view (ouiput) to be identical to the prediction of view 2 (target). Here, apply the calibrated prediction in Eq. (12) only to the target (denoted as q²) and keep the original prediction Eq. (1) for the output (denoted as p²). We use the same inference (soft nearest-neighhor or fine-tuning) with PAWS, meaning that we keep the output the same but only calibrate the target.
RoPAWS regularizes the prediction of uncertain (or out-of-class) data to be uniform, which may make the prediction too unconfident. To avoid this, we give less weight to the objectives of uncertain
"We do not cancel out C and p(y) since we will control the prior p(y) later, to consider out-of-class data. We denote the posterior prediction of RoPAWS g(ylar) to discriminate with PAWS later.

---

Published as a conference paper at ICLR 2023
Table I: Test accuracy (%) of Semi-SI. methods trained Semi-iNat using ResNet-50, using either a curated or uncurated (superset of curated) unlabeled data. Subscripts denote the gain over PAWS. RoPAWS improves PAWS for all scenarios, and shows that larger uncurated data outperform curated data. Also, RoPAWS outperforms previous tfte-offthee-rt methods initialized from MoCo-v2.
From scratch
ImageNet pre-trained
Method
Curated
Uncurated
Curated
Uncurated
Pseudo-label
18.6
18.8
40.3
40.3
FixMatch
13.5
11.0
44.1
38.5
Self-training
20.3
19.7
42.4
41.5
FixMatch + CCSSL
31.2
MoCo-v2 + Fine-tuning
30.2
31.8
41.7
40.8
MoCo-v2 + Self-training
32.0
32.9
42.6
41.5
MoCo-v2 + FixMatch CCSSL
41.3
PAWS
39.2
37.7
49.7
43.0
RoPAWS (ours)
39.4 (+0.2)
43.0 (45.3)
50.3
(40.6)
44.3 (+1.3)
Table 2: Test accuracy (%) of Se methods trained on ImageNet using ResNet-50. We compare prediction-based Semi-SL, Self-SL pre-training and fine-tuning, and representation-basee Semi-SL approaches. Subscripts denote the gain over PAWS. RoPAWS achieves thee-rff
Semi-SL (pred-based)
Self-SL Fine-tune
Semi-Si (rep-based)
UDA
FixMatch
MPL
BYOL
SwAV
SimCLRv2
PAWS
RoPAWS (ours)
Label 1%
53.2
53.9
57.9
66.5
66.9 (+0.4)
Label 10%
68.1
71.5
73.9
68.8
70.2
68.4
75.5
75.7 (+0.2)
samples (Ren et al., 2020). We simply set the weight w/ as the average in-domain posterior of two views w (g((l=)) + q(vl=)))) where & controls the power of reweighting.
To sum up. the objective of RoPAWS is (differences from PAWS Eq. (2) are highlighted in red):
LROZANE
Σ
w (H(P). H(p p(di)) (p).
(14)
2M s E
EXPERIMENTS
Setup. We follow the default setup of PAWS: ImageNet (Deng et al., 2009) setup on ResNet- 50 (He et al., 2016) for large-scale experiments, and CIFAR-IO (Krizhevsky et al., 2009) setup on WRN-28-2 (Zagoruyko & Komodakis, 2016) for small-scale experiments. RoPAWS has three hy- perparameters: the scale ratio of labeled and unlabeled batch the temperature for in-domain prior Tprir and the power of reweighted loss h: We set r = for all experiments, and set Tprior 3.0 for ResNet-50 and Tprior = 0.1 for WRN-28-2. We set k = 1 for all experiments except & = 3 for fine- tuning from ImageNet pre-trained models. To evaluate, we use the soft nearest- nearest-neighbor classifier for Semi-iNat and CIFAR-I0 and accuracy after fine-tuning for ImageNet. All the reported results use the same architecture, ResNet-50 for large-scale and WRN-28-2 for small-scale experiments. See Appendix C for additional details and discussions the choice of hyperparameters.
4.1 LARGE-SCALE EXPERIMENTS
Semi-iNat. Semi-iNat (Su & Maji, 2021b) is realistic large-scale benchmark for semi-supervised learning, containing a labeled dataset of 9,721 images with 810 classes, along with two unlabeled datasets: (a) a small curated one of 91,336 images with the same 810 classes and (b) a large uneu- rated one of 313,248 images with 2,439 classes. We compare RoPAWS (and PAWS) with various baselines: Pseudo-label (Lee al., 2013), FixMatch (Sohn et al., 2020), Self-training (Hinton al.,
RRPAAS does not suffer from representation collapse similar PAWS, discussed in Appendix B.

---

Published as a conference paper at ICLR 2023
Test accuracy (%) of Semi-SI. methods trained CIFAR-10 using WRN-28-2, using an extra unlabeled data denoted by (+ extra), using (a) no extra data, (b) CIFAR-100, or (c) Tiny- ImageNet as the extra data. Subscripts denote the gain over PAWS, RoPAWS improves PAWS for all scenarios and is comparable with the previous state-of-the-an robust Semi-SL methods.
Table 3:
(a) CIFAR-I0 (curated, no extra data)
(b) CIFAR-IO CIFAR-100)
labels
400 labels
25 labels
400 labels
FixMatch
94,9+07
95.7=0.1
78.1
94.1
OpenMatch
51.6
95.1
41.9
94.1
PAWS
92.4+2.1
95. 15.55
78.7=2.9
91.2+003
RoPAWS (ours)
94.7+0.3 (+2.3)
95.7=022 (+0.6)
85.6 (+6.9)
93,9=0.1 (+2.7)
(c) CIFAR-I0 (+ Tiny-ImageNet)
SimCLR pre-trained
From scratch
FixMatch
UASD
D
OpenMatch
OpenCos
PAWS
RoPAWS (ours)
25 labels
7..5
74.0+04
69.1
77.9
82.5
60.4
83.1 (+22.7)
2014), and CCSSL (Yang et al., 2022), trained from scratch pre-trained MoCo-v2 (Chen et al., 2020c). We include the results reported in CCSSL and Semi-iNat (Su & Maji, 2021a).
Table 1 presents the results Semi-iNat. RoPAWS achieves 43.0% of accuracy for the uncurated, from scratch setup, giving +5.3% of gain over PAWS and improving the state-of-theeart by +1.7%. Note that the prior art, CCSSL requires pre-training of MoCo-v2 and takes longer training time. Also, remark that RoPAWS trained on uncurated data from scratch exceeds the one trained on cu- rated data, verifying the benefit of representation learned from large uncurated data. In addition, RoPAWS improves the curated setups since it regularizes the prediction of uncertain in-class sam- ples." 9 Finally, RoPAWS further boosts the accuracy when using an ImageNet pre-trained model. Note that learning representations for the downstream task is still important when fine-tuning a model (Reed et al., 2022), and RoPAWS gives an effective solution for this practical scenario.
ImageNet. Following prior works, we provide (curated) ImageNet (Deng et al., 2009) experiments using the full ImageNet as unlabeled data and its 1% or 10% subset as labeled data. We compare RoPAWS and PAWS -ftte--of-hhe-ar) with various baselines: (a) prediction-based semi-supervised learning (Semi-SL) such as UDA (Xie et al., 2020a), FixMatch (Sohn et al., 2020), and MPL (Pham et al.. 2021), and (b) pre-training with self-supervised learning (Self-SL) and fine-tuning such as BYOL (Grill et al., 2020), SWAV (Caron et al., 2020), and SimCLRv2 (Chen et al., 2020b). PAWS and RoPAWS fall into the representation-based Semi-SL. Table 2 presents RoPAWS outperforms PAWS and achieves ttaee-of-the-art, e.g., +0.4% for 1% labels.
4.2 SMALL-SCALE EXPERIMENTS
CIFAR-10. We consider three settings for CIFAR-10: using (a) no extra data (curated setting), (b) 100 (hard OOD), and (c) Tiny-ImageNet (easy OOD), as extra unlabeled data. We com- pare RoPAWS with PAWS and FixMatch (Sohn et al., 2020), together with of robust Semi-SL (OOD filtering) methods: UASD (Chen et al., 2020d), D (Guo et al., 2020), Open- Match (Saito et al., 2021), and OpenCos (Park et al., 2021). We use the reported results from the FixMatch paper for setup (a) and the OpenCos paper for setup (c). For other setups, we reproduce FixMatch and OpenMatch using the default configurations. We use 25 and 400 labels for each class. We report the mean and standard deviation for 5 runs for (a) and (b) and 3 runs for (c).
Table 3 presents the results 10. RoPAWS improves PAW'S for all scenarios, especially when using extra uncurated data. For example, RoPAWS improves PAWS by +22.7% for the easy OOD (+ ImngeemageNet) and +6.9% for the hard OOD (+ CIFAR-100) setups using 25 labels.
"See Appendix for additional discussion on why RoPAWS also improves curated data. "oe Appendix F. for additional discussion on the effect of pre-training methods.
8

---

Published as a conference paper at ICLR 2023
Table 4: Ablation study of each component of RoPAWS, trained on Semi-iNat (uncurated).
Semi-sup density In-domain prior
Test acc.
PAWS
37.7
RoPAWS (ours)
43.0
(a) PAWS
(b) RoPAWS (ours)
Figure 4: t-SNE visualization of embeddings of (a) PAWS and (b) RoPAWS trained CIFAR- 10 (+ CIFAR-100) with 400 labels. Colored and black points are from CIFAR-10 and CIFAR-100 respectively. PAWS pushes OOD (CIFAR-100 100) data to the in-class (CIFAR-10) clusters. In contrast, RoPAWS pushes out-of-class data far from all the in-class clusters.
RoPAWS also exceeds the prior state-of-the-arts: Open and OpenCos. These OOD filter- ing methods are effective for easy OOD setups. However, RoPAWS even exceeds OpenCos pre- trained on SimCLR for (+ ImageNmag)Net) without such pre-training. Also, note that the OOD filtering methods are less effective for hard OOD setups (see Appendix L for additional discus- sion). RoPAWS is still effective for (+ CIFAR- 100) since it does not filter samples but learns from uncurated data. We provide additional comparison with robust Semi-SL methods in Appendix G.
4.3 ABLATION STUDY AND ANALYSIS
Ablation study. Table presents the ablation study of the components of RoPAWS, trained Semi- iNat (uncurated). Semi-supervised density estimation in Eq. (11) highly improves the accuracy, and in-domain prior in Eq. (13) (combined with a reweighted loss Eq. (14)) enhances further. We provide additional ablation studies in Appendix D, showing that RoPAWS gives reasonable results over varying hyperparameters and that all the proposed components are essential.
Analysis. We analyze the learned representations of RoPAWS. Figure 4 shows the t-SNE (Van der Maaten & Hinton, 2008) visualization of embeddings, trained on CIFAR-10 (+ CIFAR-100), 400 labels. As PAWS confidently predicts unlabeled data as in-class, OOD data (black points) are con- centrated in the in-class clusters (colored points). In contrast, RoPAWS pushes out-of-class data far from all the in-class clusters. We provide additional analyses in Appendix showing that RoPAWS increases the confidence of in-domain and decreases one OOD data.
Nearest neighbors. We visualize the nearest labeled data of out-of-class data inferred by PAWS and RoPAWS Appendix Even though the classes are different, both models can find visually similar in-class data with slight variation. However, PAWS has high similarity scores to the wrong in-class neighbors, unlike RoPAWS has lower similarities, giving uncertain predictions.
Computation time. The inference time of RoPAWS is identical to PAWS. The training time is only slightly increased (< 5% for each mini- batch) due to the (detached) targets in Eq. (12).
5 CONCLUSION
We proposed RoPAWS, a novel robust semi-supervised learning method for uncurated data, which calibrates prediction via density modeling representation space. We verified the effectiveness of RoPAWS under various scenarios, achieving new state-of-the-al results. addition, the probabilis- tic view of RoPAWS can be extended various problems, discussed in Appendix M.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government (MSIT) (No. N0.9201--0000055 Artificial Intel- ligence Graduate School Program (KAIST) and No. No.2021-0-0206 Artificial Intelligence Innovation Hub). SW thanks to Ser-Nam Lim for supporting technical issues for using fairaws cluster.
ETHICS STATEMENT
Learning representations from large and diverse data makes the model more robust and fair (Goyal al., 2022). However, the uncurated data may inherit the natural bias of data distribution. Thus, the user should not treat the model as a black-box output from arbitrary data but should carefully look the data's 's statisties and be aware of the potential biases.
REPRODUCIBILITY STATEMENT
We provide implementation details in Appendix C. We provide our code in the supplementary ma- terial, which will be publicly released after the paper acceptance.
REFERENCES
Eric Arazo, Diego Ortego. Paul Albert, Noel Connor, and Kevin McGuinness. Pseudo-labelie labeling and confirmation bias in deep semi-supervised learning. In IJNN,, 2020.
Mahmoud Assran, Nicolas Ballas, Lluis Castrejon, and Michael Rabbat. Supervision accelerates pre-training in contrastive semi-supervised learning of visual representations. arXiv preprint arXiv:2006. 10803, 2020.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In ICCV, 2021.
Maximilian Augustin and Matthias Hein. Out-distribution aware self-training in an open world setting. arXiv preprint arXiv:2012.12372 2020.
Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. label propagation and quadratic criterion. 2006.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. In ICIR, 2020.
Andrew P Bradley. The use of the under the curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7): 1455-119,, 1997.
Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers scale. arXiv preprint arXiv:2208.0566
2022.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In ICLR, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revis- iting pseudo-labeling for semi-supervised learning. In AAAI, 2021.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. 2006. Cam- bridge, Masxachusettes: The MIT' Press View Article, 2, 2006.
10

---

Published as a conference paper at ICLR 2023
Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-superviee learners. In NeurIPS, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04220 2020c.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, 2020d.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. In NeurlPS, 2018.
Geoff French, Avital Oliver, and Tim Salimans. Milking cowmask for semi-supervised image clas- sification. arXiv preprini arXiv:2003. 12022, 2020.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint 1.X.i::20030110 2021.
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Mannat Singh, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint urXiv:2202.08360, 2022.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurlPS, 2004.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Berardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
Lan-Zhe Guo, Zhen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi-
supervised learning for unscen-class unlabeled data. In ICML, 2020.
Lu Han, Han-Jia Ye, and De-Chuan Zhan. On pseudo-labeling for class-mismatch semi-supervised learning. 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2014.
Zijian Hu, Zhengyu Yang, Xuefeng Hu, and Ram Nevatia. Simple: similar pseudo label exploitation for semi-supervised classification. In CVPR, 2021.
Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning. In ICCV, 2021a.
Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NewrIPS, 2021b.
Zhuo Huang, Jian Yang, and Chen Gong. They not completely useless: Towards recycling transferable unlabeled data for class class-mismatee mismatched semi-supervised learning. JEEE Transactions Multimedia, 2022.
11

---

Published as a conference paper at ICLR 2023
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In CVPR, 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, 2020.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. In NeurIPS, 2021.
Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In NeurIPS, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Chia-Wen Kuo, Chih-Yao Ma, Jia-Bin Huang, and Zsolt Kira. Featmatch: Feature-based augmen- tation for semi-supervised learning. In ECCV, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning. ICMI., 2013.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--aalibatee classifiers for detecting out-of-distribution samples. In ICLR, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018b.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with con- trastive graph regularization. In ICCV, 2021.
Yen-Cheng Liu, Chih- Yao Ma, Xiaoliang Dai, Junjiao Tian, Peter Vadja, Zijian He, and Zsolt Kira. Open-set semi-supervised object detection. In ECCV, 2022.
Charlotte Loh, Rumen Dangovski, Shivchander Sudalairaj. Seungwook Han, Ligong Han, Leonid Karlinsky, Marin Soljacic, and Akash Srivastava. On the importance of calibration in semi- supervised learning. arXiv preprint arXiv:2210.04788, 2022.
Huixiang Luo, Hao Cheng, Fanxu Meng. Yuting Gao, Ke Li, Mengdan Zhang, and Xing Sun. An empirical study and analysis on open-set semi-supervised learning. arXiv preprint 1rXiv:2101.0823,, 2021.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and remi-supervised leaming. IEEE transactions on pattern analysis and machine intelligence, 2018.
Sangwoo Mo, Hyunwoo Kang, Kihyuk Sohn, Chun-Liang Li, and Jinwoo Shin. Object-aware con- trastive learning for debiased scene representation. In NearIPS, 2021.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In NeurIPS, 2019.
Varun Nair. Javier Fuentes Alonso, and Tony Beltramelli. Realmix: Towards realistic semi- supervised deep learning algorithms, arXiv preprint 19120069/, 2019.
12

---

Published as a conference paper at ICLR 2023
Islam Nassar, Samitha Herath, Ehsan Abbasnejad, Wray Buntine, and Gholamreza Haffari. All labels are not created equal: Enhancing supei-supeniiiio via label grouping and co-training. In CVPR, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop Deep Learning and Unsupervised Feature Learning. 2011.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep sumi-super learning algorithms. In NeurIPS, 2018.
Jongjin Park, Sukmin Yun, Jongheon Jeong, and Jinwoo Shin. Opencos: Contrastive semi- supervised learning for handling open-set unlabeled data. arXiv preprint arX1v:2107.089443 2021.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. In CVPR, 2021.
Carl Rasmussen. The infinite gaussian mixture model. In NeurlPS, 1999.
Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi-supervised learning with scarce annotations. In CVPR Workshop on Deep Vision, 2020.
Colorado J Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, et al. Self-supervised pretraining im- proves self-supervised pretraining. In WACV, 2022.
Zhongzheng Ren, Raymond Yeh, and Alexander Schwing. Not all unlabeled data are equal: Learn- ing to weight data in semi-supervised learning. In NeurlPS, 2020.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-awarr pseudo-label selection framework for semi-supervised learning. In ICLR, 2021.
Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. Towards realistic semi-supervised learning. In ECCV, 2022.
Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics, pp. 832 837, 1956.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. In NeurlPS, 2021.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep cemi-supervised learning. In NeurIPS, 2016.
Burr Settles. Active learning literature survey. 2009.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NewrlPS, 2020.
Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels. In BMVC, 2021a.
Jong-Chy Su and Subhransu Maji. The semi-supervised inaturalist challenge the fgve8 workshop. arXiv preprint arXiv:2 2021b.
long-Chyi Su. Zezhou Cheng, and Subhransu Maji. A realistic evaluation of semi-supervised learn- ing for fine-grained classification. In CVPR, 2021.
Yiyou Sun and Yixuan Li. Open-world contrastive learning. arXiv preprint arX(v:2208.00276. 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. In NeurIPS, 2020.
13

---

Published as a conference paper at ICLR 2023
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaget consis- tency targets improve sumi-supervised deep learning results. In NeurlPS, 2017..
Yonglong Tian, Olivier Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In ICCV, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)) 2008.
Jesper E Van Engelen and Holger H Hoos. A survey semi-supervised learning. Machine Learn- ing, 109(2):373-440, 2020.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In CVPR, 2022.
Meng Wang, Kian-Sheng Hua, Tao Mei, Richang Hong, Guojun Qi, Yan Song, and Li-Rong Dai. Semi-supervised kernel density estimation for video annotation. Computer Vision and Image Understanding, 113(3):384-39, 2009.
Xinlong Wang, Rufeng Zhang. Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. In CVPR, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In NeurIPS, 2020a.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020b.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scal scale semi- supervised learning for image classification. arXiv preprint arXiv: 00546, 2019.
Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng. Class-aware contrastive supervieee learning. In CVPR, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 17X.:.1708.03888 2017.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In ECCV, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In ICCV, 2019.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NewrIPS, 2003.
14

---

Published as a conference paper at ICLR 2023
A DETAILED DERIVATIONS
We provide detailed derivations for the prediction of probabilistic PAWS and RoPAWS.
PAWS (one-hot). We defined the conditional p(zly) and marginal p(x) densities as follows:
and PEsnS (z) == 1
PeAMs (xly)= ==
2(2,2")
k(z,z')
(15)
By
[
S'ESY
E'EB
where k=22, z/) = exp(2 2//T). Recall that [B:] = N. |B|| C. and p(y) C
Note that this definition satisfies the marginalization:
DoAMS (x)
Σ
PrANS (2) p(w)
(16)
y
C Σ
k(2,2))
(17)
N
y
x'
EBE
Σ
k(z,z')
(18)
N
Z'EB"
k(2, 2)
(19)
N
X
EEEB
Applying the definitions to the Bayes' rule,
PraMs (y)z²) = PeANs (zlyy) p(y)
Peams (1)
(00)
C
Σ
k(z,2')
1
Σ
k(2,2')
(21)
C
N
x' E'ERY
E
Σ
1
exp(2x2//T)) /
Σ
exp(z 2² /T)
(22)
Z
Z
E
EB
22/T)
(23)
x EB, exp(z 2-" /T)
EEB
EB
Σ
0-(3 . 27) (1z.)
(24)
r'EBy
Σ
or(2 z7) (1 1 e
(25)
Σ o-(= z/) (1z )xe
(26)
x'EB,
(z
P g/
(27)
where (1 2/) RRXN is similarity vector, 1 E R is an index vector where the index of x' is 1 otherwise 0, 1 is an indicator function, and p g/e E Nx is a (one-hot) label vector.
15

---

Published as a conference paper at ICLR 2023
PAWS (soft). Applying the weighted kernel k" (2,2/) == :/r p(ul2') k(2,2'), the extended definition
of conditional density p(a) y) is as follows:
Soae (x
Σ
(2,2'')
(28)
N
EBEB
Σ
plula') k(2,22)
(29)
NK
E'EB
Then, the marginal density p(z) is given by:
poaes- (I) ==
Σ
Peams-sot:(x1g) p(y)
(30)
y
1
Σ
plyla') k(2,22)
(31)
NK
E'E
1
Σ
Σ plube's) k(2,z²)
(32)
NK
C
e'EB:
1
Σ
k(2,z')
(33)
NK
C
E'E
Applying the definitions to the Bayes' rule,
PeaMs-soce (2/2z) = Peams-sare (zlv) p(v) / Peans-soct (2)
(34)
Σ plyla') k(2,2)) /
Σ NK E.
c M(2,2")
(35)
= NK E'EB
Σ
k(z,z') Σ kerrz)) -plykz")
(36)
s'EB,
r'EB.
Σ
exp(z 22/T// / Σ (xp(2.2217) plyla')
(37)
s' EB
x EB
= 0z 27)
p/ i/
(38)
where py E RN is (soft) label vector.
16

---

Published as a conference paper at ICLR 2023
RoPAWS. We extend the densities to consider both labeled and unlabeled data:
PRoPxSS (x)
Σ
p(ul=') k(z,z') and PROPANS (x)
Σ
1 C
k(2,zz).
(39)
K'
K
C
B,UB,
BLUB
Here, we balance the effect of labeled and unlabeled data with ratio r Specifically, we use
PROPANS (x)p))s(xll))
1
p(ylz')
k(2,2)) +
E
p(ul=") M(e)2))
(40)
K'
E'E
Proeeus (x) = K'
Σ x'EB,
a k(zzz)) E T ((z,22 Z'EB,
(41)
where r' ;= |B balances the overall effect of labeled batch. Similar to PAWS soft labels, satisfies the marginalization and gives a similar prediction formula.
We set F(x') r' LEEB LI'EB., and simply
write:
PROPAMS (x/()
1
Σ
f(x') plula') k(2,2))
(42)
K' R'EBIUB,
1
1 C k(2,z)) Σ i(z')
Pecenus (x)
(43)
K'
r'EB,UB,
Applying the definitions to the Bayes' rule.
Prevass (y)z) = Pronaum (x()) p()) (2)
(44)
Σ 7(x') p(yla') k(2,22) p(v)
NK
#'E8,
(45)
i(x*) 1 k(2,2*)
NK
C
Z'EB,
Σ
F(z') - -)(2,((()))))
Σ at EE:UU
f(x') k(2,2)
z'EBIUB.
(46)
:(()l=')
C(v))
Σ
f(a') exp(2.
Σ a*EBU
i(x") exp(2.22/7)
r'EBUUB p(y|=') C())).
(47)
Recall that do not assume p(y) be uniform nor cancel out with C'to control the in-domain prior. This formula can be simplified softmax (as above) by adding a constant vector r. Specifically, let r E RIxN+M be a vector where the index of x' E B, is log(r') and otherwise Then,
PResAuS (y/z) = 0+(2 (2- (z +T)) pul
C' p(y).
(48)
and splitting the concatenated vectors and p م| (and omitting r) into labeled and unlabeled terms gives the final formula: PRePAMM Prseam((y|2) or [z]] xz]]) [p. y/ pp
C(yy)..
(49)
17

---

Published as a conference paper at ICLR 2023
To up, the prediction of unlabeled data is computed from the predictions of N labeled and M unlabeled data. We stack this for M unlabeled data and then get the matrix formula of prediction for unlabeled data q E R RMCC and labeled data p R NNC with weights R RM*N and b R RMXM given by the similarity matrix. Namely, the simplified version of Eq. (10) looks like:
a a + b.q.
(50)
and organizing the formula gives the closed-form solution:
(I-b) a. and thus, q q= -1 (I-b) b) b)-b)-1..
(51)
which gives the final prediction formula in Eq. (11). Note that (I- - b) is invertible since b is the part of the similarity matrix, i.e., the row-wise sum of 1.
We remark that this formula can be interpreted as the label propagation, i.e., propagating the belief of labels p (and also the priors on q our case) get the posterior prediction of q. Indeed, think about the iterative formula:
q ++ a. + b.
(52)
for i 2 O and initialized from the prior q.. Here, we can think our closed-form solution as the fixed-point of the iteration (it converges at i + oo since ||b]| I, i.c., contraction mapping).
Finally, recall that the prediction (1y[z) (a row from q) can have row-wise the in-domain prior. It implies that the data x is likely to be in-domain with (posterior) probability q Ey {y/lz) and OOD with probability 1 We regularize the prediction to be close to uniform for uncertain (i.e., OOD-like) samples. Here, the final prediction value is g(yl (1 g)/C, which makes the sum of the probability for C' in-class to be one.
1 when using
B NON-COLLAPSING PROPERTY OF REPRESENTATION
One of the main advantages of PAWS is non-collapsing representations. RoPAWS also favors this property. Recall that the proof logic of Proposition in the PAWS paper is as follows:
1. When the representation collapses, the distances between representations d(2, 2;) become iden- tical for all &, in the labeled batch B;.
2. Then, the prediction is given by the average of class labels y;, which becomes uniform distribution due to the balanced sampling.
3. However, the target sharpening makes the target p+ not equal to the uniform distribution; hence, (pHi P) > O, and the model escapes from the representation collapse.
RoPAWSS follows the same logic: the prediction p becomes a uniform distribution when representa- tion collapses, being escaped by step 3 above. The difference from PAWS is that now the prediction the average of true labels y; of labeled batch B; and pscudo-labels 9. of unlabeled batch B,, Here, the initial belief of pseudo-labels ú, becomes uniform distribution by averaging the true labels yi. and averaging this uniform distribution keeps the updated belief uniform after label propagation.
18

---

Published as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
C.I COMMON SETUPS
We follow the default configurations of PAWS. Specifically, we follow the ImageNet configuration for large-scale (Semi-iNat, ImageNet) experiments and the CIFAR-10 configuration for small-scale (CIFAR-10) 10) experiments. We only adopt the number of labels and training epochs for each dataset, as the dataset has a smaller size (longer epochs) or fewer labels (smaller labeled batch).
Training. We sample an unlabeled and a labeled batch for each iteration. For the unlabeled batch, we sample 2 views for targets and 6 additional multi-crop views (8 in total) for outputs. We compute the semi-supervised prediction of RoPAWS for each view independently, and the size of the unla- beled batch (so as computation complexity) does not increase over the number of views. We use the softmax temperature T = 0.1 and sharpen the target with temperature = 0.25, i.c.
p(p):
P
(53)
1/T
Li=1 P
where ps and p(p); denotes the i-th element of probability p. p(p) E RIXC. We apply the mean entropy maximization (me-max) regularization -Hl(p) for all experiments.
For the labeled batch, we sample C' classes (from overall Call > C' classes) for each iteration, where C' depends on datasets, e.g., C' = 960 for ImageNet with Canl = 1000. We sample the same number of data for each class (i.e., class balanced) with 1 or augmentation views. Thus, the overall support size is the number of classes x samples per class number of views.
We apply the same data augmentation with PAWS, SimAug (Chen et al., 2020a), for all experiments. The same augmentation is applicd for both unlabeled and labeled data.
For large-scale experiments, we train a ResNet-50 network using an unlabeled batch of size 4096 on 64 GPUs We use 3-layer multi-layer perceptron (MLP) projection head with 2048 hidden dimen- sions and 2-layer MLP prediction head with 512 hidden dimensions. This projection/predictinn head is widely applied for self-supervised learning (Grill al., 2020). We use the LARS (You et al., 2017) optimizer with momentum value 0.9 and weight decay 10-6 We linearly warm up the learning rate from 0.3 6.4 during the first 10 epochs and apply the cosine schedule afterward.
For small-scale experiments, we train a WRN-28-2 network using an unlabeled batch of size 256 on GPU. We use a 3-layer MLP projection head with 128 hidden dimensions and do use prediction head. We use the LARS optimizer with the same configuration of the large-scale setup but linearly warm up the learning rate from 0.8 to 3.2 and then apply the cosine schedule.
Evaluation. We use a near-nearest neighbor classifier with softmax temperature + = 0.1 (same to the training) for Semi-iNat and CIFAR- 10. We fine-tune the classifier upon the learned representation for ImageNet since it gives a further gain. We follow the fine-tuning configuration of PAWS, training a linear classifier on top of the first layer of the projection head initialized from zero weights. We fine-tune the backbone and linear classifier simultancously, optimizing supervised cross-entropy loss on the small set of labeled data. We apply the basic data augmentations (random crop and hori- zontal flip) and sweep the learning rate from {0.01, 0.02, 0.05, 0.1, 0.2) and epochs from {30, 50), choosing the hyperparameter based on a 12,000 images subset from the ImageNet training set. We use SGD with Nesterov momentum (with value of 0.9) and batch size of 1024. We apply single center-crop augmentation for test samples and report the top-1 accuracy.
C.2 DATASET-SPECIFC SETUPS
Semi-iNat. We train from scratch models for 800 epochs, and ImageNet pre-trained models for 300 epochs. We use 12 classes per batch (thus, 12 x 64 = 768 < 810 classes are used for each iteration in total) and 5 images for each class (Semi-iNat long-tailed, and the tail classes only have labels) with I view. We use the official SwAV weights" for the ImageNet pre-trained experiments, where we also compare them with the supervised weights !2 used in the prior work. We found
"I "torch. hub. 1 load (' facebookresearda htwpeekkreeearrrsswwwaeesee hwwpchhhw/wwwiiimwww resnet50') "Phtpps:/dooe
woad. wtwpdd/yttrrh..gg/ooooooooo. s/ 50 9c8e357.
19

---

Published as a conference paper at ICLR 2023
that SwAV gives a better initialization for PAWS and RoPAWS as the objective (and thus, learned representations) of SwAV is similar to PAWS, and used SwAV for our main experiments.
ImageNet. We train models for 300 epochs, following the default PAWS configuration. We use 15 classes per batch (thus, 15 x 64 960 < 1, 000 classes are used for each iteration in total) and images for each class with 1 view. We the same subset split of PAWS.
We train models for 600 epochs, following the default PAWS configuration. We use the full 10 classes and 64 images for each class with 2 views for 400 labels setup. For 25 labels and 4 labels setups, we decrease the images for each class and increase the number of views to keep the total size of labeled support. Specifically, we use 16 images for each class with 8 views for 25 labels setup and 4 images for each class with 32 views for 4 labels setup.
CIFAR-10.
C.3 HYPERPARAMETERS OF ROPAWS
RoPAWS has three hyperparameters: the ratio of labeled and unlabeled batch r, the temperature for in-domain prior Tprior. and the power of reweighted loss kt.
Choice of r. We use r = 5 for all experiments following the default batch size ratio in CIFAR-IO CIFAR-10 10 has labeled batch size of 10 classes x 64 samples per class x 2 views 1280, and an unlabeled batch size of 256. Thus, the labeled and unlabeled batch ratio is 1280/256 = 5. We observed that using more weights for certain labels is a safer choice. Recall that the original batch ratio of Semi-iNat is 768 x 5 x 1/4096 0.94, giving more weights for unlabeled than labeled data.
is helpful since the prediction
the unlabeled data is uncertain; giving
Choice of Tprior- We set Tpice = 3.0 for ResNet-50 and Tprior = 0.1 for WRN-28-2. Since ResNet-50 has larger latent dimension (2048) than WRN-28-2 (128), the embeddings are more likely differ from the other embeddings, and cosine similarity decreases faster. Thus, we use larger temperature Tprior for ResNet-50 not decay the in-domain prior too rapidly.
Choice of k. We set k = 1 for all experiments except k: 3 for fine-tuning experiments on Semi- iNat, using ImageNet pre-trained models. Note that higher &o makes the model more conservative; focus on confident samples and ignore uncertain samples. Models from scratch need to learn repre- sentation from uncurated data, thus setting lower k. contrast, ImageNet pre-trained models have good representation and correct labels more matter, thus setting higher k.
C.4 OTHER BASELINES
We run FixMatch and Open for CIFAR-10 experiments. Here, we follow the default setup for CIFAR-10 for both models, using the unofficial (but matching the official performance) PyTorch implementation of FixMatch and the official PyTorch implementation of OpenMatch" 14, We did not tune hyperparameters specialized each setup. However, FixMatch was effective for both curated (CIFAR-10) and hard OOD (CIFAR-10- + CIFAR-100) setups, implying that the default choices are already well-tuned. Open was effective for easy OOD (400 labels) but did not work well for hard OOD (25 labels) setups on CIFAR- 10 (+ 100) It can be improved with a better choice of hyperparameters, yet, we remark that the OOD filtering approach (especially the ones discarding samples explicitly) is sensitive to the threshold and is not robust in the OOD setup.
httt:p////gthbb...o///////////// "https: http:///g........oooo/ooooooonn/
20

---

Published as a conference paper at ICLR 2023
D ADDITIONAL ABLATION STUDIES
We provide additional ablation studies for RoPAWS. Table presents the test accuracy of RoPAWS trained on Semi-iNat (uncurated) varying hyperparameters. Specifically, (a) shows the results vary- ing the in-domain Tprior while fixing the reweight power k = 1.0, and (b) shows the results varying the reweight power while fixing the in-domain prior Sprior 3.0. The results are stable under reasonable range of hyperparameters, yet tuning may give further boost.
Table 6 presents an extra ablation study for the components of RoPAWS. We applied reweighted loss when using in-domain prior in Table 4. Indeed, using in-domain prior without reweighted loss harms the accuracy. addition, only using in-domain prior (and reweighted loss) without eemi-supervised density harms the accuracy. It verifies that all the proposed components are essential.
Table 5: Ablation study of hyperparameters of RoPAWS, trained on Semi-iNat (uncurated).
(a) In-domain prior Tprice (k 1.0)
(b) Reweight power (Tprx 3.0)
Tptio
Test acc.
Test acc.
PAWS
37.7
PAWS
37.7
42.6
43.0
2.0
42.4
2.0
42.9
RoPAWS (ours)
RoPAWS (ours)
3.0
43.0
3.0
41.1
5.0
42.5
5.0
40.1
Table 6: Ablation study of each component of RoPAWS, trained on Semi-iNat (uncurated).
Semi-sup density
In-domain prior
Reweighted loss
Test acc.
37.7
34.4
42.3
V
43.0
E EFFECT OF TRE-TNAINING METHOD
Table 7 presents the test accuracy of PAWS and RoPAWS, initialized from ImageNet supervised or SWAV (Caron et al., 2020) pre-trained models. We used SwAV for our main table since it gives better initialization. Nevertheless, RoPAWS initialized from ImageNet supervised model is already better than prior arts using the same pre-trained model. Intuitively, the training objective of PAWS (and RoPAWS) more resembles SwAV and thus has less representation shifts.
Table 7:
Test accuracy (%) of PAWS and RoPAWS trained on Semi-iNat (uncurated), initialized from ImageNet pre- trained models. SwAV gives a better initialization for PAWS than the supervised model; the loss of PAWS resembles of SwAV and shifts less the learned representation.
Pre-trained
Method
Test acc.
PAWS
42,0
ImageNet supervised
RoPAWS (ours)
42.4
PAWS
43.0
ImageNet SwAV
RoPAWSS (ours)
44.3
21

---

Published as a conference paper at ICLR 2023
F VISUALIZATION OF NEAREST NEIGHBORS
Figure visualizes the nearest labeled data and its cosine similarity to queries inferred from PAWS and RoPAWS trained on Semi-iNat (uncurated). Even for the out-of-class data, both models can find visually similar in-class data with slight variation. However, PAWS generates predictions with high confidence (high similarity score) while RoPAWS has lower confidence. For the in-class data, both models generate high confidence predictions and can find the correct labels.
Out-of-class data
In-class data
Query
Nearest labeled data (PAWS)
1.00
0.99
0.98
0.91
0.96
0.98
Nearest labeled data (RoPAWS)
0.87
0.82
0.75
0.74
0.68
1.00
Figure Nearest labeled data and its cosine similarity to queries.
G ADDITIONAL COMPARISON WITH ROBUST SEMI-SL METHODS
We conduct additional experiments comparing with robust SSL algorithms, using different in-class datasets: (CIFAR-14) 10, CIFAR- 100) and out-of-class datasets (SVHN, Tiny-IN), We follow the same configuration of CIFAR-10 experiments in Table 3, except increasing the classes per batch to 100 and unlabeled batch size to 1024 when the in-class dataset C10A We used 25 labels per class, set the labeled batch per class as 16, and reported the average and standard deviation over three runs. The baseline results from the OpenCos (Park al., 2021) paper.
Table shows that RoPAWS consistently outperforms the SOTA robust SSL algorithms under dif- ferent settings. In particular, RoPAWS improves the SOTA accuracy with a large margin (+5.8% in the most challenging scenario of CIFAR-100 (+ Tiny-IN).
Table 8: Test accuracy (%) of various robust Semi-SL methods under various scenarios. We bold the best results. RoPAWS achieves state-of- the results all considered scenarios.
In-class
CIFAR-10
CIFAR- 100
Out-of-class
SVHN
Tiny-IN
SVHN
Tiny-IN
FixMatch (Sohn et al., 2020)
68.0.0.7
7..5
41.7413
46.0+110
PAWS (Assran et al., 2021)
47.1
60.415.3
36,9102
46.0+05
UASD (Chen et al., 2020d)
66.721.0
74.020.4
39.5408
44.61108
RealMix (Nair et al., 2019)
58.2.5.3
69.212.3
44.1+10
47.6:114
L (Guo al., 2020)
50.048.3
69.1
29.0=110
40.2:10
OpenMatch (Saito et al., 2021)
74.7+1..
78.0108
29.0+13
32.2-26
OpenCoS (Park et al., 2021)
79.0
82,5+13
49.2+12
54.0+18
RoPAWS (ours)
81.2=0.4
83.1+1.8
51.5+117
59.8+11.
22

---

Published as a conference paper at ICLR 2023
H COMPARISON WITH SSKDE
SSKDE (Wang et al., 2009) also compute the posterior probability of labels based on the KDE over the feature space. However, SSKDE has several critical limitations:
SSKDE requires an ad-hoc feature extractor. SSKDE applies KDE to the precomputed features, unlike RoPAWS jointly learning the representation and the KDE generative classifier. The ability to learn good representations is one of the main differences between RoPAWS and pre-deep- learning approaches like SSKDE.
SSKDE is infeasible for large-scale data. Since SSKDE is nonparametric method, it computes the kernel matrix between all training data. Thus, the training cost of SSKDE is O(N3), where N is the number of training data. In contrast, RoPAWS only computes the kernel matrix within the mini-batches, reducing the training cost to O(N *M where M is the size of the mini-batch and M < N. This enables the training under arbitrarily large data.
Due to the limitations listed above, SSKDE cannot be applied to real-world large-scale datasets with- out significant modification. In addition, RoPAWS has multiple technical novelties over SSKDE:
The target problem is different. The main goal of RoPAWS is robust semi-supervised learning under uncurated data. Thus, we propose in-domain prior to handling out-of-distribution (OOD) samples. In contrast, SSKDE aims for the curated setup and does not address the overconfi- dence issue of OOD samples. In addition, SSKDE mainly targets the video annotation problem and only conducts small-scale experiments, while RoPAWS verifies its effectiveness on realistic large-scale image benchmarks.
The definition of kernel is different. RoPAWS discovers that PAWS is implicitly modeling KDE with cosine similarity kernels, where the kernel definition comes from the design of recent self- supervised learning methods. In contrast, SSKDE uses Gaussian or Exponential kernels, which may not fit with deep learning techniques.
The formulation and required assumption are different. RoPAWS leverages the class-balanced sampling of PAWS, canceling out the denominator of p(rl)) in the Bayes rule. SSKDE considers a different assumption that p(y|æ) is a linear combination of p(ylar) and ground-truw labels (if exists), where p(y) (r) is again a function of p(y) (z) and kernels. As result, the final prediction formula of SSKDE in Eq. (14) of their paper and RoPAWS in Eq. (11) are different.
Despite the fundamental limitations of SSKDE, we provide an experiment comparing RoPAWS and SSKDE. Since SSKDE requires an ad-hoc feature extractor, we run SSKDE upon PAWS and RoPAWS backbones, in addition to a randomly initialized linear layer of dimension 3x32x32- 128. We follow the default configuration of SSKDE: use the Gaussian kernel with gamma- 1/(# of features), weight for the linear interpolation & = 0.9, and number of iterations M = 40.
Table 9 shows the test accuracy models on CIFAR-10 10 using 25 labels per class. First, one can see that the quality of the feature extractor mostly decides the final accuracy. We remark that RoPAWS learns good representation, which is a clear advantage over SSKDE. Second, the Soft-NN classifier (default of PAWS/ROPAWS) performs better than the SSKDE classifier. Indeed, the representation of PAWS/ROPAWS is optimized for the Soft-NN classifier. The results confirm that RoPAWS is empirically better than SSKDE. However, exploring different designs of generative classifiers (c.g., extending SSKDE to learn representations jointly) would be an interesting future research direction.
Table 9: Test accuracy of RoPAWS and SSKDE under different feature extractors. The quality of representation mostly decides the final accuracy. Upon the PAWS/RoPAWS representation, the Soft-NN classifier performs better than SSKDE, as PAWS/RoPAWS are optimized for this. Note that SSKDE only provides the inference method and does not learn the corresponding feature extractor.
Feature extractor
Random linear
PAWS
RoPAWS
Inference
SSKDE
SSKDE
Soft-NN
SSKDE
Soft-NN
Test acc.
13.9
91.0
92.4
92.8
94.7
23

---

Published as a conference paper at ICLR 2023
1 IMPROVEMENTS ON CURATED DATA
RoPAWS improves PAWS for curated data since refines the pseudo-labels via label propagation. This benefit is more significant when the original pseudo-label of PAWS is inaccurate, e.g., only few labels are available. As result, RoPAW'S gives a large gain (+2.3%) for CIFAR-10 using 25 labels. However, the gap saturates when using more labels, e.g., 250 labels for CIFAR-10 or 10K (1%) or 100K (10%) labels for ImageNet. Table 10 shows the test accuracy of models trained on CIFAR-10 using 4 labels. We report the average and standard deviation over five runs. RoPAWS highly improves (+3.0%) PAWS in this more challenging scenario.
Table 10: Test accuracy (%) of semi-supervised learning methods trained on curated CIFAR-10.
FixMatch
PAWS
RoPAWS (ours)
4 labels
86.2+3.4
85.7=2.4
88.7=22. (42.3)
We provide additional analyses verifying the benefit of RoPAWS. Specifically, RoPAWS reduces confirmation bias (Arazo et al., 2020) and improves calibration (Loh al., 2022) of pseudo-labels. Recall that lowering wrong predictions (i.e., less confirmation bias) and understanding right confi- dence (i.c., better calibration) play a crucial role in label propagation.
First, we verify that label propagation refines the pseudo-labels and reduces confirmation bias (i.e., improves correctness). To this end, we present the accuracy of pseudo-labels over label propagation, following the same setup of Appendix K. Table 11 shows the test accuracy of RoPAWS trained on curated CIFAR- 10 using 25 labels, using different epochs of checkpoints (total: 600) and different soft neighstt (Soft-NN) classifiers. Iter. O means the Soft-NN classifier only uses labeled data, and lter. & 2 means the iterations of label propagation (using both labeled and unlabeled data) where k = x is the fixed-point solution. The pseudo-label becomes more accurate iteration goes on. Also, note that the effect of label refinement is more impactful in early epochs, where the pseudo-labels are inaccurate. This correction makes RoPAWS converge to better optima.
Test accuracy (%) of a RoPAWS model trained on curated CIFAR-10 10 using 25 labels. We compare the different inference strategies: the original soft-NN and the label propagation version varying the number of iterations. Label propagation improves the accuracy. Also, note that the effect of label refinement more impactful in early epochs when the pseudo-labels are inaccurate.
Table 11:
Epoch
Iter. 0
Iter. 1
Iter. 2
Iter. 3
Iter. 0o
10
73.8
74.6
74.6
75.0
75.0
100
87.9
88.7
88.7
88.7
88.7
600
95.0
95.3
95.3
95.3
95.3
Second, we compare the calibration of PAWS and RoPAWS classifiers. To this end, we measure expected calibration error (ECE) (Loh et al., 2022). We apply the TorchMetric implementation' 15 and use the default 15 bins. Table 12 shows the ECE (lower is better) of models trained on CIFAR- 10 using 25 labels. RoPAWS gives better calibration than PAWS.
Table 12: ECE (%, lower is better) of models trained on curated CIFAR- 10 using 25 labels.
PAWS
RoPAWS (ours)
1.52
0.92
rer.cheerriis.reeaeedd...... 1tee..o//e// 1o/en/v0...voo/ 0.2/clasa fi fication/ calibration ier_attrrnerrereeee
htm1 html
24

---

Published as a conference paper at ICLR 2023
J ADDITIONAL ANALYSES FOR ROPAWS
Table 13 shows the prediction confidence and AUROC for PAWS and RoPAWS, trained on CIFAR- 10 (+ CIFAR-100) using 25 400 labels. AUROC a popular metric for OOD detection, measuring the area under the receiver operating characteristic (ROC) curve (Bradley, 1997). Precisely, the ROC curve gives the trade-off between true/false positive/negatiie varying the threshold for OOD detection, and AUROC computes the overall OOD discriminability over thresholds.
We use the soft neighrt classifier for prediction p(y(æ) and maximum representation simi- larity to labeled data s(x) == E CIFAR-10 10 (in) and CIFAR-100 (out) with a similar confidence level. In contrast, RoPAWS predicts CIFAR-10 with high confidence while assigning low confidence for OOD data.
for OOD detection (low s(x) OOD). PAWS predicts
Table 13: Prediction confidence (%) and AUROC (%) of PAWS and RoPAWS trained on CIFAR-10 (+ CIFAR-100), RoPAWS predicts CIFAR-IO (in) with high confidence while calibrating prediction for CIFAR-100 (out). In contrast, PAWS assigns mid-level confidence for both data.
25 labels
400 labels
Conf. (in)
Conf, (out)
AUROC
Conf. (in)
Conf, (out)
AUROC
PAWS
87.1
85.2
70.8
94.0
90.9
81.2
RoPAWS (ours)
92.4
65.6
81.5
95.0
67.1
87.0
K EFFECT OF LABEL PROPAGATION
We compare the original Soft-NN classifier using only labeled data (Eq. (1)) and the semi-supervised (label propagation) version using both labeled and unlabeled data (Eq. (52); becomes Eq. (12) at Iter. oc). Table 14 shows the effect of label propagation during inference, using the RoPAWS model trained on CIFAR-10 (+ CIFAR- 100), 400 labels. We sample a batch of labeled and unlabeled data and evaluate the accuracy of unlabeled (in) data, along with the prediction confidence of unlabeled (in) and unlabeled (out) data. The semi-supervised soft-NN calibrates the prediction and improves the accuracy as epochs go, outperforming the original soft-NN. converges after 3 iterations, so one may apply the iterative formula instead of the matrix inversion for RoPAWSS prediction.
Test accuracy (%) and prediction confidence (%) CIFAR- 10 (in) and CIFAR-100 (out), using a RoPAWS model trained on CIFAR-10 10 (+ 100),, 400 labels. We compare the different inference strategies: the original soft-NN, and the label propagation version varying the number of iterations. Label propagation calibrates the prediction and improves the accuracy.
Table 14:
Iter.
Test
Conf. (in)
Conf. (out)
Soft-NN
93.8
93.9
67.3
1
93.8
76,8
42.6
2
94.1
84.2
46.7
Soft-NN (semi-sup)
94.5
85.0
47.2
Oc
94.5
47.2
25

---

Published as a conference paper at ICLR 2023
L OPENMATCH UNDER SMALL DISTRIBUTION SHIFTS
OOD filtering approaches are less effective when in- vs, out-of-domain samples are not discrimi- native. Since they cannot identify which data is OOD, they conservatively select training samples hence the model is optimal. Consequently, OpenMatch performs well for CIFAR-10 (+ CIFAR-100) using 400 labels but underperforms when using 25 labels. Table 15 shows that OpenMatch fails to detect OOD samples for 25 labels setup, leading to meager selected ratio. Also, the OOD filtering approaches underperform on curated datasets since it filters uncertain in-class samples.
Selected ratio (%) of confident samples and AUROC (%) of CIFAR-10 vs. CIFAR-100 for the OpenMatch models trained on CIFAR-10 and CIFAR-10 (+ CIFAR-100) The model cannot distinguish unseen in-class (not covered by a few labels) and near out-of-elass samples, as shown by AUROC. As result, the model filters too many samples and is under-trained.
Table 15:
CIFAR-10
CIFAR-10 (+ CIFAR-100)
25 labels
400 labels
25 labels
400 labels
Selected ratio
27.5
98.9
0.2
37,4
AUROC
54.2
91.5
M FUTURE RESEARCH DIRECTIONS
We believe the probabilistic formulation of RoPAWS, i.e., KDE modeling view over representation space, has the potential to be extended to various scenarios in a principled manner.
For example, we assume the prior class distribution p(y) is uniform over all classes. One can extend RoPAWS to class- imballnedd setup (Kim et al., 2020; Wei et al., 2021) by assuming non-uniform prior. RoPAWS can also handle label shift, i.c.. class distribution of labeled and unlabeled data are different. Indeed, the prediction of RoPAWS only assumes the prior on unlabeled data, which can be controlled independently from the prior on labeled data. On the other hand, the entropy mean entropy maximization (me-max) regularization (p)) in the objective is identical to minimizing the KL divergence between the mean prediction and the uniform prior. This can extended to minimizing the (posterior) mean prediction and the prior when using non-uniform prior.
Also, one can extend RoPAWS to open-world setup (Cai et al., 2022: Vaze et al., 2022), which clusters novel classes in addition to predicting known classes. Here, densities p(xly) for novel classes y E [C++,.... C' (oooee Then, finding the maximum likelihood assignment of cluster y for unlabeled data would cluster both labeled and unlabeled data similarly to the infinite Gaussian mixture model (Rasmussen, 1999) but under KDE cluster modeling.
model conditional
Finally, one may use the estimated probability, e.g., p(in(z:) or p(= ((]yy), to choose the most uncertain sample to label for active learning (Settles, 2009). Combining semi-supervised learning and active learning under large uncurated data would be important for practical scenarios.
We remark that the principle of RoPAWS can be generalized beyond PAWS and image classification. For example, the standard softmax classifier can be interpreted as a generative classifier modeling densities with the Gaussian mixture model (GMM) (Lee et al., 201 8b). Thus, one can adapt the soft- max classifier of previous algorithms, such as FixMatch (Sohn et al., 2020), as GMM classifier to estimate the uncertainty of samples. However, GMM needs sufficient labels to estimate the variance of each class; hence, less fits for semi-supervised learning.
One may extend RoPAWS for cosscaassification tasks such as object detection (Liu et al., 2022) by applying dense self-supervision (Wang et al., 2021) or object-aware augmentation (Mo al., 2021). Also, one may tackle non-image domains, using different augmentations (Lee et al., 2021).
26


Reviews:
[{"review_id": "zAPGPHFZClP", "reviewer": "Reviewer_Q9GX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper at hand prososes an extension of a recent proposed semi-supervised representation learning approach PAWS. The extension focuses in robustness, hence, RoPAWS, concerning uncurated data from, e.g., out-of-distribution data. This is a relevant problem in many real world applications.", "summary_of_the_review": "The paper addresses a relevant problem and proposes an extension to \"fix\" limitations of the recently proposed PAWS method. The extensions seems incremental but I'm still in favor for the paper and think its worth publishing.", "strength_and_weaknesses": "+ Quite simple extensions to pave the way for real world problems.\n+ Experiments a thoughtful and support the main aim of the approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wyk_N_GDen", "reviewer": "Reviewer_oxEy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper argues that the PAWS is not robust to unlabeled real-world data. This paper tries to tackle this problem by suggesting the RoPAWS. RoPAWS uses in-domain prior to prevent a model from overconfidently predicting unseen class data by adopting regularization utilizing similarity between labeled and unlabeled data. Thereby RoPAWS shows better performance in various settings. RoPAWS especially excels in uncurated settings with fewer labels.\nAlso, this paper gives a novel probabilistic view of PAWS. \n\n", "summary_of_the_review": "The motivation of RoPAWS is well supported. I believe that sufficient advances in the performance on uncurated settings shows the effectiveness of RoPAWS. Also the analysis using t-SNE demonstrates that RoPAWS works as the authors intended.\n", "strength_and_weaknesses": "Strengths\n\n- Probabilistic explanation of PAWS that it works as generative classifier and extension to RoPAWS make sense and interesting.\n- The problem definition and formulation were clear.\n- Straightforward solution for the given limitation (overconfident prediction).\n- It obtains SOTA results with moderate additional time cost.\n- Fig. 4 shows apparently how RoPAWS works effectively compared to PAWS\n\nWeaknesses \n\n- The performances of the proposed RoPAWS as well as PAWS will be highly dependent on the choice of data augmentation methods for creating different views. The details are missing.\n- This is rather a question: I wonder if the self-supervisory signals between the two views of the labeled data gives better performance.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pDYTuqhAkHq", "reviewer": "Reviewer_R1Ae", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the semi-supervised image classifier PAWS (Assran et al., 2021) by using unlabeled data in addition to the labeled data to inform the representation. Using the unlabeled data in this way allows the model to learn from out-of-distribution classes rather than assuming unlabeled data is curated to contain the same classes seen in labeled data. Experimental results show that the proposed RoPAWS method performs roughly on-par with baseline PAWS method on curated datasets and improves on PAWS in uncurated (OOD) settings.", "summary_of_the_review": "This is a good paper overall. The method provides an improvement over an existing method that makes is more applicable in real-world settings where OOD data is common. The text is well written and experiments are clearly laid out. Empirical results are mixed, in some settings performing on-par with existing methods and in some improving on them.", "strength_and_weaknesses": "Strengths: \n*  The work reframes the PAWS method as a generative classifier and uses this probabilistic interpretation to clearly introduce the calibration using the unlabeled data in this setting.\n* The method prevents over-confident class predictions for OOD data in the unlabeled set, unlike PAWS, which yields better representations for classification.\n* Ablation study is provided to provide a careful study of the components of the method\n* Sensitivity of hyper parameters are empirically investigated and show improvements over baseline PAWS for a range of hyperparameter settings.\n\n\nWeaknesses:\n* In the original PAWS, class-balanced sampling plays an important role to avoid collapse of representations. By introducing the unlabeled data, it is not clear if this trivial collapse is now a problem in RoPAWS. Can this point be clarified in the exposition of the method? \n\n* When applying this method to curated data, it is not clear any improvements in performance should be expected. The model will be learning from more data, but the labels inferred for unlabeled data may not be accurate and therefore hurt performance. Indeed the experiments show only minor, perhaps negligible, improvement over PAWS.  The text does claim superiority in these cases as well but the numbers do not generally support the claim. The performance seems generally equal. An interesting point to investigate is the exception to this pattern where RoPAWS conclusively improves on PAWS in the curated CIFAR-10 dataset with only 25 labels per class. Here RoPAWS does benefit from using the (in-distribution) unlabeled data. This could be a strength of the proposed method if investigated further.\n\n* Likewise, the results on ImageNet (Table 2) show similarly small improvements in performance of RoPAWS over PAWS at both 1% and 10% of labels.  Some discussion of this could improve the empirical analysis.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TNTUpcz_ue", "reviewer": "Reviewer_NjZV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to make semi-supervised learning robust when unlabeled data contains out-of-class data. This is a practical problem. The authors improve the semi-supervised learning algorithm PAWS by calibrating the prediction of PAWS based on the data densities. Experimental results show the proposal achieves better performance compared with the PAWS algorithm.", "summary_of_the_review": "Overall, the contribution and novelty of this paper is limited, the experimental results are also not convincing.", "strength_and_weaknesses": "Strength:\n1) This paper focuses on the robust semi-supervised learning problem with out-of-class unlabeled data. This is an important problem.\n2) The proposal achieves better performance compared with PAWS.\n\nWeakness:\n1) This paper only gives a minor improvement on the PAWS algorithm. The contribution and novelty are quite limited. Can the proposal be applied to the general semi-supervised learning algorithms?\n2) There are also some other robust SSL algorithms that consider the out-of-class unlabeled data. More related methods should be compared in the experiments.\n\n\n---------Update-------\nI notice that the authors compared with some robust SSL algorithms on CIFAR-10 datasets. I recommend the authors conduct experiments on more settings, such as different datasets, various labels, various extents of OOD unlabeled data, etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0L-UmKsPDp", "reviewer": "Reviewer_yYAe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with the problem of semi-supervised representation learning and proposes a method based on the specific previous method called PAWS. More specifically, the previous method PAWS can be regarded as kernel density estimation (KDE) for modeling the distribution of latent variables, and the proposed method called RoPAWS extends it in a semi-supervised way. The proposed method also introduces data-dependent in-domain priors assuming that an example is likely to have the same label if it is close to some labeled example.", "summary_of_the_review": "I have to recommend this paper be rejected since it requires major revisions to justify the novelty of the proposed method. It should conduct detailed bibliographical surveys and experimental comparisons with previous related methods.", "strength_and_weaknesses": "[Strength]\n\nS1. Representation learning is one of the hot topics at this conference. Technically solid methods for this topic will draw attention from a broad range of researchers and engineers.\n\nS2. The current manuscript is basically well-written and easy to follow.\n\n[Weakness]\n\nW1. I am afraid that the main idea of the proposed method is similar to semi-supervised kernel density estimation (SSKDE) presented in the following paper.\nWang, Hua, Mei, Hong, Qi, Song, Dai, \"Semi-supervised kernel density estimation for video annotation,\" Computer Vision and Image Understanding, Volume 113, Issue 3, 2009. https://doi.org/10.1016/j.cviu.2008.08.003.\nI understand that the specific implementation of the proposed method is different from that of the above paper; however, the advantages of the proposed method against SSKDE should be clearly presented both literally and experimentally.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster