RankingMatch: Delving into Semi-Supervised Learning with Consistency Regularization and Ranking Loss
Authors: Trung Quang Tran
Mingu Kang
Daeyoung Kim
Venue:  (2021)
Keywords: BatchMean Triplet Loss
Semi-Supervised Learning
Consistency Regularization
Metric Learning

Abstract: Semi-supervised learning (SSL) has played an important role in leveraging unlabeled data when labeled data is limited. One of the most successful SSL approaches is based on consistency regularization, which encourages the model to produce unchanged with perturbed input. However, there has been less attention spent on inputs that have the same label. Motivated by the observation that the inputs having the same label should have the similar model outputs, we propose a novel method, RankingMatch, that considers not only the perturbed inputs but also the similarity among the inputs having the same label. We especially introduce a new objective function, dubbed BatchMean Triplet loss, which has the advantage of computational efficiency while taking into account all input samples. Our RankingMatch achieves state-of-the-art performance across many standard SSL benchmarks with a variety of labeled data amounts, including 95.13% accuracy on CIFAR-10 with 250 labels, 77.65% accuracy on CIFAR-100 with 10000 labels, 97.76% accuracy on SVHN with 250 labels, and 97.77% accuracy on SVHN with 1000 labels. We also perform an ablation study to prove the efficacy of the proposed BatchMean Triplet loss against existing versions of Triplet loss.

Under review as a conference paper at ICLR 2021
RANKINGMATCH: DELVING INTO SEMI-SUPERVISED LEARNING WITH CONSISTENCY REGULARIZATION AND RANKING LOSS
Anonymous authors Paper under double-biee blind review
ABSTRACT
Semi-supervised learning (SSL) has played an important role leveraging un- labeled data when labeled data is limited. One of the most successful SSL ap- proaches based on consistency regularization, which encourages the model produce unchanged with perturbed imput. However, there has been less attention spent on inputs that have the same label. Motivated by the observation that the inputs having the same label should have the similar model outputs, we propose novel method, RankingMatch, that considers not only the perturbed inputs but also the similarity among the inputs having the same label. We especially introduce a new objective function, dubbed BatchMean Triplet loss, which has the advan- tage of computational efficiency while taking into account all input samples. Our RankingMatch achieves state-of-the-an performance across many standard SSL benchmarks with a variety of labeled data amounts, including 95.13% accuracy on CIFAR-10 with 250 labels, 77.65% accuracy on CIFAR- 100 with 10000 la- bels, 97.76% aceuracy on SVHN with 250 labels, and 97.77% accuracy on SVHN with 1000 labels. We also perform an ablation study to prove the efficacy of the proposed BatchMean Triplet loss against existing versions of Triplet loss.
INTRODUCTION
Supervised learning and deep neural networks have proved their efficacy when achieving outstand- ing successes wide range of machine learning domains such as image recognition, language modeling, speech recognition, or machine translation. There is an empirical observation that better performance could be obtained the model trained on larger datasets with more labeled data (Hes-) iness et al. 2017 Mahajan al. 2018 Kolesnikov et al. 20119] Xie et al. 2020, Raffel al. 2019). However, data labeling is costly and human-laboo labor-demandding even requiring the participation of experts (for example, in medical applications, data labeling must be done by doctors).
In many real-world problems, it is often very difficult to create a large amount of labeled training data. Therefore, numerous studies have focused on how leverage unlabeled data, leading variety of research fields like self-supervised learning (Doersch et al.) 2015 Noroozi & Favaro
2016 Gidaris et al. (0118), semi-supervised learning (Berthelot al. 2019b Nair al.) 2019
Berthelot et al. 2019a Sohn et al. 2020), or metric learning (Hermans et al. 2017: Zhang et al.
2019). In self-supervised learning, pretext tasks are designed that the model can learn meaningtul information from large number of unlabeled images. The model is then fine-tuned on a smaller set of labeled data. In another way, sepi-super learning (SSL) aims leverage both labeled and unlabeled data in single training process. On the other hand, metric learning does not directly predict semantic labels of given inputs but aims to measure the similarity among inputs.
In this paper, we unify the idea of semi-supervised learning (SSL) and metric learning to propose RathingMatch, a more powerful SSL method for image classification (Figure []. We adopt Fix- Match SSL method (Sohn al. 2020). which utilized pseudo-labeling and consistency regular- ization to produce artificial labels for unlabeled data. Specifically, given an unlabeled image, its reakly--ugmment and wroongly-augmented version created. The model prediction correspond- ing to the weakly-augmennt image is used as the target label for the auggenedd image, encouraging the model to produce the same prediction for different perturbations of the same input.

---

Under review as a conference paper at ICLR 2021
Prediction
Weak Augmentation
Cross-Entropy
dog
Model
Softmax
Loss
Labeled image
Ranking
L2-Norm
Loss
Unlabeled
Prediction
Pseudo-label
image
Weak Augmentation
Model
Softmax
Prediction
Strong Augmentation
Cross-Entropy
Model
Softmax
Loss
Ranking
L2-Norm
Loss
Figure Diagram of RankingMatch. In addition Cross-Enns Entso-y loss, Ranking loss is used to encourage the model to produce the similar outputs for the images from the same class.
Consistency regularization approach incites the model to produce unchanged with the different per- turbations of the same input, but this is not enough. Inspired by the observation that the images from the same class (having the same label) should also have the similar model outputs, loss functions of metric learning, called Ranking losses, to apply more constraints to the objective function of our model. Concretely, we use Triplet and Contrastive loss with the aim of encouraging the model to produce the similar outputs for the images from the same class. Given an image from class (saying dog, for example), Triplet loss tries pull positive samples (images from class dog) nearer the given image and push negative samples (images not from class dog) further the given image. the other hand, Contrastive loss maximizes the similarity of the images from the same class and minimizes the similarity of the images from different classes. However, instead of apply- ing Triplet and Contrastive loss to the image representation as previous works did (Hermans et al, 2017 Chen et al. 2020a), directly apply them to the model output (the "logits" score) which the output of the classification head. We argue that the images from the same class do not have have similar representations strictly, but their model outputs should be similar possible. Our motivation and argument could be consolidated in Appendix [A] Especially, we propose new version of Triplet loss which is called BatchMean. Our BatchMean Triplet loss has the advantage of computational efficiency of existing BatchHard Triplet loss while taking into account all input sam- ples when computing the loss. More details will be presented in Section 3.3. Our key contributions are summarized as follows:
utilize
We introduce a novel SSL method, RankingMatch, that encourages the model to produce the similar outputs for not only the different perturbations of the same input but also the input samples from the same class.
Our proposed BatchMean Triplet loss surpasses two existing versions of Triplet loss which are BatchAll and BatchHard Triplet loss (Section 4.5).
Our method simple yet effective, achieving state-of-the-art results across many standard SSL benchmarks with various labeled data amounts.
RELATED WORK
Many recent works have achieved success in semi-supervised learning (SSL) by adding loss term for unlabeled data. This section reviews two classes of this loss term (consistency regularization and entropy minimization) that related to work. Ranking loss is also reviewed in this section.
Consistency Regularization This is a widely used SSL technique which encourages the model to produce unchanged with different perturbations of the same input sample. Consistency regulariza- tion was early introduced by Saajadi et al. (2016) and Laine & Aila (2016) with the methods named "Regularization With Stochastic Transformations and Perturbations" and "TI-Model", respectively.
2

---

Under review as a conference paper at ICLR 2021
Both of these two approaches used Mean Squared Error (MSE) to enforce the model to produce the same output for different perturbed versions of the same input. Later state-of-thh-aar art methods adopted consistency regularization in diverse ways. In MixMatch (Berthelot et al., 2019b), a guessed label, computed based on K weakly-augmented versions of an unlabeled sample, was used as the target label for all these K' weakly-augmented samples. On the other hand, in FixMatch (Sohn) iet al. 2020), a pseudo-label, which is computed based on the weakly-augmentee unlabeled sample, became the target label for the aaggmetee version of the same unlabeled sample.
Entropy Minimization One of the requirements in SSL is that the model prediction for unlabeled data should have low entropy. Grandvalet Bengio (2005) and Miyato et al.) (2018) introduced an additional loss term, which is explicitly incorporated in the objective function, to minimize the entropy of the distribution of the model prediction for unlabeled data. On the other hand, MixMatch
(Berthelot et al. 2019h) used a sharpening function to adjust the model prediction distribution and thereby reduced the entropy of the predicted label. FixMatch (Sohn et al. 2020) implicitly ob- tained entropy minimization by constructing hard labels from high-confidence predictions (predic-
tions which are higher than a pre-defined threshold) on weakly-augment hard labels were then used as the target labels for wrrongl-auugment
unlabeled data. These
unlabeled data.
Metric Learning and Ranking Loss Metric learning is an approach that does not directly predict semantic labels of given images but trains the model to learn the similarity among samples (Kulis) jet al. 2012 Kaya & Bilge, 2019). There are various objective functions used in metric learning, including Triplet and Contrastive loss which used in work. Triplet loss successfully ploited in person re-identification problem (Hermans et al. 2017). A triplet contains a person image referred to as anchor, a positive sample which the image from the same person with the anchor, and a negative sample being the image from the different person with the anchor. Triplet loss was used enforce the distance between the anchor and negative sample to be larger than the distance between the anchor and positive sample by at least a margin m. Besides. SimCLR (Chen et al.) 2020a) utilized Contrastive loss to maximize the similarity between two different augmented ver- sions of the same sample while minimizing the similarity between different samples. Both Hermans jet al. (2017) and Chen et al. (220a) applied Triplet and Contrastive loss to the image representa- tion. Contrastive loss was also used by Chen et al. (2020b) for semi-supervised image retrieval and person re-identification. Given feature (or image) representations, Chen et al. (2020b) computed class-wise similarity scores using similarity measurement to learn crmmantics-oriented similarity representation. Contrastive loss was then applied to both image and semantics-oriente
similarity representation in two learning phases. If the model output in image classification is viewed form of class-wise similarity scores, the high-level idea of our method might be similar to Chen al. (2020b) in utilizing Contrastive loss. However, in our case, the model itself obtains class-wise similarity scores, and Contrastive loss is only applied to the model output ("logits" score, but not image representation) single training process, More details will be presented in Section 3.3
3 RANKINGMATCH
This section starts to describe the overall framework and objective function RankingMatch, Next, two important factors of the objective function, Crros--Enrrop and Ranking loss, will be presented in detail. Concretely, Triplet and Contrastive loss will be separately shown with our proposed and modified versions.
3.1 OVERALL FRAMEWORK
The overall framework of RankingMatch illustrated in Figure Both labeled and unlabeled data are simultaneously leveraged in a single training process. Two kinds of augmentation are used to per- turb the input sample. While weak augmentation uses standard flipping augmentation strategies, more complex transformations are used for strong augmentation. We utilize Rand/Aumment (Cubuk et al. 2020) for strong augmentation, consisting of multiple trans- formation methods such as contrast adjustment, shear, rotation, translation, etc. Given a collection of transformations, two of them are randomly selected to strongly perturb the input sample. Cutout (De & Taylor 2017) is followed to obtain the final auumene- sample.
and horizontal
3

---

Under review as a conference paper at ICLR 2021
As shown in Figure only weak augmentation is used for labeled data. The veakly-augmentt labeled image is fed into the model to produce scores for labels. These scores are actually the output of the classification head, and call them "logits" score for convenient explanation. A softmax function is used to convert the "logits" scores to the probabilities for labels. These probabilities are then used along with ground-truth labels to compute Cross-Entropy loss. An Izz-normalization is applied to the "logits" scores before using them for computing Ranking loss. We experimented and found that uz-normalization is important factor contributing the success of Ranking.Match, which will be shown in Section 4.5 The ground-truti -ruth labels are used to determine positive samples (images from the same class) and negative samples (images from different classes) in computing Ranking loss. The same procedure is used for unlabeled data except that pseudo-labels, obtained from weakly-augmented unlabeled samples, are used instead of the ground-truth labels.
Let X' ((rb, b (1...., B)} define a batch of B labeled samples, where Ib is the training sample and I, the corresponding one-hot label. Let 2 {u, b E (1....,BB)) be a batch of μB unlabeled samples with coefficient μ determining the relative size of X and U. We denote weak and strong augmentation as (.) and A.(.) respectively. Let Pinoeel (y) z; 0) be the "log- its" score produced by the model for given input x. As result, ceftmaxx(Puodee (y 2; f)) and 122orm(Pmodee1( æ;; 0)) are the softmax function and Lg-normalization applied to the "logits" score, respectively. Finally, let H(v, q) be Cross-Entropy loss of the predicted class distribution q and the target label v. Notably, v corresponds to the ground-truth label or pseudo-label in the case of labeled or unlabeled data respectively.
As illustrated in Figure I there are four elements contributing to the overall loss function of Rank- ingMatch. Two of them are Cross-Entropy loss for labeled and unlabeled data, denoted by L and - respectively. Two remaining ones are Ranking loss for labeled and unlabeled data, corre- sponding to L Rank and L Rank respectively. The objective minimizing the loss function defined as follows:
-- A. + (r.Rookk + + LRank)
where X, and A, are scalar hyperparameters denoting the weights of the loss elements. In Section 3.2 and 3.3 we will present how Cross-Entropy and Ranking loss are computed for labeled and unlabeled data in detail. We also show comparisons between RankingMatch and other methods in Appendix B The full algorithm of RankingMatch is provided in Appendix
3.2 CROSS-ENTROPY LOSS
For labeled data, since the ground-truth labels are available, the standard Cross-Entrops loss is computed as follows:
B
C
H(.. Softmax(pmodee(9 Amo
For unlabeled data, we adopt the idea of FixMatch (Sohn et al., 2020) to obtain the pseudo-label which plays the similar role the ground-truth label of labeled data. Given unlabeled im- age ug, the model first produces the "logits" score for the weakly-augmented unlabeled image: qo Pmodda | Ann (us); 0). A softmax function is then applied to qi to obtain the model predic- tion: Th = Softmax (q.). The pseudo-label corresponds to the class having the highest probabil- ity: qu argmax(@;, ).). Note that for simplicity, argmax is assumed pseudo-label. A threshold 7 is used to ignore predictions that have low confidence. Finally, the high-confidence pseudo-labels are used as the target labels for strongly-augmente versions of cor- responding unlabeled images, leading to:
produce the valid one-hot
BB
CCE
(max (1)) 2 7) H( Softmax(pmodaal A.(u)):)))
(3)
PB
1=1
Equation] 3 satisfies consistency regularization and entropy minimization. The model is encouraged
to produce consistent outputs for mrongly-augmenter samples against the model outputs for weakly- augmented samples; this is referred to as consistency regularization. As advocated in Lee (2013) and (Sohn et al. (2020), the use of a pseudo-labed label, which is based on the model prediction for an unlabeled sample, hard target for the same sample could be referred as entropy minimization.
4

---

Under review as a conference paper at ICLR 2021
3.3
RANKING LOsS
This section presents two types of Ranking loss used in our RankingMatch,, which are Triplet and Contrastive loss. We directly apply these two loss functions to the "logits" scores, which different from previous works such as Hermans et al. (2017) and Chen et al. (2020a). Especially, our novel version of Triplet loss, which is BatchMean Triplet loss, will also be presented this section.
Let be a batch of Lz-normalized "logits" scores of the network shown in Figure [ Let denote the label the Ly-normallzed "logits" score i. This label could be the ground-truth label or pseudo- label in the case of labeled or unlabeled data, respectively. The procedure of obtaining the pseudo- label for unlabeled data was presented Section 3.2 Notably, Ranking loss is separately computed for labeled and unlabeled data, LRank and L.aan in Equation | could be either Triplet loss (Section 3.3.1 or Contrastive loss (Section (.3.2). Let a, p. and n be the anchor, positive, and negative sample, respectively. While the anchor and positive sample represent the Lo-nomaalized "logits" scores having the same label, the anchor and negative sample are for the Le-normalized "logits" scores having the different labels.
3.3.1
BATCHMEAN TRIPLET LOSS
Let di denote the distance between two "logits" scores i and j. Following Schroff' et al. (2015) and Hermans al. (2017), two existing versions of Triplet loss, BatchAll and BatchHard, could be defined as follows with the use of Euclidean distance for di..
BatchAll Triplet loss:
CBA =
1 V
Σ
f(m + da - da,n)
(4)
a.P.nEC
No
where V is the number of triplets. A triplet consists of an anchor, positive sample, and a negative sample. BatchHard Triplet loss:
LB
f(m max da.p min da,n)
(5)
|C|
pec 39
neC
aEC
9a?
In Equation and 5 is the margin, and f(o) indicates the function to avoid revising "already correct" triplets. A hinge function (f( = max(0, (0, +)) could be used in this circumstance. For instance, if triplet already satisfied the distance between the anchor and negative sample is larger than the distance between the anchor and positive sample by at least a margin m. that triplet should be ignored from the training process by assigning it zero-value (f (mm-dd.p-dd
0f m+ da,p da,n < 0, corresponding to da.n da.p 2 m). However, as mentioned in Hermans et al. (2017), the softplus function (In (1 exp ))) gives better results compared to the hinge function. Thus, we decided to use the softplus function for all our experiments, which is referred to as soft-margin.
While BatchAll considers all triplets, BatchHard only takes into account hardest triplets. A hardest triplet consists of an anchor, a furthest positive sample, and nearest negative sample relative to that anchor. The intuition behind BatchHard that if we pull an anchor and its furthest positive sample together, other positive samples of that anchor will also be pulled obviously. BatchHard is more computationally efficient compared to BatchAll. However, because max and min function are used in BatchHard, only the hardest triplets (anchors, furthest positive samples, and nearest negative samples) are taken into account when the network does backpropagation. We argue that would be beneficial if all samples are considered and contribute updating the network parameters. Therefore, we introduce a novel variant of Triplet loss, called BatchMean Triplet loss, as follows:
LBM
do.p
da
C
C
C
aEC
NEC
Vp-Wa
By using "mean" function (/) 2 instead of max and min function, our proposed BatchMean
Triplet loss not only has the advantage of computational efficiency of BatchHard but also takes into account all samples. The efficacy of BatchMean Triplet loss will be clarified in Section 4.5
5

---

Under review as a conference paper at ICLR 2021
3.3.2 CONTRASTIVE LOSS
Let s denote the similarity between two Lz-normalized "logits" scores and j. Referring to Chen et al. (2020a), we define Contrastive loss applied to our work as follows:
LCT
1 N
Σ
In exp (sima.p/ (T) + Σ
exp (sima.p/ (T)
exp (sima,n/T))
P.PCC
nEC 4ya
where N is the number of valid pairs of anchor and positive sample, and T is constant denoting the temperature parameter. Note that if the jth and jth "logits" score of C have the same label, there will be two valid pairs of anchor and positive sample. The 7th "logits" score could become an anchor, and the jth "logits" " score a positive sample; and vice versa. The form of LCT is referred to the normalized corresponds to maximizing s and minimizing sima,n- Moreover, we also want the anchor and positive sample to be similar possible. As result, cosine similarity is suitable choice for the similarity function of LCT. For instance, if two "logits" score vectors are the same, the cosine similarity between them has the maximum value which
sepeerattue-scaled cross-entropy loss. The objective is minimizing Lor; this
EXPERIMENTS
We evaluate the efficacy of RankingMatch on standard semi-supervised learning (SSL) benchmarks such as CIFAR-10 (Krizhevsky et al. 2009), CIFAR-100 (Krizhevsky et al. 2009). SVHN (Netzer al. 2011), and STL-10 (Coates et al. 2011 We also conduct experiments on Tiny ImageNe
to verify the performance of our method on larger dataset. Our method is compared against Mix- Match (Berthelot al. 2019b), RealMix (Nair et al., 2019). ReMixMatch (Berthelot et al.) (019a), and FixMatch (Sohn et al. 2020). As recommended by Oliver et al. 2018), all methods should be implemented using the same codebase. However, due the limited computing resources, we only re-implemented MixMatch and FixMatch. Our target is not reproducing wfte-of-theeart results of these papers, but making the comparison with our method as fair as possible.
4.1 IMPLEMENTATION DETAILS
Unless otherwise noted, we utilize Wide ResNet-28-2 network architecture (Zagoruyko & Ko- modakis 2016) with 1.5 million parameters, and our experiments are trained for 128 epochs with batch size of 64. Concretely, for our RankingMatch, we a same of hyperparameters (B 64, = 7, T 0.95, m = 0.5, T 0.2, X = 1. and X 1) across all datasets and all amounts of labeled samples except that batch size of 32 (B 32) used for the STL-10 dataset. More details of the training protocol and hyperparameters will be reported in Appendix D In all our ex- periments, RA and FixMatch refer to FixMatch with using RandAugment and CTAug- ment respectively (Sohn et al. 2020); RankingMatch BM, MathingMatch BH RankingMatch and RankingMatch refer to Ranking Match with using BatchMean Triplet loss, BatchHard Triplet loss, BatchAll Triplet loss, and Contrastive loss respectively. For each benchmark dataset, our results are reported on the corresponding test set.
4.2 CIFAR-10 AND CIFAR-100
Results with same settings We first implement all methods using the same codebase and evaluate them under same conditions to show how effective our method could be. The results are shown in Table Note that different folds mean different random seeds. As shown in Table T RankingMatch outperforms all other methods across all numbers of labeled samples, especially with a small portion of labels. For example, on CIFAR- 100 Ranking with 40 labels reduces the error rate by 29.61% % and 4.20% compared to MixMatch and FixMatch respectively. The results also show that cosine similarity might be more suitable than Fuclidean distance if the dimension of the "logits" score grows up. For instance, on CIFAR- 100 where the "logits" score is 100-dimensional vector, RankingMatch" reduces the error rate by 1.07% and 1.19% compared to Ranking BM in the case of 2500 and 10000 labels respectively.
Stanford University. http::/ca23 stt.

---

Under review as a conference paper at ICLR 2021
Table 1: Error rates (%) for CIFAR-I0 and CIFAR-100 on five different folds. All methods are implemented using the same codebase.
CIFAR-10
CIFAR-100
Method
40 labels
250 labels
4000 labels
400 labels
2500 labels
10000 labels
MixMatch
44.83+8.70
19.46
7.74=0.21
02.70=0.78
48. 18.98=00.88
35.11
FixMatch RA
19.42:66.46
7.30+
4.84=0.23
61.02++
38.17=0.40
30.23++0.43
RankingMatch RM
15.22++4.51 16.66++2.77
6.77+t0.89
4.76=#0.1
60.59
08.26+00.99
30.46:+0.24
RathingMatch
7.26++1.20
4.81=
64.26=+0.00
37.19+0.555
29.27+t0.30
CIFAR-10 with more training epochs Since FixMatch, which our RankingMatch is based on, was trained for 1024 epochs, we attempted to train our models with more epochs make our results comparable. Our results on CIFAR-10 10, which were trained for 256 epochs, are reported on the left of Table 2 RankingMatch BM achieves state-of-the-ar result, which is 4.87% error rate with 250 labels, even trained for fewer epochs compared to FixMatch. With 40 labels. RankingMatch BM has worse performance compared to Fatch but reduces the error rate by 0.38% compared to FixMatchRR, Note that both Ranking BM and FixMatch RA use RandAugment, so the results are comparable.
CIFAR-100 with larger model We use larger version of Wide ResNet-28-2 network archi- tecture, which is called Wide ResNet-28-2-Large by using more filters per layer. Specifically, the number of filters of layers in the second group is increased from 32 to 135, and similarly for the following groups with multiplication factor of 2. This Wide ResNet-28-22Laagg network architec- ture with 26 million parameters comparable. As shown on the right of Table 2 RankingMatch achieves a state-of-the-an result, which is 22.35% error rate with 10000 labels, even trained for only 128 epochs.
also used in MixMatch and FixMatch, making results
Table Comparison with state-of-the-art methods in error rate (%) on CIFAR-10 and CIFAR-100. Methods marked by denote results cited from respective papers. On CIFAR-10, our models are trained for 256 epochs. For CIFAR- 100, Wide 2--Neet22-22Large network architecture is used.
CIFAR-10
CIFAR-106 -00
Method
40 labels
250 labels
4000 labels
400 labels 2500 labels
10000 labels
MixMatch*
11.08++0.87
6.24-00.06
25.88
RealMix
9.79:10.75
6.39-10.27
ReMixMatch**
6.27:10.4
5.14:10.4
FixMatch
13.8113..77
5.07:00.5
4.26=0.05
48.85
28.29110111
22.60:10.12
FixMatch CTA
11.39-13.35
5.07+10.33
4.31=0.15
49.95=3.01
28.6440.24
28.1111..11
RankingMatch
13.43:1:233
4.87:0.08
4.29-10.3
49.57:0.67
29.6810.60
18.1030
RathingMatch"
14.98+13.06
5.13+-0.02
4.32-t0.12
16.90
28.39
22.35
4.3 SVHN AND STL-10
SVHN The results for SVHN are shown in Table 3 We achieve state-of-thed ohethe-art results, which are 2.24% and 2.23% error rate in the case of 250 and 1000 labels, respectively. With 40 labels, our results are worse than those of FixMatch; this may be excusable because our models were trained for 128 epochs while FixMatch's 's models were trained for 1024 epochs.
STL-10 STL-10 a dataset designed for unsupervised learning, containing 5000 labeled images and 100000 unlabeled images. To deal with the higher resolution of images in the STL-10 dataset (96 x 96), we add one more group to the Wide ResNet-28-2 network, resulting in Wide ResNet-37-2 architecture with 5.9 million parameters. There
pre-defined folds with 1000 labeled images each. Table shows our results on three of these ten folds. The result of SWWAE and CC-GAN cited from Zhao et al. (2015) and Denton et al. (0066) respectively. We achieve better results
7

---

Under review as a conference paper at ICLR 2021
compared numerous methods. Our RankingMatch RM obtains rate of 5.96% while the current ofte-offthheart method (FixMatch) has the error rate of 7.98% and 5.17% the case of using RandAugment and CTAuummett respectively.
Table 3: Comparison with state-of-the-an methods in error rate (%) on SVHN, denotes the results cited from respec- tive papers. Our results are reported on five different folds.
Table Error rates (%) STL-10 1000-label splits. denotes the results cited from respective papers.
Method
40 labels 250 labels
1000 labels
Method
Error Rate
MixMatch
3.7811..2
3.27.10.31
SWWAB"
25.67
RealMix
3.53
CC-GAN
22.21
ReMixMatch
3.10=0.50
2.83+0030
MixMatch
10.18:1
FixMatchKA
3.96=2.17
2.48+0.38
2.28
ReMixMatch*
6.18
FixMatch ClAg
7.65
2.64
2.36:10.19
FixMatch
1.98
FixMatch
5.17110.63
MixMatch
42.95.115.94
6.25.1017
5.87110.12
FixMatch
20.99
2.3710026
2.28.10.12
FixMatch
6,10+10.11
RankingMatch
21.02.8..06
2.24
2.32.10.07
RankingMatch
5.96+00.07
Match
27.20-2.90
2.33:00066
2.23:0.111
RankingMatch
7.55++0.37
4.4 TINY IMAGENET
Tiny ImageNet compact version of ImageNet, consisting of 200 classes. We use the Wide ResNet-37-2 architecture with 5.9 million parameters, used for STL-10. Only 9000 out of 100000 training images are used as labeled data. While FixMatch obtains an error rate 52.09 14%% Matc and RankingMatchhT achieve the error rate of 51.47 and 49.10:10.41 % respectively (reducing the error rate by 0.62% and 2.99% compared to RixMatch respectively). Moreover, RankingMatchh"T has the better result compared RankingMatch advocating the efficiency of cosine similarity in high-dimensional space, as presented in Section 4.2
4.5 ABLATION STUDY
Table 5: Ablation study. All results rates (%). Results of the same label split (same column) are reported using the same seed. In the second row, NaN-70 means the loss becomes NaN after 70 training steps, and similarly for other cases. means the methods without La-normalization.
CIFAR-10
CIFAR-100
SVHN
Ablation
250 labels
4000 labels
2500 labels
10000 labels
250 labels
1000 labels
RankingMatch RM
5.50
4.49
37.79
30.14
2.11
2.23
RankingMatch" RM1
NaN-70
NaN-310
NaN-730
NaN-580
NaN-30
2.29
RankingMatch
11.96
8.59
38.83
31.19
3.13
2.72
RatchngMatch
24.17
12.05
49.06
34.96
3.00
3.45
RatchngMatch
5.76
4.64
36.53
28.99
2.25
2.19
CathhngMatthhTi 6.31
4.67
36.77
29.15
2.43
2.20
We carried out the ablation study, summarized in Table 5 Our BatchMean Triplet loss out- performs BatchHard and BatchAll Triplet loss. For example, on CIFAR-10 with 250 labels, RankingMatch reduces the error rate by factor of two and four compared to Ranking and RankingMatch respectively. We also show the efficacy of Læ-normalization which con- tributes to the success of RankingMatch. Applying Ly nz-oommalizttion helps reduce the emror rate of RathingMatch across all numbers of labels. Especially for RankingMatch BM if L--nommalization not used, the model may not converge due the very large value of the loss.
8

---

Under review as a conference paper at ICLR 2021
4.6 ANALYSIS
Qualitative results We visualize the "logits" scores of the methods, as shown in Figure 2 While MixMatch has much more overlapping points, the other three methods have class cyan (cal) and yellow (dog) close together. Interestingly, RankingMatch has the shape of the clusters being different from MixMatch and FixMatch; this might open a research direction for future work. More details about our visualization are presented in Appendix
in
(a) MixMatch
(b) RixMatch
(c) RankingMatch
(d) RankingMatch
t-SNE visualization of the "logits" scores of the methods on CIFAR- 10 test set. The models were trained for 128 epochs with 4000 labels. The same color means the same class.
Figure 2:
Computational efficiency B the training time per epoch of RankingMatch with using BatchAll, BatchHard, or BatchMean Triplet loss on CIFAR-10, SVHN, and CIFAR-100. While the training time of HatchHard and BatchMean stable and unchanged among epochs, BatchAll has the training time gradually increased during the training process. Moreover, BatchAll requires more training time compared to BatchHard and BatchMean. For example, on SVHN, BatchAll has the average training time per epoch is 434.05 seconds, which is 126.25 and 125.82 seconds larger than BatchHard and BatchMean respectively.
CIFAR-10
SVHN
CIFAR-100
Training Sime Ejoch
BatchHard:
BatchAll: 365.87 15.35
3 460
2 450
BatchAll:
BatchHard::
323.03 : 8.1 (6)
(
BatchMean: 309.27 2 0.74
3
BatchMean:
309.33 1.03 (9)
420
820
X20
Luni 2
590
BatchAll:
634.05 + 20.07 (9)
Home
BatchHard:
540
BatchMean:
308. 23 0.57 (4)
540
200
120
120 220
200 110
Epoch
Epoch
Figure 3: Training time per epoch (seconds) during 128 epochs.
We also measure the GPU memory usage of the methods during the training process. On average, BatchAll occupies two times more GPU memory than BatchHard and BatchMean. For instance, on CIFAR-10, the GPU memory usage of BatchAll is 9039.72:+043.. 30MMB while this value is 1845. in BatchHard and BatchMean, More details are presented in Appendix
CONCLUSION
In this paper, we propose RankingMatch, novel semi-supervised learning (SSL) method that uni- fies the idea of consistency regularization SSL approach and metric learning. Our method encour- ages the model to produce the same prediction for not only the different augmented versions of the same input but also the samples from the same class. Delving into the objective function of metric learning, we introduce new variant Triplet loss, called BatchMean Triplet loss, which has the advantage of computational efficiency while taking into account all samples. The extensive experi- ments show that our method exhibits good performance and achieves state-of- of-te-of-theann art results across many standard SSL benchmarks with various labeled data amounts. For future work, ested in researching the combination of Triplet and Contrastive loss in a single objective function so that can take the advantages of these two loss functions.
inter-

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang. and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In International Conference on Learning Representations, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5049-5059 2019b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv: 200000 2020a.
Yun-Chun Chen, Chao-Te Chou, and Yu-Chiang Frank Wang. Learning to learn a semi-supervised fashion. arXiv preprint arXiv:2008. /203, 2020b.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223, 2011.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Emily Denton, Sam Gross, and Rob Fergus. Semi-supervised learning with context-conditional generative adversarial networks. arXiv preprint arXiv:1611.06430) 2016.
Terrance De Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv::708. 04552,, 2017.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0778 2018.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances neural information processing systems, pp. 529-536, 2005.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense the triplet loss for person re- identification. arXiv preprint arXiv: /103:0770007777 2017.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arXiv: 1712.0777000000 2017.
Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey, Symmetry, 11 11(9):1066, 2019.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint /9X2v: 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Brian Kulis et al. Metric learning: A survey, Foundations and trends in machine learning, 5(4): 287-364, 2012.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 1rX.::1610.02222.. 2016.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, volume 3, 2013.
10

---

Under review as a conference paper at ICLR 2021
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: (6t.00980803883,,, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2570))2557- 2008
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. Proceedings of the European Conference on Computer Vision (ECCV), pp. 181- 196, 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1999 2018.
Varun Nair, Javier Fuentes Alonso, and Tony Beltramelli. Realmix: Towards realistic semi-
supervised deep learning algorithms. arXiv preprint arXiv:1912.08766, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervised learning algorithms. In Advances in neural information processing systems, pp. 3235-3246 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 106633 2019.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems, pp. 1163-1171, 2016.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 315-823, 2015.
Kihyuk Sohn, David Berthelot, Chun-Liang Li. Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying ceni-supervieed learning with consistency and confidence. arXiv preprint arXiv:2001.06685 2020.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Sergey Zagoruyko and Nikos Komodakis. 1605.077 2016.
Wide
residual networks.
arXiv preprint
Yingying Zhang, Qiaoyong Zhong, Liang Ma, Di Xie, and Shiliang Pu. Learning incremental triplet margin for person re-identification. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 33, pp. 9243-9250, 2019.
Junbo Zhao, Michael Mathieu, Ross Goroshin, and Yann Lecun. Stacked what-where auto-encoders. arXiv preprint arXiv:1506.02355/ 2015.
11

---

Under review as a conference paper at ICLR 2021
A DETAILS OF OUR MOTIVATION AND ARGUMENT
For our motivation of utilizing Ranking loss in semi-supervised image classification FixMatch (Shnn et al. (020) is simple combination of existing semi-supervised learning (SSL. approaches such as consistency regularization and pseudo-labeling, FixMatch, as well as the consistency regu- larization approach, only considers the different perturbations of the same input. The model should produce unchanged with the different perturbations the same input, but this not enough. Our work is to fulfill this shortcoming. Our main motivation is that the different inputs of the same class (for example, two different cat images) should also have the similar model outputs. We showed that by simply integrating Ranking loss (especially proposed BatchMean Triplet loss) into Fix FixMatch, we could achieve the promising results, as quantitatively shown in Section
For our argument We argue that the images from the representations strictly, but their model outputs should be as similar as possible. Our work aims to solve the image classification task. Basically, the model for image classification consists of two main parts: feature extractor and classification head. Given an image, the feature extractor is responsible understanding the image and generates the image representation. The image representation is then fed into the classification head to produce the model output (the "logits" score) which is the scores for classes.
class do not have have similar
If the feature extractor can generate the very similar image representations for the images from the same class, it will be beneficial for the classification head.
Otherwise, if these image representations are not totally similar, the classification head will have to pay more effort to produce the similar model outputs for the same-class images.
Therefore, the model outputs somehow depend on the image representations. For image classifica- tion, the goal is to get the similar model outputs for the same-class images even when the image representations are not totally similar. That is also the main motivation for to apply Ranking loss directly to the model outputs. Figure Souureeillusrratte illluttrates the image representations and model outputs of model when given same-class images.
Cat
Ship
Input image
Image representation
Cosine similarity: 0.6813
Cosine similarity: 0.7814
Model output ("logits" score)
Cosine similarity: 0.9633
Cosine similarity: 0.9924
Prediction
Cat 100%))
Cat (s:100%)
Ship (=100%)
Ship (=-10%%)
Figure 4: Illustration of image representation and model output on CIFAR- 10.
As shown in Figure given two images from the same class, although the model can exactly predict the semantic labels and get the very similar model outputs, the image representations are not totally similar. For instance, two cat images can have the model outputs with the cosine similarity of 0.9633, but the cosine similarity of two corresponding image representations is only 0.6813. To support why applying Ranking loss directly to the model outputs is beneficial, visualize the image representations and model outputs of our method on the CIFAR- dataset, as shown in Figure
12

---

Under review as a conference paper at ICLR 2021
bed
32-5
ANC TOTW
TUS
(a) Image representation
(b) Model output ("logit" score)
Figure 5: t-SNE visualization the image representations and model outputs on CIFAR-10 test set. The model was trained for 128 epochs with 4000 labels. The same color means the same class.
As illustrated in Figure 5b the model outputs of the samples from the same class are clustered relatively well. As result, the image representations the same-class samples are also clustered relatively well, as shown in Figure Consequently, by forcing the model outputs of the same-class samples to as similar as possible, we obtain the similar image representations as well.
B COMPARISON OF METHODS
we evaluate our method against four methods: MixMatch (Berthelot et al. 2019b). RealMix Nair et al., (2019), ReMixxatch (Berthelot et al.) (019a). and FixMatch (Sohn et al. 2020). The comparison of the methods shown in Table 6 RankingMatch Ranking
As presented in Section 4
RankingMatch and RankingMatch refer to Ranking with using SH BH
BatchAll Triplet, BatchHard Triplet, BatchMean Triplet, and Contrastive loss respectively.
Table 6: Comparison of methods.
Data augmen- tation
Pseudo-label processing
post-
Ranking loss
Method
Note
Uses squared Lu loss for labeled data
MixMatch
Weak
Sharpening
None
Sharpening & Confi- dence threshold
Uses training signal anneal- ing (TSA) to avoid overfitting
RealMix
Weak & Strong
None
Uses extra rotation loss for unlabeled data
ReMixMatch
Weak & Strong
Sharpening
None
FixMatch
Weak & Strong
Confidence threshold
None
BatchAll Triplet loss
RankingMatch"
Weak & Strong
Confidence threshold
BatchHard
RankingMatch BH
Weak & Strong
Confidence threshold
Triplet loss
BatchMean
RankingMatch BM
Weak & Strong
Confidence threshold
Triplet loss
Contrastive
Matchng
Weak & Strong
Confidence threshold
loss
13

---

Under review as a conference paper at ICLR 2021
C RANKINGMATCH ALGORITHM
The full algorithm of RankingMatch is provided in Algorithm [ Note that the meaning of a, n, yaa Yp- yn, da,ps da.n. and f()) in Algorithm were defined in Section 3.3 Algorithm iillsstrates the use of BatchMean Triplet loss. When using Contrastive loss, most of the parts of Algorithm are kept unchanged, except that LBM and CBM are replaced by Contrastive losses as presented in Section 3.3.
Algorithm 1: RankingMatch algorithm for computing loss function with using BatchMean Triplet loss.
Input: Batch of В labeled samples and their one-hot labels X' = (Ir. 4)) : be (L...., B)}, batch of μB unlabeled samples = {u, b E (1, pB)). }. confidence threshold т, margin m, and loss weights Au and Xy.
/x
Compute loas aiementa for labeled data
for b 1 to B do Sn - (1y (wo)): // "logits" score for weakly-auu auguented labeled data
end
B H(,, Softmax(2s)) // Cross-Entrapy loss for labeled data Σ
LCE
B
X (L2Norm($s) b E (I...,B)) // Batch of B normalized "logits" scozes
weakiy
Peeaki--uummettee augmented labeled data
L BM
Σ
f(m +
1
Σ da.p
Σ damn) Batchwean Triplet
[
X
PE
neX
ya
labeied data
Compute losa etements unlabeled
for = to μB do
qb Pmodel (3) Aw (us);
"logits" score augmentee unlabeled data (6 Softmax (b) // Model prediction for wookly-augmented unlabeled data
11
I argmax (bb) // One-hot poesdc-label for acrongly-oummnnted unlabeled data
Üs Pmodel (1 (uu)); // "logits" scoze for strongly augmented unlabeled data
14 end
LCE
(max(()) 2 T) H(4v, Softmax((u)) // Crose Czose-Bntropy loss uniabeled
data
16 ú (L2Norm(ar) : uLB}
/1 Batch of nB La normallzed "logits" scores for
strongly augmented unisbeled
17 L BM
Σ f(mm+
Σ
dap
Σ dan)) // Setchilean triplet loss for nEU Un t3a
[
aEú
pEll
Up =0
unlabeled data
18 /-
Compute the cotal loss
19 L = LCE + A. + (.BB + LBM)
return L
D DETAILS OF TRAINING PROTOCOL AND HYPERPARAMETERS
D.1 OPTIMIZER AND LEARNING RATE SCHEDULE
We use the same codebase, data pre-processing, optimizer, and learning rate schedule for methods implemented by us. An SGD optimizer with momentum used for training the models. Addition- ally, apply cosine learning rate decay (Loshchilov & Hutter 2016) which effectively decays the learning rate by following a cosine curve. Given a base learning rate 7,, the learning rate at the training step is set to
14

---

Under review as a conference paper at ICLR 2021
7
cos
(8)
16s
where S is the total number of training steps.
Concretely, S equal to the number of epochs multiplied by the number of training steps within one epoch. Finally, we use Exponential Moving Average (EMA) obtain the model for evaluation.
D.2 LIST OF HYPERPARAMETERS
For all our experiments, we use
A batch size of 64 for all datasets except that STL--10 uses batch size of 32,
Nesterov Momentum with momentum of 0.9,
A weight decay of 0.0005 and base learning rate of 0.03.
For other hyperparameters, we first define notations as in Table 7
Table 7: Hyperparameter definition.
Notation
Definition
Tmiz
Temperature parameter for sharpening used in MixMatch
K
Number of augmentations used when guessing labels MixMatch
Hyperparameter for the Beta distribution used in MixMatch
Confidence threshold used in FixMatch and RankingMatch choosing high-confidence pre- dictions
Margin used in RankingMatch with using Triplet loss
Temperature parameter used in RankingMatch with using Contrastive loss
A hyperparameter weighting the contribution of the unlabeled examples to the training loss. In RankingMatch, the weight determining the contribution of Cross- Cross-Entropy loss of unla- beled data to the overall loss.
Au
A hyperparameter used in RankingMatch to determine the contribution of Ranking loss ele- ment to the overall loss
A
The details of hyperparameters for all methods shown in Table
Table 8: Details of hyperparameters. As presented in Section 4.1 FixMatch refers FixMatch
with using RandAugment; RankingMatch BM and RankingMatch" refer to RankingMatch with us- ing BatchMean Triplet loss and Contrastive loss respectively.
Method
An
Xx
Tina
C
T
m
soft-margin
Izg-orrmaiization
MixMatch
75
0.5
2
0.75
RixMatch RA
0.95
RankingMatch BM
0.95
0.5
True
True
RankingMatch CT
0.95
0.2
True
D.3
AUGMENTATION DETAILS
For weak augmentation, we adopt standard padding--ndderoppil and horizontal flipping augmen- tation strategies. We set the padding 4 for CIFAR-10, CIFAR-100, and SVHN. Because S0
15

---

Under review as a conference paper at ICLR 2021
and Tiny ImageNet have larger image sizes, padding of 12 and 8 is used for STL-10 and Tiny ImageNet, respectively. Notably, we did not apply horizontal flipping for the SVHN dataset.
For strong augmentation, first randomly pick 2 out of 14 transformations. These 14 transfor- mations consist of Autocontrast, Brightness, Color, Contrast, Equalize, Identity, Posterize, Rotate, Sharpness, ShearX, ShearY, Solarize, TranslateX, and TranslateY. Then, Cutout (DeVries & Tay- lor 2017) is followed to obtain the final strongly-augmentdd sample. We set the cutout size to 16 for CIFAR-IO, CIFAR-100, and SVHN. A cutout size of 48 and 32 is used for STL-10 and Tiny ImageNet, respectively. For more details about 14 transformations used for strong augmentation, readers could refer FixMatch (Sohn al. 2020).
D.4 DATASET DETAILS
CIFAR-10 and CIFAR-100 are widely used datasets that consist of 32 32 color images. Each dataset contains 50000 training images and 10000 test images. Following standard practice, as men- tioned in Oliver et al. (2018), we divide training images into train and validation split, with 45000 images for training and 5000 images for validation. Validation split is used for hyperparameter tun- ing and model selection. In train split, we discard all except a number of labels (40, 250, and 4000 labels for CIFAR-10: 10; 400, 2500, and 10000 labels for CIFAR-100) to vary the labeled data set size.
SVHN is a real-world dataset containing 73257 training images and 26032 test images. We use the similar data strategy as used for CIFAR- 10 and CIFAR-100, We divide training images into train and validation split, with 65937 images used for training and 7320 images used for validation. In train split, we discard all except a number of labels (40, 250, and 1000 labels) to vary the labeled data set size.
STL-10 is a dataset designed for unsupervised learning, containing 5000 labeled training images and 100000 unlabeled images. There are ten pre-defined folds with 1000 labeled images each. Given fold with 1000 labeled images, use 4000 other labeled images out of 5000 labeled training images as validation split. The STL-10 test set has 8000 labeled images.
Tiny ImageNet is a compact version of ImageNet, including 100000 training images, 10000 vali- dation images, and 10000 test images. Since the ground-truth labels test images are not available, 10000 validation images and them the test set. There 200 classes in Tiny ImageNet. We divide training images into 90000 images used for train split and 10000 used for validation split. For the semi-supervised learning setting, we use 10% of train split as labeled data and treat the rest as unlabeled data. As a result, there are 9000 labeled images and 81000 unlabeled images.
evaluate our method
E QUALITATIVE RESULTS
E. RANKINGMATCH VERSUS OTHER METHODS
To cast the light for how the models have learned classify the images, we visualize the "logits" scores using t-SNE which was introduced by Maaten & Hinton (2008). t-SNE visualization reduces the high-dimensional features reasonable dimension to help grasp the tendency of the learned models. We visualize the "logits" scores of four methods, which are MixMatch, FixMatch RankingMatch BM and Ranhing as shown in Figure These four methods were trained on CIFAR-10 10 with 4000 labels and were trained for 128 epochs with the same random seed.
At first glance Figure 6 both four methods tend to group the points of the same class into the same cluster depicted by the same color. The shape of the clusters is different among methods, and it is hard to say which method the best one based on the shape of the clusters. However, the less the overlapping points among classes are, the better the method is. We can casily that MixMatch (Figure (6a) has more overlapping points than other methods, leading to worse performance. This statement is consistent with the accuracy of the method. We quantify the overlapping points by
computing the confusion matrices, as shown in Figure 7
If pay more attention to t-SNE visualization in Figure 6 realize that all methods have many overlapping points between class (cat) and (dog). These overlapping points could be regarded as the confusion points, where the model misclassifies them. For example, as shown in the
16

---

Under review as a conference paper at ICLR 2021
3ed D
ANC
sPD SUZE
(a) MixMatch: Test accuracy of 92.26%
(b) FixMatch Test accuracy of 95.02%
pxtomobil
Euf
hores DrA
(c) RatcingMatch Test accuracy of 95.39%
(d) RankingMatch"" Test accuracy of 95.14%
t-SNE visualization of the "logits" scores of the methods CIFAR-10 test set. The models were trained for 128 epochs with 4000 labels. Note that this figure contains higher-resolution versions of the figures shown in Figure 2
Figure 6:
confusion matrices in Figure 7 MixMatch misclassifies 100 points as dog while they are actually cat. This number is 66, 60, or 64 in the case of FixMatchRA, RankingMatch RankingMatch respectively. We leave researching the shape the clusters and the relationship between t-SNE visualization and the confusion matrix for future work.
F.2 RANKINGMATCH WITH VARIANTS OF TRIPLET LOsS
Figure shows (-SNE visualization for the "logits" scores of the models in Table 5 in the case of trained CIFAR-IO with 4000 labels. Triplet loss utilizes series of triplets {a, p. n satisfy the objective function. Once the input was given, the loss function is optimized to minimize the distance between a and p while maximizing the distance between and n, implying that the way of treating the series of triplets might significantly affect how the model is updated. BatchAll, for instance, takes into account all possible triplets when calculating the loss function. Since BatchAll treats all samples equally, it is likely to be biased by the samples with predominant features, which might hurt expected performance. overlapping points and even gets lower accuracy by large margin compared to others. Especially at the center of the figure, the model is confusing almost all the labels. is thus natural to argue that BatchAll is poor generalizing unseen data. BatchHard (Figure 8b) is better than BatchAll, but still has many overlapping points at the center of the figure. Our BatchMean surpasses both BatchHard and BatchAll when much better clustering classes, leading to the best accuracy compared
shore up our argument, let see in Figure Sal BatchAll has numerous
17

---

Under review as a conference paper at ICLR 2021
066/100
1:20
1.5%
1.0%
400
0.4
+3.3%
12%
0.4%
1.4%
27.3%
315%
2.1%
1.0%
47.4%
17%
17
51%
11%
1.2%
5.9
51%
5.%
300
23
24
400
6.1%
1.7%
3.4%
2
a.m
6.2%
6:2%
5.7%
2
0.5%
0.7%
034%
6.4%
6.3%
5.5%
3.2%
1.2%
D1G
1.4
4P%
1.7%
(a) MixMatch: Test accuracy of 92.26%
(b) FixMatch RA, Test accuracy of 95.02%
10001.30
23
56112000
23
3.5%
0.34
2.5%
1.7%
1.0%
1:7%
1.7%
ES7%
9.7%
2.1%
5.7%
1.%%
1:00
1.5%
0L7%
6.1%
FLES
3.2%
5%
6.0%
1.5%
1.2%
2.2%
LG
0.4%%
3.3%
1%
0.4%
0.%
04%
MM
3.9%
1
0.%
10
LACE
400
10
3.3%
3.4%
0.5%
LO0
0.0%
444%
3.2%
3.5%
1.7M
502009
P.
PM
PM
3.4
1.5%
11.%
5%
1.5%
3.5%
1.7%
11.7%
(c) RankingMatch BM, Test accuracy of 95.3%%
(d) RankingMatch CT, Test accuracy of 95.1%%%
Figure 7: Confusion matrices for models in Figure 6 Classes in Figure 6 are numbered from 0 to 9, respectively.
to other methods. The confusion matrices shown in Figure P quantify overlapping points, which could be regarded as confusion points where the model misclassifies them.
E.3 RANKINGMATCH WITH L IZ-NORMALIZATION
We use the models reported in Table 5 in the case of trained on CIFAR-10 with 4000 labels. No- tably, we do not visualize RankingMatch without Lg-normalization because that model does not converge. t-SNE visualizations of the "logits" scores of RankingMatch"1 models and correspond- ing confusion matrices are shown Figure 10 and II respectively. There is not much difference between RankingMatchhT with and without L Ly-normalization in terms of the cluster shape and overlapping points. However, in terms of accuracy, L2 nommalliizttion actually helps improve classi- fication performance, as shown in Table 5
18

---

Under review as a conference paper at ICLR 2021
B Med D
AC
22
(a) RankingMatch RA Test accuracy of 87.95%
(b) RankingMatch""; Test accuracy of 91,41%
autormbbe
041
C2g Toos Trrck
(c) Rathing BM; Test accuracy of 95.51%
Figure 8: t-SNE visualization of the "logits" scores of RankingMatch with variants of Triplet loss on CIFAR-10 test set. The models were trained for 128 epochs with 4000 labels.
F COMPUTATIONAL EFFICIENCY OF BATCHMEAN TRIPLET Loss
As presented in Section
BatchAll Triplet loss considers all possible triplets when computing the loss.
BatchHard Triplet loss only takes into account the hardest triplets when calculating the loss.
Our BatchMean Triplet loss only considers the "mean" triplets (consisting of anchors, "mean" positive samples, and "mean" negative samples) when computing the loss.
Because BatchMean does not consider all triplets but only the "mean" triplets, BatchMean has the advantage of computational efficiency of BatchHard Triplet loss. On the other hand, all samples are used to compute the "mean" samples, BatchMean also takes into account all samples as done in BatchAll Triplet loss. The efficacy of BatchMean Triplet loss was proved in Table 5 when achiev- ing the lowest error rates compared to other methods. Therefore, this section only focuses on the contents of computational efficiency. Firstly, let us take simple example to intuitively show the computational efficiency of BatchHard and BatchMean against BatchAll Triplet loss. Assume we have an anchor a, three positive samples corresponding a: Pt. P2, and pa. and two negative samples with respect to a: nn and 1/2.
In BatchAll, there will have six possible triplets considered: (a, Pt, n1), (a, Pt. n2), (a, P2. n1) (a, P2, 122), (a, P3, n1), and (a, Ps, n2).
19

---

Under review as a conference paper at ICLR 2021
6%%
10%
12%
62%
23%
124
1.%
22
2.2%
0.
2s Ph
03%
02
024
221
144%
30%
62%
26.7%
62%
33%
2.0%
0.m
3.5
L.5
17%%
3.%
SUN
6.%
1%
7.4%
10%
51%
57%
Lo0%
530
10
13
20
0.25
0.273
2/2%
4.54
224
1.5%
3.7%
6.5%
6.2%
1.%%
3.8%
2
3.7%
100%
0.%%
1.4%
2.2%
33%
50.0%
6.5%
6.7%
3.3%
2.3%
233
3.2%
7.1h
5-0%
DTT
D3%
1.%
1.4
VVV
Loc
3.0%
Proct
(a) RankingMatch RA Test accuracy of 87.95%
(b) RankingMatch D11, Test accuracy 91.41%
0.%%
7%7%
1.%%
1.5%
1-0%
1.5%
117%
0.%%
6.6%
000
0.1%
64%
33%%
3
3-4%
1000
0.2%
6.2%
8.5%
1/th
P:1
6.7%
DALAL
0.5%
14%%
011%
1.1%
62-1%
(c) Ranking BM, Test accuracy of 95.51%
Figure 9: Confusion matrices for models in Figure g Classes in Figure are numbered from 0 to 9, respectively.
BatchHard only takes into account one hardest triplet: nearest(rı, n2))..
(a.
furthest(p1, P2, Pa).
Finally, in our BatchMean, there is only one "mean" triplet
considered:
(a,
mean(p1, P2, Ps), mean(n1, n2))..
As a result, BatchHard and BatchMean take fewer computations than BatchAll Triplet loss.
To quantitatively prove the computational efficiency of BatchHard and our BatchMean compared to BatchAll Triplet loss, we measure the training time and GPU memory usage, as presented in
Appendix F. and F2 We use the same hyperparameters for all methods to ensure fair com- parison. Notably, for clearance and simplicity, we use BatchAll, BatchHard, and BatchMean for Matchng RankingMatch and BM respectively.
20

---

Under review as a conference paper at ICLR 2021
B
B
AC
BXC
(a) RankingMatch Test accuracy of 95.36%
(h) RanhingMMttch without L Iy-Normalizaaton: Test accuracy of 95.33%
Figure 10: t-SNE visualization of the "logits" scores of Ratking with Contrastive loss CIFAR-10 test set. The models were trained for 128 epochs with 4000 labels.
35%
3,5
0.4%
0,2
0.7%
0.5%
1.2%
LE%
06.2%
6.%
6.7%
2.3%
2LN
+77%
32%
0%
0.%
0.7%
35%
0.1%
35.3%
3.%
1.7%
Lift
1:7
1.7%
9.1%
2.%%
5
L.E%
5.P
31%
5.5%
3.2%
11%
30%
L2h
7.%
L0%
0.2%
0.2%
64%
6.%%
6.%%
07.5%
2%
7.3%
15%%
57
522000
400
3.4%
3.5%
1.5%
0.2%
6.4%
8.5%
4%4%
2%
0
6.1%
0.7%
4%4%
XM
3.m
TT
2:44
1.7%
0-1%
5.4%
36V
5.5%
1.5
Tredittd
(a) Mathhng CI Test accuracy of 95.36%
RathingMatchh CT without Ly Norraaizzzai:::: Test
accuracy of 95.33%
Figure 11: Confusion matrices for models in Figure 10 Classes in Figure 10 are numbered from 9, respectively.
F.1 MEASUREMENT PER EPOCH
Table 9 shows the training time per epoch (seconds) and GPU memory usage (MB) of the methods for 128 epochs CIFAR-10, SVHN, and CIFAR-100. As shown in Table 9 BatchHard and BatchMean have the similar training time and the similar GPU memory usage among datasets. The results also show that HatchHard and BatchMean are more computationally efficient than BatchAll across all datasets. For example:
On SVHN, BatchHard and BatchMean reduce the training time per epoch by 126.25 and 125.82 seconds compared BatchAll, respectively.
21

---

Figure 12: Training time per epoch (seconds) and GPU memory usage (MB) during 128 epochs on Under review as a conference paper at ICLR 2021
BatchAll occupies much more GPU memory than BatchHard and BatchMean, which is about 1.87, 1.85, and 1.79 times on CIFAR-10, SVHN, and CIFAR-100 respectively.
Table 9: Training time per epoch (seconds) and GPU memory usage (MB) for 128 epochs on CIFAR- 10, SVHN, and CIFAR-100.
Training time per epoch
Method
CIFAR-10
SVHN
CIFAR-IOX
BatchAll
385.87=15.35
434.05=220007
323.03=8.81
BatchHard
30..99:0.79
307 0.7.
310.16=0.98
BatchMean
3.9.
00..23=0.57
309.33=1.03
GPU memory usage
Method
CIFAR-10
SVHN
CIFAR- 100
BatchAll
9039.72:1:243.330
8967.59 1535,999
8655.811:2299
BatchHard
4845.92:10.72
08.5.92:10.72
4847.86=0.97
BatchMean
4845.92:10.72
4845.92:10.72
0.99
The training time per epoch (seconds) and GPU memory usage (MB) are measured during 128 epochs, as illustrated Figure 12] In addition to computational efficiency against BatchAll, Batch- Hard and BatchMean have the stable training time per epoch and the stable GPU memory usage. On the other hand, the training time of BatchAll is gradually increased during the training process. Especially, there is time when the training time of BatchAll grows up significantly, and this time different among datasets. Moreover, it seems that the amount of computations of BatchAll is also different among datasets. These differences will be clarified in the following section (Appendix
CIFAR-10
SVHN
CIFAR-100
490
480
490
Training time per E20chh
0.0
BatchAll:
BatchHard:
308. 99 0.79
385.87 1 15.35 (6)
6663
MV
693
BatchAll:
Batchikard:
310. 16 0.98
323.03 1 8.1
(8)
(9)
33 (4)
600
8003
400
+ 307. 0. (6)
300
BatchHard:
300
BatchMean:
23 308.
10
00
100 310 220 130
100 110 120 130
100 310
Epoch
Epoch
Epoch
(a) Training time per epoch
BanchHird:
9039.72 2043.0
92 0.
(MB)
12000
BatchHard:
4845,
1535.90
(MP)-
12000
BatchHard:
865581
4847.86 +
2299.56
(MB)
BatchMean:4444
4845. 92.5 0. 0.
(MB)
BouchMeann
4545 + 0.72
(MB)
BatchMican:40
4847.8866
0.
[MB)
1000
2000
2000
GPU mamorry
7000
7000
7000
5000
Epoch
Epoch
Epoch
(b) GPU memory usage
CIFAR-10. 10. SVHN, and CIFAR-100.
F.2 MEASUREMENT PER BATCH
Different from the previous section (Appendix F.I. this section measures the training time and monitors the GPU memory usage per training step. Each training step corresponds to the training
22

---

Under review as a conference paper at ICLR 2021
batch. Table 10 and Figure 13 show the measurement of the methods for the first 5100 training steps on CIFAR-IB 10 and SVHN.
Table 10: Training time per batch (milliseconds) and GPU memory usage (MB) for the first 5100
training steps on CIFAR-10 and SVHN.
CIFAR-10
SVHN
Method
Time per batch
GPU memory usage
Time per batch
GPU memory usage
BatchAll
318.48--21.00
6643.7114222.58
371.044428.64
8974,99+2167.56
BatchHard
300.97:17.28
4844.00-3.10
302.2117.77
4843.9713.11
BatchMean
302.37:17.22
4843.9813.11
303.1811.87
4843.95::3.12
Table T0 demonstrates that BatchHard and our BatchMean much more computationally efficient than BatchAll. For instance, on SVHN, BatchHard and BatchMean reduce the training time per batch by 68.83 and 67.86 milliseconds compared BatchAll, respectively. It would be found more interesting in Figure
450
450
318.48 (m))
BatchAll:
371.04 28.64 (ms)
Barsh (152)
410
430
BatchMean:
Batchitard: 300. 97 7.28 (ms)
302.37 7.92
430
BatchHard:
302 21 7.77 (ms)
303. (m))
Traning time Batth
Training
270
270
250
250
500
1000
1500
2000
2500
5000
3500
4000
4500
5000
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Training step
Training stop
(a) Training time per batch on CIFAR-10
(b) Training time per batch on SVHN
12500
BatchABl: 6643.71 t 2222.58 (MB)
12500
BatchAll: 8974.89 : 2167.56 (MB)
11500
BatchHard: 4844. 00 : 3. 10 (MB)
11500
BatchHard:
4843.97. (M4)
BatchMaan:
4843.
BatchMean:
(MR)
10500
mymory urags urage
8500
7500
7500
6500
6500
5500
5500
4500
4500
500
1000
1500
2000
2500
3000 3500 4000
4500
5000
1500
2000 2500 3000 3500 0000 4500
Training step
Training stop
(c) GPU memory usage on CIFAR-10
(d) GPU memory usage on SVHN
Figure 13: Training time per batch (milliseconds) and GPU memory usage (MB) during the first
5100 training steps on CIFAR-IO and SVHN.
13b the "peak" values indicate the starting time of new epoch. At that time, there are some extra steps like initialization, so it might take more time. As shown in Figure 13 BatchAll starts to take more computations from the th and 500 training step on CIFAR-TO TO and SVHN, respectively; this is reasonable because predictions for unlabeled data (Section 3.2). At the beginning of the training process, the model is not well trained and thus produces the predictions with very low confidence, many samples discarded. As a result, there are a few possible triplets for unlabeled data at the beginning of the training process, leading to fewer computations of BatchAlI.
In Figure 13a
used a threshold to ignore the low-confidence
When the model is progressed. it is trained more and produces more high-confidence predictions, leading to possible triplets. Therefore, BatchAll has more computations. Figure 1 also shows
23

---

Under review as a conference paper at ICLR 2021
that the starting point of increasing the computation of BatchAll is earlier in the case of SVHN compared to CIFAR-10. This is reasonable because the SVHN dataset only consists of digits from O to 9 and thus is simpler than the CIFAR-I0 dataset. As result, it is easier for the model to learn SVHN than CIFAR-10, leading to more high-confidence predictions and more possible triplets at the beginning of the training process in the case of SVHN compared CIFAR-10. Moreover, the training time per batch and GPU memory usage of BatchAll on SVHN are larger than those on CIFAR-10 over the first 5100 training steps. Therefore, dataset is, the earlier and more BatchAll takes computations. This is also the reason for us to monitor the computational efficiency with more training steps on CIFAR-100.
can argue that the less complex the
Since CIFAR-100 100 has 100 classes, it is more complex than CIFAR- 10 and SVHN. Therefore, the model needs more training steps to be more confident. Table and Figure 14 show the training time per batch and GPU memory usage of the methods on CIFAR- 100 over the first 31620 training steps. BatchHard and our BatchMean are still more computationally efficient than BatchAll, but not too much especially for the training time per batch. To show discernible changes, we need to monitor the training time per batch and GPU memory usage with more training steps, as presented in the previous section (Appendix F.I.. Figure 14 also shows that BatchAll starts consume more computations at around the 17500* training step, which much later than on CIFAR- and SVHN.
Training time per batch (milliseconds) and GPU memory usage (MB) for the first 31620 training steps CIFAR-100.
Table 11:
Method
Time per batch
GPU memory Usage
BatchAll
306.32.18.48
$255.91 593.29
BatchHard
303.29::7.67
4846.80-:22.58
BatchMean
302.86:1..99
4846.77
370
6500
BatchAll:
306.32 + 848 (ms)
BatchAll:
5255.91 t: 593.29 (M1)
Trannins Lint par Batch (ns)
BatchHard:
303.29 7.67 (ms)
BatchHHrdd
2.58
350
DatchMeen:
(mss)
61.00
BatchMMaan
4846. (MB)
memory USagg
330
5700
320
5500
5300
300
$100
280
4700
10000
20000
25000
30000
5000
15000
20000
30000
Training stop
Training step
(a) Training time per batch on CIFAR-100
(b) GPU memory usage on CIFAR-100
Figure 14: Training time per batch (milliseconds) and GPU memory usage (MB) during the first 31620 training steps on CIFAR-100.
24


Reviews:
[{"review_id": "JJXd0Dk1ppq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of semi-supervised learning, where the training set contains both labeled images and unlabeled images. To address this task, the paper proposes a method called RankingMatch which exploits the idea that input images having the same label should have similar model outputs. To incorporate this idea into training, the paper develops a BatchMean Triplet loss to guide the model learning from the unlabeled images. Experiments are conducted on the CIFAR-10, CIFAR-100, SVHN, and STL-10 datasets. Results of RankingMatch show some improvements over competing methods in some cases.\n\nQuality: The presentation of this paper is clear. Figure 1 clearly illustrates the pipeline of the framework. This paper reads well and is easy to follow.\n\nClarify: This paper clearly presents the proposed method and the motivation of each design choice is clearly described. While some of the components are from existing papers, the paper clearly acknowledges the source. This allows me to understand which parts of the method are built upon existing approaches and helps me better identify the contributions and new components of this paper.\n\nOriginality: Generally, the idea of this paper makes sense. However, there are several parts similar (if not the same) to existing methods. The cross-entropy loss in Equation 2 is a standard loss function that is commonly adopted in classification tasks. The loss in Equation 3 is from FixMatch (mentioned in the paper as well). The triplet loss and contrastive loss are two loss functions that are widely applied in recognition and representation learning tasks. All of these loss functions to me are standard and common knowledge in the community. The idea of applying ranking loss on the model outputs is highly similar to a recent paper published in ECCV 2020 [a]. The idea of learning from the unlabeled data in [a] is to leverage class-wise similarity scores to form a representation and use this representation to describe an unlabeled data. For a pair of unlabeled images, [a] uses the class-wise similarity representation to assign positive/negative labels to that pair of images by comparing the distance between the two similarity representations. After assigning pseudo labels to that pair of images, [a] applies contrastive loss to further learn from the unlabeled data. The idea in this paper is very similar. The output of the model (classifier) can also be viewed as a form of class-wise similarity distributions. Namely, for each unlabeled image, the classifier outputs a distribution that indicates the similarity of that input image to each class of the labeled set. To me, the high-level idea is similar. However, the authors did not acknowledge the similarity with [a] in the submission. This will make readers feel like the idea of applying ranking loss on the model outputs is original in this paper.\n\n[a] Chen et al. Learning to learn in a semi-supervised fashion. In ECCV, 2020. https://arxiv.org/pdf/2008.11203.pdf\n\nSignificance: Given that some parts of this paper are highly similar to [a], the significance of this paper is downplayed. Also, there are several components in this paper are from existing/well-known papers. Based on these grounds, the contribution of this paper is limited.\n\nRequest for author response: I would like to see how the authors compare their paper with [a] in terms of the idea of class-wise similarity and representation learning. In particular, I would like to know the pros and cons of this paper compared to [a]. It would be great if the authors can talk about whether the semantics-oriented similarity representation in [a] could be used (and how to use it) to help improve the performance of the proposed method in the setting concerned by the paper.\n\nRating: Given that there are many parts similar to [a] and they are not acknowledged in the paper and many of the components in the paper are from existing approaches, I can only rate 4 for this submission at this point. I will reevaluate this paper after seeing the reviews from the other reviews as well as the author response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "65zhsCeEq0n", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "********Summary\nIn this paper, they introduced a new objective function, dubbed BatchMean Triplet loss, for Semi-supervised learning on image classification problems. In this framework both labeled and unlabeled data are used for training at the same time. The labeled data will be augmented weakly and cross entropy loss as well as ranking loss will be computed on their output. The unlabeled data will be augmented weakly and strongly, and the label based on the weakly augmented will be a pseudo label for the strongly augmented input, and similarly cross entropy and rank loss will be computed for the unlabeled data. For the rank loss, they used Contrastive loss and BatchMean Triplet loss, which is a variant of Triplet loss. They applied Triplet and Contrastive loss to the model output instead of image representation.\n\n********Positives\n- The paper is well-written and well-organized. Figure 1 is self-explanatory, showing the general framework introduced in this paper.\n\n- They covered background and related works to this paper when explaining their method, which makes the work clear.\n\n********Notes\n- My main concern about this work is lack of contribution. This could be negligible if I see a better result in their experiments section. Table 1, their performance is very similar to FixMatch(RA). I do not see much improvement of their method over baselines in Table 2 (they are on CIFAR-10 and CIFAR-100 only).\n\n- They have mentioned that \"Our BatchMean Triplet loss has the advantage of computational efficiency of existing BatchHard Triplet loss while taking into account all input samples.\" It would be interesting to quantitatively see how much their method is computational efficient.\n\n- It would be interesting to see the comparison of two ranking losses explained here (BatchMean Triplet loss vs Contrastive loss).\n\n- I would like to see the comparison of their method against baselines on all of the datasets explained here. I understand the limited computing resources issue, that is really unfortunate that some institutes cannot provide enough computational resources for their researchers experiments. But as a reviewer, I need to see how this method works on a wider range of datasets.\n\n********Reason to accept or reject\nThe paper is well-written and they explained their method clearly, which is important. My concern is lack of contribution. If they could provide more thorough experiments, this could be more helpful work for other researchers in this area. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "35ADZa7bqBk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents an SSL method extending FixMatch by introducing an auxiliary loss motivated from the metric learning literature. For example, the triplet loss is utilized to define the loss, where any triplet of anchor, positive and negative examples, either using ground-truth labels for labeled data or pseudo-labels for unlabeled data, are used to compute the loss. Variants using hard triplets of the mean as well as using the contrastive loss are proposed. The proposed methods are evaluated on standard SSL benchmarks.\n\nReview comments:\nWhile authors presented ablation study with various metric learning losses proposed in this work, the biggest concern is that none of these losses shows the clear performance improvement over FixMatch when combined with FixMatch losses. This begs the essential question on the effectiveness of the proposed losses on top of FixMatch. If RankingMatch is not clearly more performant than FixMatch, is there any scenario that FixMatch cannot be applied but RankingMatch can? \n\nIt is also unclear to me the motivation for introducing auxiliary metric learning losses -- why is it a good addition to the FixMatch loss? Which aspect of FixMatch loss is problematic and how does adding metric learning loss fix that issue?\n\n\"We argue that the images from the same class do not have to have similar representations strictly, but their model outputs should be as similar as possible\" --> There seems no supporting empirical evidence to this statement.\n\nReason for decision:\nGiven the fact that the proposed method is constructed incrementally to the previous method (i.e., additional loss to FixMatch loss), the initial decision criteria is its empirical effectiveness. Unfortunately, I am not convinced that the proposed method is clearly improving upon previous methods in any settings considered in the paper, so I would recommend for rejection.\n\nI have read author response and thank authors for their response. I have decided to keep my initial rating. I am not convinced by the response that the proposed method has a clear advantage over Fixmatch as some of their reported numbers for Fixmatch are very different (worse) from the ones from the original paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1A7wARY3sl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel method, called RankingMatch to the problem of semi-supervised learning (SSL). It extends known consistency regularization - based SSL methods by adding an additional loss that encourages similarity of outputs for samples of same class. In addition to, the paper proposes a new version of the triplet loss, called Batch Mean Triplet loss (BMT).\n\nIn general, there are some innovative and novel parts in this work, but in total it is somewhat incremental since big parts of the method and the results result from FixMatch, which serves as the basis for this method. The experimental results are promising, although not conclusive, for instance with larger amount of classes in cifar100, a different version of the method works better compared to a dataset with less classes, cifar10 (see Tables 1 and 2).\n\nPros:\n\nThe proposed idea to regularize the class output consistency between samples of the same class is interesting and should be studied further by the research community.\n\nAnother interesting idea is the BatchMean triplet loss that seems to be novel and could also be utilized in other settings.\n\n\nCons:\n\nThe clarity of the paper could be improved by clearly stating the differences and similarities to the closest method (I assume FixMatch in this case) in other parts than the intro as well. This applies to all variants of the proposed RankingMatch method.\n\nOne problem I see in this work is that the method is based on a relatively ad-hoc idea of class output similarity between samples of the same class. This assumption might hold for some datasets and not for others - this should have been studied in itself for various datasets. Definitely there may be correlation for many datasets, but one could imagine that for instance in tasks with big amount of classes, the correlation could sometimes be relatively low. For instance, in Imagenet, the output could include other objects that accidentally happen to be in the image together with the main object. Anyway it is interesting to see it works to some extent with the simple datasets, but it should be tested with Imagenet like was done with FixMatch.\n\nAnother problem is the batch-specific nature of the proposed loss variants that require also samples from the same class. This is not typically a problem in non-class-specific contrastive losses, but in this case, especially when the amount of classes increase (e.g., 1k or 10k), the proposed may start to perform worse when the same-class samples co-appear less often in the same batch. \n\nEdit: I have read the changes and the author responsed, but have decided the keep the score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Despite the performance gains of RankingMatch over the benchmarks used in the paper, the reviewers remained concerned about how the paper compares to state of the art in several respects.
Decision: Reject