X-model: Improving Data Efficiency in Deep Learning with A Minimax Model
Authors: Ximei Wang
Xinyang Chen
Jianmin Wang
Mingsheng Long
Venue: ICLR 2022 Poster (2022)
Keywords: Data Efficiency
Deep Learning
Minimax Model

Abstract: To mitigate the burden of data labeling, we aim at improving data efficiency for both classification and regression setups in deep learning. However, the current focus is on classification problems while rare attention has been paid to deep regression, which usually requires more human effort to labeling. Further, due to the intrinsic difference between categorical and continuous label space, the common intuitions for classification, \textit{e.g.} cluster assumptions or pseudo labeling strategies, cannot be naturally adapted into deep regression. To this end, we first delved into the existing data-efficient methods in deep learning and found that they either encourage invariance to \textit{data stochasticity} (\textit{e.g.}, consistency regularization under different augmentations) or \textit{model stochasticity} (\textit{e.g.}, difference penalty for predictions of models with different dropout). To take the power of both worlds, we propose a novel \Chi-model by simultaneously encouraging the invariance to {data stochasticity} and {model stochasticity}. Further, the \Chi-model plays a minimax game between the feature extractor and task-specific heads to further enhance the invariance to model stochasticity. Extensive experiments verify the superiority of the \Chi-model among various tasks, from a single-value prediction task of age estimation to a dense-value prediction task of keypoint localization, a 2D synthetic and a 3D realistic dataset, as well as a multi-category object recognition task.

Published as a conference paper at ICLR 2022
X- X-MODEL: IMPROVING DATA EFFICIENCY IN DEEP LEARNING WITH A MINIMAX MODEL
Ximei Wang, Xinyang Chen, Jianmin Wang, Mingsheng Long (Ex)
School Software, BNRist, Tsinghua University, China
wxm1 7@mails ts tsinghua. edu. en, cwwnxinyangg com j iwwangetss nnghua edu. en, minnghheng@tsinghuaa edu. cn,
ABSTRACT
To mitigate the burden of data labeling, aim improving data efficiency for both classification and regression setups in deep learning. However, the current focus is classification problems while rare attention has been paid to deep regression, which usually requires more human effort to labeling. Further, due the intrinsic difference between categorical and continuous label space, the common intuitions for classification, e.g. cluster assumptions or pseudo labeling strategies, cannot be naturally adapted into deep regression. To this end, we first delved into the existing data-efficien methods in deep learning and found that they either encourage invariance to data stochasticity (e.g., consistency regularization under different augmentations) or model stochasticity (e-g., difference penalty for predictions of models with different dropout). To take the power of both worlds, we propose novel X-Model by simultaneously encouraging the invariance to data stochasticity and model stochasticity. Extensive experiments verify the superiority of the X-Model among various tasks, from a single-value prediction task of age estimation to a dense-value prediction task of keypoint localization, a 2D synthetic and 3D realistic dataset, well as a multi multi-category object recognition task.
INTRODUCTION
In the last decade, deep learning has become the de facto choice for numerous machine learning applications in the presence of large-scale labeled datasets. However, collecting sufficient labeled data through manual labeling, especially for deep regression tasks such as keypoint localization and age
estimation, prohibitively time-costly and labor-expensive in real-world scenarios. To mitigate the requirement of labeled data, great effort (Lee) 20133] Laine & Aila 2017 Grandvalet & Bengio 2005 Sohn et al., 2020) Chen et al. (020b) has been made to improve data efficiency in deep learning from the perspectives of simultancously exploring both labeled and unlabeled data based on the intuitions for classification, e.g. cluster assumptions, pseudo labeling strategies, consistency regularization under different data augmentations.
However, most of the existing methods of improving data efficiency focus on classification setup while rare attention has been paid to the other side of the coin, i.e. deep regression which usually requires more human effort or expert knowledge to labeling. Moreover, due to the intrinsic difference between categorical and continuous label space, the methods based on the cluster or low-density separation assumptions (Lee) 2013; Grandvalet & Bengio (005) for data-cfficient classification tasks cannot be directly adapted into deep regression. Mcanwhile, existing data-efficient regression methods adopt k-nearest neighbor (kNN) (Zhou & Li. 20056] Yu-Feng Li 2017). decision tree (Levati et al. 2017) Gaussian Process (Srijith et al. 2013) regressors
fixed and shallow feature space, causing them difficult to be extended into deep learning problems.
To develop general data-efficient deep learning method for both classification and regression setups, we first delved into the existing methods and found that they can be briefly grouped into two categories: 1) Encourage invariance to data stochasticity with consistency regularization to make the predictions of the model invariant to local input perturbations, such as 11-model (Lainc)
"Correspondence Mingsheng Long imingsheng@@tsinggna..coo

---

Published as a conference paper at ICLR 2022
Pad() Crop() Flip()
Color() Sharp()
Cutout() Bright() Contrast
Rotate() Affine()
Data
Data Stochasticity: Weak
Data Stochasticity:
Strong
Random Initial()
?
Dropout()
R2
Noise()
(d) Model
(c) Model Stochasticity: Weak
(f) Model Stochasticity: Strong
Figure 1:
Comparisons among data-efficient techniques. To enhance the invariance to data stochas- ticity, strong data augmentation has been proposed. Similarly, can we propose novel approach to further enhance the invariance model stochasticity?
Aila 2017), UDA (Xie et al. 2020) and FixMatch (Sohn
2020 ; 2) Encourage invariance
to model stochasticity with difference penalty for predictions of models generated from different dropout (Laine & Aila 2017) or initialization (Zhou & Li 2005b), as well models exponentially
averaged from history models, such Mean Teacher (Tarvainen & Valpola 2017).
Through the success of these above two strategies of consistency regularization that encourages invariance under data and model transformations respectively, it is intuitive to draw conclusion that the invariance to stochasticity matters for improving data efficiency deep learning. To take the power of both worlds, we propose novel X-Model by simultaneously encouraging the invariance to data stochasticity and model stochasticity. First, instead of the weak augmentations (e.g., flip and erop) adopted in I1-model. FixMatch (Sohn et al. 2020) to enhance invariance to data stochasticity, as shown in Figure I with some example images tailored from Jung et al. (2020) A natural question arises: Can further enhance the invariance to model stochasticity similar to that of data stochasticity? This paper gives positive answer by introducing a minimax game between the feature extractor and task-specific heads. Compared to the manually designed strategy of adding dropout layer, this novel approach directly optimizes minimax loss function the hypothesis space. By maximizing the inconsistency between task specific heads, more diverse learners are generated to further enhance the invariance to model stochasticity and thus fully explore the intrinsic structure of unlabeled data.
utilize the strong augmentations (e.g.. cutout and contrast) adopted in
In short, our contributions can be summarized as follows:
We propose the X- MModel that jointly encourages the invariance to data stochasticity and model stochasticity to improve data efficiency for both classification and regression setups.
We make the -Model play a minimax game between the feature extractor and task-specific heads to further enhance invariance to model stochasticity.
Extensive experiments verify the superiority of the X-Model among various tasks, from an age estimation task a dense-value prediction task keypoint localization, 2D synthetic and 3D realistic dataset, well as multi-categon object recognition task.
2 RELATED WORK
2.1
CLASSIFICATION
In absence of abundant labeled data, it reasonable to further explore the additional unlabeled data. popular approach among these algorithms is Pseudo Labeling (Lee) 2013) which leverages the
2

---

Published as a conference paper at ICLR 2022
model itself to generate labels for unlabeled data and uses generated labels for training. Besides Pseudo Labeling, there is another family of algorithms under the umbrella of traiinnta which has received much attention both empirically (Sohn et al. 2020) and theoretically (Wei et al. 2021). They either enforce stability of predictions under different data augmentations (Tarvainen & Valpola |017 Xie al.) 2020] [Sohn et al., 2020) (a.k.a. input consistency regularization) or fit the unlabeled data on its predictions generated by a previously learned model (Lee, 2013; Chen et al. 2020b), Further, Co-Training (Blum & Mitchell 1998). Deep Co-Training Qiao et (2018) and
Tri Tri-Tnaining (Zhou & Li, 2005a) improve data effciency from an interesting perspective of different
views of classifiers. MixMatch (Berthelot et al., 2019). ReMixMatch (Berthelot et al.) 2020) and
UDA (Xie et al. (020)) reveal the crucial role of noise produced by advanced data augmentation methods. MCD is also a minimax model originally proposed for domain adaptation. However, it focuses enhancing invariance to model stochasticity and has a different network architecture with X-Model. Fix.Maach (Sohn et al. 2020) uses predictions from weakly-augmented images to supervise the output of strongly augmented data. Meta Pseudo Labels (Pham et al. 202T) further improves data efficiency by making the teacher constantly adapted by the feedback of the student's s performance the labeled dataset. SimCLRv2 (Chen et al.) (2020b) first fine-tunes the pre-trained model from the labeled data and then distills on the unlabeled data. Self-Tuning (Wang et al., 2021) introduces pseudo group contrast (PGC) mechanism but limited on classification setup. Besides of involving unlabeled data from the same distribution, another promising direction for improving data efficiency is introducing a complementary perspective to further improve data efficiency by introducing a related but different domain (Long et al. [001 Ganin & Lempitsky ; Long et al.
2017 Saito et al. 2018b Lee et al. 2019 Zhang et al. 2019] /aato et al. 201 Sa 2019). Moreover, various recent methods (van den Oord et al. 2018: He et al. 2020: Wu et al.
et al.)
2006; Tian et al.] 2019:; Chen et al. (020aa) improve data efficiency by self-supervised learning. However, most existing data-efficient methods focus classification setup while rare attention has been paid to deep regression.
Table 1: Comparison among various methods for improving data efficiency in deep learning,
stochasticity
setup
Method
data
model
classification
regression
Pseudo Label Lee
2013)
weak
Entropy (Grandvalet et & Bengio 2005
X
VAT (Miyato al. 2016)
weak
X
11-model (Lanne & Aila (2177)
weak
weak
Data Distillation (Radosavovic
2017
weak
Mean Teacher (Tarvaunen & Valpola
2017
weak weak
weak
MCD (Saito 20188)
strong
UDA (Xie et al. 2020)
strong x
FixMatch (Sohn et al. 2020
strong
Self- (Wang el al. 2021
strong x strong
X-Model (proposed)
strong
2.2 DATA-EFFICIENT REGRESSION
Data-efficient regression methods mainly fall into three categories; Co- To-TTrinnng,, kernel regression, and graph Laplacian regularization paradigm. COREG (Zhou & Li 2005b) adopt two regressors (kNNs) learned by different distance metrics and predict unlabeled data on one regressor and using the most promising predictions to train the other one. Transductive Regression (Cortes Mohri exploits local linear regression for unlabeled data, takes those with relatively close neighbors, and trains a kernel regressor to produce the final prediction. Graph Laplacian regularization is a commonly used manifold regularization technique (Belkin et 2006). It reasonable to assume that data points with close input values should have similar output values, thereby regularizing the model output with respect to the unlabeled data. Levati et al. (2017) and Srijith et al. (2013) adopt decision tree and Gaussian Process as regressors respectively. However, these existing data- data-efficient regression methods are mainly designed for shallow regression, which requires closed-form or convex-solvers to solve the problem, causing them not suitable for deep regression problems. The comparison of various methods for improving data efficiency in deep learning is shown in Table [
3

---

Published as a conference paper at ICLR 2022
0
db
y1
Φ
Confidence Thresholè C
ensemble
X2
22
{x)
Φ
(a) II-model
(b) Pscudo Labeling
(c) Data Distillation
Y/1
Φ
0 a0l 1-1 + (1 - a)0G
0
$ aΦ. + (1 a)
9
1/2
x2
82
(d) Mean Teacher
(e) MCD
(I) X X-Model
Figure 2: Comparisons among data-efficient learning techniques. (a) IImodel: uses a consistency
regularization term predictions of two different augmentations; (b) Pseudo-Labeling: leverages model itself to generate labels for unlabeled data and uses generated labels for training: (c) Data Distillation: extends the dual-copies of I1-model into multiple transformations; (d) Mean Teacher: maintains the teacher model with an exponential moving average of model weights of the student model and encourages consistency between them. (e) MCD: minimax model originally proposed for domain adaptation that focuses on enhancing invariance to model stochasticity. (f) X-Model: a minimax model for improving data-efficiency by enhancing model stochasticity and data stochasticity.
PRELIMINARIES
Denote labeled dataset L = { (zef, y(r) )im with nz samples (xrf v."). and an unlabeled dataset U = { ( with ny unlabeled samples. Usually, the size nt of C is much smaller than that no of lf and we define the the label ratio as n + +.)). Denote the feature generator network, and the successive task-specific head network. We aim at improving data efficiency deep learning by fully exploring the labeled and unlabeled data from the perspective of stochasticity.
3.1 INVARIANCE TO DATA STOCHASTICITY
Data-efficient methods with the insight of invariance to data stochasticity aim at making the pre- dictions of the model invariant to local input perturbations by consistency regularization term. As shown in Figure [(a) IT-model (Laine & Aila 2017; Sajjadi et al. 2016) first generates two examples with different stochastic data augmentations and then introduces loss term to minimize the distance between their predictions. With the strategy of invariance to data stochasticity, I-model believed augment the model with information about the intrinsic structure ("manifold") of U4 avoiding overfitting to the labeled data C. Realizing that training model on its own predictions (e.g.. II-model) often provides no meaningful information, Data Distillation (Radosavovic et 2017)
extends the dual-copies of II-model to multiple transformations as shown in Figure 2(c) It first generates an ensembled prediction for each unlabeled input from the predictions of a single model run on different transformations (e.g., flipping and scaling), and then uses it to guide the training of this input. In summary, the training loss of invariance data stochasticity can be formalized as
min Iatta (ar, (x, Ex.eu & ((ø o (angi (a;)), (ø (a.gg()))) 0,6
where ((...) is a proper loss function for the target task. For clarity, we focus on a particular data example here and the superseript U is omitted. Recently, instead of the weak augmentations (e.g., flip and crop) adopted in [--model, many strong augmentations (e.g., cutout and contrast) are adopted in FixMatch (Sohn et al. 2020) to further enhance invariance data stochasticity.
4

---

Published as a conference paper at ICLR 2022
X1
Aug. #1
Φ
91
X1
OO P1 O
J1
X2
Ø2
92
y
X2
Aug. #2
2
L de
a dha a 0Φb
basi-grepagation
Labeled Dataset
Unlabeled Dataset
Figure 3: The network architecture of X X-Model on the labeled and unlabeled dataset.
3.2 INVARIANCE TO MODEL STOCHASTICITY
Another strategy encourages invariance model stochasticity with difference penalty for predictions of models generated from different dropout (Laine & Aila 2017) or network initialization (Zhou Li. 2005b), as well as models exponentially averaged from history models. For example, Mean Teacher (Tarvainen & Valpola 2017) uses the prediction of an unlabeled sample from the teacher model to guide the learning of a student model as
min Tmmadel (2: (æ: Uf) = Eun, eur & ((dr o (x:),, ( (xx)) 0.6
where (d, o 02).
are the current model the exponential moving averaged model respectively. Note that of = a (1 a)or, =/ at%- + (1 a)); where is a smoothing coefficient hyperparameter. Similarly, the strategy of the invariance to model stochasticity tries to improve the quality of the teacher prediction to guide the training of the student model.
METHOD
Through the success of the above strategies, it is intuitive that the invariance stochasticity matters for improving data efficiency. To this end, we propose a novel X-Model with two new designs: 1) Simultaneously encourage the invariance data stochasticity and model stochasticity, detailed in Section 4.1 and 2) Enhance model stochasticity via a minimax model, detailed in Section 4.2
4.1 DATA STOCHASTICITY MEETS MODEL STOCHASTICITY
To take the power of both worlds, we propose novel X-Model by simultaneously encouraging the invariance to data stochasticity and model stochasticity for improving data efficiency in deep learning. Given an input æ,, we first transform it with two different data augmentations as auga (æx) and (x;) respectively. After that, they will pass the same feature extractor 0 and two different specific heads Φ1 and Φ2 to attain predictions ÿ., and ÿr.2 with both data stochasticity and model stochasticity
fr = (ø1 a ()nug (æx))
J..2 = (d2 (angz(a;))
where ④1 and d2 are two randomly initilized heads with different parameters. For each example a;; in the labeled dataset L {xt v. ) ;=1" focus partieular data example a!! and omit superscript L. When the invariance to data stochasticity meets with the invariance to model stochasticity, the supervised learning loss on the labeled data L can be formalized
Tog (z, L) = Em,ec &, (Di., y:) C, (Di.2 y:):
(4)
where (o(-.-) is a proper loss function for the target task, e.g. LI or L2 loss for regression tasks and cross-entropy loss for classification tasks. Similarly, a direct form of unsupervised loss is
La (1) = Ex, su Ey [[01 (augi (x::), (de 0) (x,))
(5)

---

Published as a conference paper at ICLR 2022
where Lu -1 is a proper loss function quantify the difference of these predictions, e.g. L1 or L22 loss for regression tasks and the symmetric Kullback- Leibler divergence for classification tasks. Different from the strategy that only applies data stochasticity model stochasticity, this strategy will greatly increase the invariance to stochasticity by unifying the benefit of both worlds.
Note that, instead of the weak augmentations (e-g., flip and crop) adopted in II-model, we utilize the strong augmentations (e.g., cutout and contrast) adopted in FixMatch (Sohn et al. (020) to enhance invariance to data stochasticity, as shown in Figure [ A natural question arises: Can we further enhance the invariance to model stochasticity similar to that of data stochasticity?
4.2 ENHANCE MODEL STOCHASTICITY VIA A MINIMAX MODEL
By minimizing La C) on the labeled data and (æ, U) on the unlabeled data using a proper trade-off hlpee-parameter 7), Ls + 7/Ln is an intuitive and effective way to simultaneously encourage the invariance to data stochasticity with model stochasticity. However, since the dropout or random initialization is kind of weak stochasticity, only posing minimization loss function between predictions of two sses--sccific heads may make them generate similar outputs, causing a degeneration problem and providing little meaningful information. To further enhance model stochasticity, propose a minimax game between the feature extractor and task-specific heads as
i arg min LL, C) m (x, 2),
(6)
(ōl 22) = arg min L, C) - - U),
01,02
In this minimax game, the inconsistency between the predictions of ΦJ and de on unlabeled data is maximized while the feature extractor is designed make their predictions as similar as possible. As shown in Figure 3 the blue lines denote the back-propagatin while the red ones denote the maximization ones. In the implementation, we can update all parameters simultaneously by multiplying the gradient of the feature extractor 0 with respect to the loss on unlabeled data by a certain negative constant during the backpropagation- based training.
of the minimization of loss function
By this minimax strategy, the task-specife heads are believed to capture meaningful information from complementary perspectives to avoid the degeneration problem. Providing more diverse information, the minimax model can further enhance the invariance to model stochasticity. Compared to the manually designed strategy of adding a dropout layer to models or just applying random initialization, this novel approach directly optimizes minimax loss function in the hypothesis space to fully explore the intrinsic structure of unlabeled data. In this way. X-Model further enhances invariance to stochasticity from two comprehensive perspectives: data and model.
The comparison between X-Model with other baselines is summarized in Figure 2 Figure 2( augı (a:) and aug2 (a::) can be regarded as the corners of "X" on the upper left and lower left respectively, and 6i and Φ2 can be denoted by the corners on the upper right and lower right respectively. Inspired by the name of II-model that has shape of "II", the proposed strategy with both data stochasticity and model stochasticity is more like "X", so we call it -Model.
As shown in
5 EXPERIMENTS ON REGRESSION TASKS
5.1 2D SYNTHETIC DATASET: dSprites-Scream
dSprites (Higgins et al., i017)]is is a standard 2D synthetic dataset consisting of three subsets each with 737, 280 images: Color, Noisy and Scream, from which we select the most difficilt one Scream as the dataset. Example images of Scream are shown in Figure 6(a) In each image, there are factors of variations each with definite value, detailed in Table 7 We adopt position X and position Y and scale as deep regression tasks while excluding the tasks of classification and orientation regression.
Since recently proposed methods such UDA, FixMatch, and Self-Tuning adopt pseudo-labele labels that cannot be exactly defined in deep regression, we did not report their results and these methods omitted in the following tables. As shown in Table 2 X-Model achieves the best performance across various label ratios from 1% to 50% over all baselmnes, evaluated on the commonly measure. It is noteworthy that -Mdel achieves MAE of 0.092 provided with only 5% labels,
MAE

---

Published as a conference paper at ICLR 2022
Table 2: MAE (1) on tasks of Position X, Position Y and Scale in dSprites-Screaum (ResNet-18)
Label Ratio
1%
20%
50%
Method
Sccaee
X
Y
All
Scale
Y
All
|caale
Y
All
Scale
X
All
Laheled Only
.130
.073
.075
.277
.072
.036
.035
.44
.051
.03
0028
.08
.046
.026 .025
.097
VAT Miyato el. 6
.67
.042
038
046
034
109
045
098
.037
084
TI-modar [Lanne Atln
2017
.844
.05
.35
154
058
031
025
.14
045
024
023
092
040
.01
001
.082
Data Distilaton (Kadosavova vovie et
2017
.066
.039
003
138
.45
.027
J01
104
.043
003
002
092
.037
.023
.021
.081
Mean Teacher (Tarvanen Valpola
2017
.062
.035
.037
.134
.045
.024
.033
103
.422
023
004
.059
.038
.021
.020
.079
UDA Xie al 2020)
(Splin 2020)
Self-Tuning ( Wang
2021
X-Moeel (wl minimax)
.880
.021
.024
.125
.044
.029
.028
101
.40
.077
0011
.077
.030
.027
.018
.074
X- Model (wlo data aug.)
0074
025
.023
.119
005
026
.02
093
037
.19
.017
073
.038
.018
.017
.074
X- Model (ours)
.61
030
024
.115
.04
.023
025
092
.037
.014
021
.072
.32
.018
018
.068
which is even lower than that of the laheled-only method (0.097) using 50% labels, indicating a large improvement of data efficiency (10x). Meanwhile, the ablation studies in Table 2 and other ablation studies, verify the effectiveness of both minimax strategy and data augmentation.
5.2 3D REALISTIC DATASET: MPI3D-Realisic
MPI3D (Gondal et al. 2019) is a simulation-to--ret dataset for 3D object. It has three subsets: Toy, Realistic and Real, in which each contains 033,, 800 images. Here, since have used the synthetic dataset above, we select the Realistic subset here to demonstrate the performance of X-Model. Example images of Realistic are shown in Figure 6(b) In each image, there are 7 factors of variations each with a definite value, detailed in Table g In MPPIID-Realiste there are two factors that can be employed for regression tasks: Horizontal Axis (a rotation about a vertical axis at the base) and Vertical Axis (a second rotation about horizontal axis). The goal of this task is to predict the value of the Horizontal Axis and the Vertical Axis for each image via less labeled data.
Table 3: MAE (1) on tasks of Horizomtal Axis and Vertical Axis in MPlsD-realistic (ResNet-18).
Label Ratio
1%
20%
Method
Hor.
Vert.
all
Hor.
Vert.
all
Hor.
Vert.
Hor.
Vert.
Labeled Only
.163
.112
.275
.075
.042
.117
.034
.030
.064
.023
.019
.042
2016 VAT (Miyato et
.102
.087
.189
.048
.109
H-moder [ainn & Aila 2017
.099
.086
.185
.051
.041
.092
.031
.024
.055
.023
.017
.040
Data Distilatoon Kolo et al
2017
.095
.092
.187
.053
.046
.099
.029
.024
.053
.023
.021
.044
Mean Teacher Tarvainen Valpola
2017
.091
.81
.172
.045
.036
.081
028
.059
.044
.018
.042
X-Model (ours)
.087
.068
.155
.036
.060
.042
Following the previous setup, we evaluate X-Model various label ratios including 1%, 5%. 20% and 50%. As shown in Table 3, X X-Model beats the strong competitors of data stochasticity and model stochasticity-basee some data-efficient methods underperform than the vanilla label-only model (e.g. Data Distillation (0.044) and VAT (0.046) achieves a higher MAE than label-only method (0.042) when provided with a label ratio of 50%), while our MModel always benefits from exploring the unlabeled data.
methods by large margin when evaluated the MAE measure. Note that,
5.3 AGE ESTIMATION DATASET: IMDB-WIK
IMDB- IMDB-WIKI ( Rasmuus Rothe 2016) is a face dataset with age and gender labels, which can be used for age estimation. It contains 223.0K face images and the corresponding ages. After splitting, there are 191.5K images for training and 11.0K images for validation and testing. For data-efficient deep regression, we construct 5 subsets with different label ratios including 1%, 5%, 10%, 20% and 50%. For evaluation metrics, we use common metrics for regression, such as the mean-average (MAE) and another metric called emror Geometric Mean (GM) (Yang al. 2021) which is defined as for better prediction fairness, where e, is the prediction error of each sample é.
(II )
As reported in Table no matter evaluated by MAE or GM, Model achieves the best performance across various label ratios from 1% 50% over all baselines. It is noteworthy that the value of the
7

---

Published as a conference paper at ICLR 2022
Table 4: MAE (4) and GM (4) on task of age estimation in IMDB- -IIKI (ResNet-50).
Label Ratio
5%
10%
20%
50%
Method
MAE
GM
GM
MAE
Labeled Only
17.72
12.69
14.79
9.93
13.70
9.05
11.38
7.16
9.53
VAT (Miyato al. 2016
13.02
8.55
12.09
7.73
11.06
6.74
10.43
6.49
8.38
4.88
momole [I [Lane Alla
7017
12.99
8.65
11.92
7.69
10.90
6.61
10.12
6.42
8.36
4.80
Co-Trining //hno
2005h
13,46
8.74
12,63
7.93
11,28
6.78
10.78
6.58
8.61
4.99
Data Distillationn Rabissvovic vt al 2017
13.12
8.62
12.07
7.71
11.02
6.71
10,37
6.43
8.33
4,84
Tarvamen Valpola
X-Model (ours)
12.75
8.32
11.55
7.33
10.57
6.29
9.89
3.98
8.06
4.59
unlabeled data is highlighted in this dataset since all baselines that try explore the intrinsic structure of unlabeled data work much better than the vanilla one: labeled-only method.
5.4 KEYPOINT LOCALIZATION DATASET: Hand-3D-Studio
Bond-3D-Studiie (H3D) (Zhao et al. 2020) is a real-world dataset containing 22 frames in total. These frames are sampled from videos consisting hand images 10 persons with different genders and skin colors. We use the Percentage of Correct Keypoints (PCK) as the measure of evaluation. A keypoint prediction is considered correct the distance between with the ground truth less than fraction 0.05 of the image size. PCK @0.05 means the percentage of keypoints that can be considered as correct over all keypoints.
As shown in Table 5 most baselines that explore unlabeled data show competitive performance and proposed X-Model still consistently improves over these baselines. For example, when provided with 1% labels, X- X-Model achieves average PCK of 73.6, which has absolute improvement of 3.4 over the strongest baseline.
Table 5: PCK@0.05 (f) on task of keypoint localization in Hand-3D-Stuile (ResNet-101).
Label Ratio
1%
5%
Method
MCP
PIP
DIP
Fingertip
Avg
MCP
PIP
DIP
Fingertip
Avg
Labeled Only Xiso
2018
72.3
70.3
66.3
62.8
68.5
90.5
86.7
82.0
76.3
84.4
VAT (Miyato et ml 2015
76.2
71.2
66.0
61.6
69.6
91.2
87.9
83.1
78.5
85.6
I--mder [Lanne
2017
76.1
70.3
65.6
69.2
90.1
86.4
82.0
76.8
84.3
Data Distilation Radorvovig al 2017
75.8
70.1
65.8
62.0
69.3
90.9
88.9
84.7
79.6
86.3
(arvamen Valpola
X X-Moeel (wle minimax)
79.2
74.6
69.2
72.3
91.3
88.5
83.5
78.7
85.9
X-Madel (wlo data mug.)
79.4
75.2
70.0
72.9
91.8
89.0
84.5
79.6
86.6
-Model (ours)
79.8
75.6
70.7
65.5
73.6
92.3
89.4
85.0
80.8
87.2
6 EXPERIMENTS ON CLASSIFICATION TASKS
We adopt the most difficult CIFAR-100 dataset (Krizhevsky 2009 with 100 categories among the famous data-efficient classification benchmarks including CIFAR-IOM CIFAR-JO, SVHN, and S7Z.-0, where the last three datasets have only 10 categories. As shown in Figure X-Model not only performs well on deep regression tasks but also outperforms other baselines on classification tasks. Note that, MCD with strong data augmentation still underperforms X-Model, since the dual branch architecture of MMdde enables itself to further enhance data stochasticity as compared in Figure
7 INSIGHT ANALYSIS
An Analysis Two-Moon We the classical Two-Moon dataset with scikit-learn (Pedregosal et al. 2011) to visualize decision boundaries various data-efficient learning methods on a binary classificatin task with only 5 labels per class (denoted by "+"). As shown in Figure from the
8

---

Published as a conference paper at ICLR 2022
Table Error rates (%) of classification on CIFAR-IO0 (WRN-28-8).
Method
400 labels
2500 labels
10000 labels
Pscudo-Labeling (Lee
2013)
57.38 10.46
36.21100.1
MC Dropout (aal & Ghahramani 2016)
58.27:10.44
38.36110.19
Deep Co- ( (Qao
2018)
53.38=0061
34.63-+0.14
11-Model Laine Aula 2017
57.25
37.88:+0.11
MME Saito et 20119) 2019)
1.75
32.54-10.81
Mcan Teacher (Tarvainon & Valpola
2017
+0.57
35.83-40:24
MCD (weak aug.) (Saito et al.
201 8b)
36. 17.333
28.47:+0.38
MCD (strong aug, (Saito et al.
(018b
29.59
23.10++0:33
MixMatch (Bertheloi et al. 2019
6..32
39.94
2:.31
UDA (Xie el al. 2020)
59.28
33.13
24.50.10.25
ReMixMatch (Berthelot et al. 2020
44.2811.00
27.43
23,03
FixMatch (Sohn et al. 20200
48. 1.78
28. =0.11
22.60++0.12
Meta Pseudo Labels (Pham et al.
2021
48.18+1.29
27.31
22.02++0.18
Self-Tuning (Wang et al. 2021)
54.7440.35
42.08
21.75++0.7
X-Model
47.21+1554
27.11
20.98+0.33
(a) Label Only
(b) Minimize Entropy
(c) 11-Model
(d) X-Model
Figure 4: Decision boundaries of various cata-effficient learning methods on Two- Two-Moon.
(a) Label Only
(b) II-model
(c) X-Model
(d) Ground Truth
Figure 5: 1-SNE features dSprites-Scream. (Red: labeled samples; Blue: unlabeled samples).
label-only method to X-Model, the decision boundary is more and more discriminable and a more accurate hyperplane between two classes is learned.
Feature Visualization via t-SNE We use the r-SNE tool (Maaten Hinton 2008) visualize the features of several models on regression task named dSprites-Scream with 5% labels. As shown in F5 from the label-only method X- MModel. the intrinsic structure ("manifold") of the unlabeled data is more discriminable and much more similar to that of the fully-supervised ground-truth one.
CONCLUSION
We propose the -Modee, which unifies the invariance to data stochasticity and model stochasticity to take the power of both worlds with form of "X". We make the X X-Model play minimax game between the feature extractor and spec-fc heads to further enhance invariance to stochasticity. Extensive experiments demonstrate the superior performance of the -Model among various tasks.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This work was supported by the National Megaproject for New Generation AI (2020AAA0109201), National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (Z201 100006820041), BNRist Innovation Fund (BNR202IRCO10) Fund. This work was done when Ximei was also an intern at Kuaishou Technology and supervised by Guoxin Zhang and Pengfei Wan.
and Kuaishou Technology
REFERENCES
Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled examples. JMLR, 2006.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurlPS, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. ICLR, 2020.
Avrim Blum and Tom M. Mitchell. Combining labeled and unlabeled data with co-training. In COLT, 1998.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. arXiv preprint arXiv:2002.05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. NeurIPS, 2020b.
Corinna Cortes and Mehryar Mohri. On transductive regression. In NeurIPS, 2007.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model
uncertainty in deep learning. In ICML, 2016.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
MW Gondal, M Wuthrich, D Miladinovic, F Locatello, M Breidt, V Volchkov, J Akpo, O Bachem, B Schölkopf, and S Bauer. On the transfer of inductive biasfrom simulation to the real world: new disentanglement dataset. In NeurlPS, 2019.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurlPS, 2005.
R. Hadsell, S. Chopra, and Y. Lecun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Alexander B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving, Christoph Reinders, Sarthak Yadav, Joy Banerjee, Gábor Vecsei, Adam Kraft, Zheng Rui, Jirka Borovec, Christian Vallentin, Semen Zhydenko, Kilian Pfeiffer, Ben Cook, Ismacl Fernández, François-Micher De Rainville, Chi-Hung Weng. Abner Ayala-Acevedo, Raphael Meudec, Matias Laporte, et al. cww/alejm..eeeoeeooeeoo jiggguuuu//mg/////// 2020.
imgaug. httpps://
10

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Samuli Laine and Timo Aila, Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Donghyun Lee. Pseudo-label: The simple and efficient cemi-supervised learning method for deep neural networks. In ICML Workshop Challenges in Representation Learning, 2013.
Seungmin Lee, Dongwan Kim, Namil Kim, and Seong- Jeong. Drop to adapt: Learning discriminative features for unsupervised domain adaptation. In ICCV, 2019.
Jurica Levati, Michelangelo Ceci, Dragi Kocev, and Sao Deroski. Self-training for multi-target regression with tree ensembles.
Syst. 2017.
M. Long, Y. Cao, J. Wang, and M. I. Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In ICML., 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(Nov):2579-2605. 2008.
Takeru Miyato, Shin ichi Maeda, Masanori Koyama, Ken Nakac, and Shin Ishii. Distributional smoothing with virtual adversarial training. In ICLR, 2016.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. JMLR, 2011.
Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V. Le. Meta pseudo labels. CVPR, 2021.
Siyuan Qiao, Wei Shen, Zhishuai Zhang. Bo Wang, and Alan L. Yuille. Deep co-training for semi-supervised image recognition. ECCV, 2018.
Ilija Radosavovic, Piotr Dollár, Ross B. Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards oupi-supervised learning. CVPR, 2017.
Lue Van Gool Rasmus Rothe, Radu Timofte. Deep expectation of real and apparent age from a single image without facial landmarks. In IJCV, 2016.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regulariza- tion. In ICLR, 2018
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018b.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In ICCV, 2019.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi remi-supervised learning. In NeurIPS, 2016.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS, 2020.
P. Srijith, Shirish Shevade, and Sundararajan Sellamanickam. Semi-supervised gaussian process ordinal regression. In CoRR, 2013.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NewrIPS, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 1906:99060000 05849, 2019.
11

---

Published as a conference paper at ICLR 2022
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018.
Ximei Wang, Jinghan Gao, Mingsheng Long, and Jianmin Wang. Self-tuning for data-efficient deep learning. In ICML, 2021.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In ICLR, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non con-parametric instance discrimination. In CVPR, 2018.
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. ECCV, 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh- Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. NeurlPS, 2020.
Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In ICML, 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurlPS, 2014.
Zhi-Hua Zhou Yu- Yu-Feng Li, Han-Wen Zha. Learning safe prediction for semi-supervised regression. In AAAI, 2017.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML, 2019.
Zhengyi Zhao, Tianyao Wang, Siyu Xia, and Yangang Wang. Hand-3d-studio: A new multi-view system for 3d hand reconstruction. In ICASSP, 2020.
Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. TKDE, 2005a.
Zhi-Hua Zhou and Ming Li. Semi-supervised regression with co-training, In IJCAI, 2005b.
12

---

Published as a conference paper at ICLR 2022
A EXPERIMENT DETAILS
We empirically evaluate X- X-Model in several dimensions:
Task Variety: four regression tasks with various dataset scales including single-value prediction task of age estimation, to dense-value prediction ones of keypoint localization, as well as a 2D synthetic dataset and a 3D realistic dataset. We further include multi-category object recognition task verify the effectiveness of X-Model on classfication setup.
Label Proportion: the proportion of labeled dataset ranging from 1% to 50% following the popular protocol of data-efficient deep learning.
Pre-trained Models: mainstream pre-trained models are adopted including ResNet- 18, ResNet-50 and ResNet-101 (He et al. 2016).
Baselines We compared X- X-Model against three types of baselines: (1) Data Stochasticity: I--
model (Laine & Aila (017). VAT (Miyato et al.) (016). and Data Distillation (Radosavovic et al.
2017); (2) Model Stochasticity: Mcan Teacher (Tarvainen & Valpolal 2017); (3) Label-Only
Models: task- task-specific model trained on only the labeled data is provided. For example, Simple Baseline (Xiao et al. 2018) with a ResNet101 pre-trained backbone trained on only the labeled data
is used strong baseline for keypoint localization.
Implementation Details We use PyTorch with Titan V to implement our methods. Models pre- trained on ImageNet such as ResNet-18, ResNet-50, ResNet-101 (He et al. (016) are used for the backbones of experiments dSprites-Scream, IMDB- WIKI, and Hand-3D- Studio respectively.
On dSprites-Scream, there are 5 factors of variations each with a definite value in each image. Among these factors, there four factors that can be employed for regression tasks: scale, orientation, position and position Y. Therefore, we adopt position X and position Y and scale as deep regression tasks while excluding the task of orientation regression. We treat all tasks equally and address there tasks as a multi-task learning problem with a shared backbone and different heads. Labels are all normalized to [0, 1] to eliminate the effects of diverse scale in regression values, since the activation of the regressor is sigmoid whose value range is also [0, 1].
On Realisti,, there are two factors that can be employed for regression tasks: Horizontal Axis (a rotation about vertical axis at the base) and Vertical Axis (a second rotation about horizontal axis), since each object mounted on the tip of the manipulator and the manipulator has two degrees of freedom. The goal of this data-efficient deep regression task is to train model to accurately predict the value of the Horizontal Axis and the Vertical Axis for each image via less labeled data.
On IMDB-WII following the data pre-process method of a recent work (Yang et al., 2021), we also filter out unqualified images, and manually construct balanced validation and test set over the supported ages. After splitting, there are 191.5K images for training and 11.0K images for validation and testing. For data-efficient deep regression, we construct 5 subsets with different label ratios including 1%, 5%. 10%, 20% and 50%. For evaluation metrics, we use common metries for regression, such as the meaa-averaage-enen (MAF) and another metric called error Geometric Mean (GM) (Yang et al. 2021) I which is defined as 1e)) A for better prediction fairness, where is the prediction error of each sample
On Slund-DD-tudiion we first randomly split the dataset into a testing set with 3.2k frames and a training set containing the remaining part. Further, we follow the setup of data-efficient learning and construct subsets with label ratios of 1% and 5%. We use the Percentage of Correct Keypoints (PCK) as the measure of evaluation. A keypoint prediction is considered correct if the distance between it with the ground truth is less than fraction a 0.05 of the image size. PCK @0.05 means the percentage keypoints that can be considered as correct over all keypoints.
All images are resized to 224 x 224. The tradeoff hyperparameter 1) is set 0.1 for all tasks unless specified. The learning rates of the heads are set as 10 times to those of the backbone layers, following the common fine-tuning principle (Yosinski et al. 2014). We adopt the mini-batch SGD with momentum of 0.95. Code will be made available at https://gittub..co
http://pytorch.org
13

---

Published as a conference paper at ICLR 2022
B EXAMPLE IMAGES AND FACTORS OF DSPRITES AND MPI3D
(a) Example images of dSprites-Scream
(b) Example images of MPI3D-Realistie
Table Factors of variations in dSpirtex
Table 8: Factors of variations MP13D
Factor
Possible Values
Task
Factor
Possible Values
Task
Shape
square, ellipse, heart recognition
Object Size
small=0, large=1
recognition
Scale
6 values in [0.5, 1]
regression
Camera Height
top=0, center=1, bottom=2 purple=0, green= salmon=2
recognition
Orientation
40 values in [0, 2
regression
B.G. Color
recognition
Position
values in [0,
regression
Horizontal Axis
40 values in [0. 1
regression
Position
32 values in [0,
regression
Vertical Axis
40 values in [0,
regression
C FULL RESULTS OF IMDB-WIKI
IMDB-WIKI |Rasmus Rothe 2016) is face dataset with age and gender labels, which can be used for age estimation. It contains 523.0K face images and the corresponding ages. Following the data pre-process method of recent work (Yang et al. 2021), and manually construct balanced validation and test set over the supported ages. After splitting, there 191.5K images for training and 11.0K images for validation and testing. For data-efficient deep regression, we construct 5 subsets with different label ratios including 1%, 5%, 10%, 20% and 50%. For evaluation metrics, we use common metries regression, such as the mean-average-erron (MAE) and another metric called error Geometric Mean (GM) (Yang al.) 2021) which is defined as (Mm)ee) )
also filter unqualified images,
for better prediction fairness, where e, is the prediction error of each sample i.
As reported in Table 9 no matter evaluated by MAE or GM, X- XModel achieves the best performance across various label ratios from 1% 50% over all baselines. It is noteworthy that the value of the unlabeled data highlighted this dataset since all baselines that try to explore the intrinsic structure of unlabeled data works much better than the vanilla one: labeled-only method.
Table 9: MAE (L) and GM (4) task of age estimation in IMDB- WIKI (ResNet-50).
Label Ratio
1%
5%
10%
20%
50%
Method
MAF
GM
MAE
GM
MAE
GM
MAE
GM
MAE
GM
Labeled
VAT (Miyato et al
2016
13.02
8.55
12.09
7.73
11.06
6.74
10.43
6.49
8.38
4,88
I-moar Lanne Alb
12.99
11.92
7.69
10.90
10.12
8.36
Data Distilation Radesivovic 2017
13.12
8.62
12.07
7.71
11,02
6.71
10.37
6.43
8.33
4.84
Mean Teacher Tarvamen & Valpola 2017
12.94
8.45
12.01
7.64
10.91
6.62
10.29
6.35
8.31
4,78
Model (wlo minimax)
12.87
8,51
11.91
7.60
10.73
6.55
10.00
6.06
8.22
4.59
Model (win ang.)
13.02
11.71
6.86
8.17
Model (ours)
12.75
8,32
11.55
7.33
10.57
6.29
9.89
5.98
8,06
4.59
wi http://///.c.......ccccce/e///er
14


Reviews:
[{"review_id": "kXId6Dfkbm", "reviewer": "Reviewer_6Dgh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n* The paper is meticulously written, well structured, and tackles the really interesting and challenging problem learning from\n scarcely labeled data.\n* The paper has good flow, it builds from the existing literature, and first points out the drawbacks of the existing methods,\n which then motivates the proposed X-model to tackle those issues.\n* Extensive experiments on both classification and regression tasks show the ability of the proposed method’s ability to exploit\n strong data and model stochasticity, resulting in enhanced performance. It is shown that the proposed method is able to outperform all the methods compared in the paper.\n\nConcerns\n\n* One of the major concerns is regarding the technical aspect of the proposed approach. It seems to be very similar to another\n method proposed for semi-supervised domain adaptation [1], which is also proposed to improve the data efficiency of the model. While they don’t explore a task beyond classification and the problem beyond domain adaptation, the overall approach seems to be\n very similar. Furthermore, the overall idea to enhance model stochasticity has a lot of similarities with [1], [2], [3], [4] which are not mentioned in the paper at all. Hence, it is essential to have the method proposed in [1] for comparison and needs to have a discussion on the differences between the proposed method and [1]. Another experiment to consider would be Mean-Teacher + [1] which would be almost the same as the proposed idea.\n* The paper does provide an analysis with the UDA method, not sure why semi-supervised DA methods are not considered as it is\n more similar to the proposed problem setup than UDA, having such comparison would strengthen the claims provided in the experimental section. \n\n[1] Saito, Kuniaki, et al. \"Semi-supervised domain adaptation via minimax entropy.\" Proceedings of the IEEE/CVF\n International Conference on Computer Vision. 2019.\n\n[2] Saito, Kuniaki, et al. \"Adversarial Dropout Regularization.\" International Conference on Learning Representations.\n 2018.\n\n[3] Saito, Kuniaki, et al. \"Maximum classifier discrepancy for unsupervised domain adaptation.\" Proceedings\n of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[4] Lee, Seungmin, et al. \"Drop to adapt: Learning discriminative features for unsupervised domain adaptation.\"\n Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\n\n", "summary_of_the_paper": "\nThe paper focuses on reducing data labeling efforts by improving data efficiency. In contrast to most existing approaches that address this problem only in the classification setup, the paper focuses on both classification and regression set ups. The proposed method primarily is built on leveraging invariance to data stochasticity and model stochasticity. Experiments are conducted on various tasks like age estimation, key point localization, and object recognition. ", "summary_of_the_review": "My concerns are primarily due to similarities to some of the works in semi-supervised DA techniques. Considering that the paper addresses the problem in a more general setting of classification and regression, I am leaning towards accept for now. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OmrLIMJRUrE", "reviewer": "Reviewer_3Tfv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n*  Data-efficient regression is a very interesting direction. Most previous works on data-efficient methods focus only on classification setups. For example, pseudo-labels, which is proven to be one of the most effective methods for semi-supervised classification, cannot be exactly defined in a deep regression problem. This paper sheds some light on this new direction.\n* The proposed method seems to be simple but effective. \n\nWeaknesses:\n* One missing related work on pseudo labels [1]. It would be great to include some discussion or even add as a baseline in the classification experiments.\n* It is unclear whether the proposed method can generalize well to more challenging tasks (for example, depth estimation or larger dataset). Though, I'm satisfied with the current evaluation (from a single-value toy dataset to dense-value prediction datasets).\n\n[1] Pham, Hieu, et al. \"Meta pseudo labels.\" CVPR 2021.\n", "summary_of_the_paper": "The paper presents a g data-efficient approach that encourages invariance to both data and model stochasticity that works for both classification and regression tasks. Furthermore, the proposed minimax loss function can specifically enhance invariance to model stochasticity. The extensive experimental results verify that the proposed method is effective.", "summary_of_the_review": "In general, the proposed method is well-motivated with sufficient experimental comparisons. Most importantly, the paper seems to be the first step toward data-efficient regression leveraging strong stochasticities from both data and model. To this end, I believe this paper has enough merit to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-n8fQLJeHhZ", "reviewer": "Reviewer_DfEj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points\nGood performance on a wide range of benchmark datasets.\nEasy to read, experiments clearly described.\nApproach is fit for regression as well.\n\n\n## Weak points\nTwo heads which i assume do not share weights are similar to an ensemble and can hence be compared with ensembling based approaches. Even more if comparing with a MC Dropout based ensemble, where the dropout might be applied in the final layers only, hence effectively creating an ensemble with a backbone with shared weights. [4]\nTo my understanding the proposed approach is the same as if applying random data augmentations and passing through a DNN with dropout sampling twice. Training one model on the label provided by another model is known as co-training [1] and has been applied for Semi-supervised learning (SSL) also recently [2] e.g. on CIFAR100 results, especially w/o data augmentation would aid in judging the contribution of individual components of the proposed approach.\nThere is a huge amount of SSL literature that would be needed to be taken into account, on guide could be [3] - please check for more recent ones as well.\n\n\n## Questions\nDo both heads of the chi-model have the same architecture and do not share weights?\nAre the heads only one-layer or multi-layered architectures?\n\n\n## Minor comments\nEvaluations w/ and w/o data augmentation especially when comparing with other methods would be helpful. At least stating which methods use the same data augmentations.\n\n[1] Blum, Avrim, and Tom Mitchell. \"Combining labeled and unlabeled data with co-training.\" Proceedings of the eleventh annual conference on Computational learning theory. 1998.  \n[2] Qiao, Siyuan, et al. \"Deep co-training for semi-supervised image recognition.\" Proceedings of the european conference on computer vision (eccv). 2018.  \n[3] Zhu, Xiaojin Jerry. \"Semi-supervised learning literature survey.\" (2005).  \n[4] Gal, Yarin, and Zoubin Ghahramani. \"Dropout as a bayesian approximation: Representing model uncertainty in deep learning.\" international conference on machine learning. PMLR, 2016.", "summary_of_the_paper": "This paper proposes the so called Chi-model, which combines data augmentation with a two-headed network architecture to include model stochasticity. It specifically addresses data efficient learning in regression settings.", "summary_of_the_review": "Approach seems promising, especially the empirical results. I am lacking a bit of information to set this into relation to previous works as outlined above. Without these details it is hard to judge how much can be attributed to the proposed approach. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The SAC wrote a very good meta review and I just copy and paste it here. I completely agree with the SAC that the contribution of the paper due to the similarity to MME and MCD. Hopefully adding data augmentation to MCD and providing empirical results on new tasks can shed some lights to the community.

--------------------
Based on a request from the authors, the SAC read the paper and the reviews and engaged two additional expert reviewers to provide an additional assessment.

The paper addresses semi-supervised learning (SSL) and makes two contributions: 1) a method called χ-model, which combines data augmentation with a two-headed network that has an adversarial loss between the heads and the feature backbone; 2) an empirical evaluation on classification and regression SSL tasks.

As pointed out by reviewer 6Dgh, the proposed model is very similar to two existing methods: MME and MCD. These related works were not mentioned in the submission. During the rebuttal the authors compared to MME but not MCD.

Similarity to MME: The core idea in MME (Min-Max Entropy) is to introduce a min-max game between the head and backbone to regularize the model. The authors point out that one difference is that X-model uses two heads and regression loss, while MME uses a single head and entropy loss. They also have other components like data consistency regularization, which are borrowed from prior work (FixMatch). They further point out that MME was evaluated and motivated for unsupervised domain adaptation whereas their paper evaluates on the same distribution (regular SSL). In general, there is a lot of overlap and borrowed techniques between UDA and SSL methods, but the experimental benchmarks tend to be different.

Similarity to MCD: A larger concern is that the proposed method is actually more similar to MCD than MME. In Eq.5, the data is augmented into two ways and two different output heads are applied. If the data is not augmented this way, the formulation is the same as MCD. The authors did not discuss this point, even though reviewers pointed out the similarity. From the perspective of the technical approach, the novelty wrt MCD appears to be mainly in adding the data augmentation. They do show in the paper that this type of min-max regularization is effective in various regression tasks, which is a good empirical contribution. In the ablation studies of the data augmentation part, the difference over MCD is not very large, but the augmentation provides good gains in some experiments (although MCD is not mentioned, in Table 2 and 5, the "χ-model (w/o data aug.)" is likely essentially MCD.)

Overall, the strength of the paper appears to be in adding data augmentation to MCD and providing empirical results on new tasks showing that it works on same-distribution test sets and on regression tasks. The technical contribution seems somewhat limited, if accepted, the paper should include a clear discussion w.r.t MCD in the method and experiment sections. Furthermore, some baselines may potentially be missing, e.g. an MDD-based (similar to MCD) UDA method for regression can be used as a baseline (Regressive Domain Adaptation for Unsupervised Keypoint Detection, CVPR'21).
Decision: Accept (Poster)