Self-Joint Supervised Learning
Authors: Navid Kardan
Mubarak Shah
Mitch Hill
Venue: ICLR 2022 Poster (2022)

Abstract: Supervised learning is a fundamental framework used to train machine learning systems. A supervised learning problem is often formulated using an i.i.d. assumption that restricts model attention to a single relevant signal at a time when predicting. This contrasts with the human ability to actively use related samples as reference when making decisions. We hypothesize that the restriction to a single signal for each prediction in the standard i.i.d. framework contributes to well-known drawbacks of supervised learning: making overconfident predictions and vulnerability to overfitting, adversarial attacks, and out-of-distribution data. To address these limitations, we propose a new supervised learning paradigm called self-joint learning that generalizes the standard approach by modeling the joint conditional distribution of two observed samples, where each sample is an image and its label. Rather than assuming samples are independent, our models explicitly learn the sample-to-sample relation of conditional independence. Our framework can naturally incorporate auxiliary unlabeled data to further improve the performance. Experiments on benchmark image datasets show our method offers significant improvement over standard supervised learning in terms of accuracy, robustness against adversarial attacks, out-of-distribution detection, and overconfidence mitigation.

Published as a conference paper at ICLR 2022
SELF-JOINT SUPERVISED LEARNING
Navid Kardan Mitchell Hill* & Mubarak Shah
"Center for Research Computer Vision 'Department of Computer Science
'Department of Statistics and Data Science
University of Central Florida
I '[Kardan, Shah (knightt crev] .revvjucf... .ucf.edu Mi MMtthhel. Hi miilluucf... euu
ABSTRACT
Supervised learning is fundamental framework used to train machine learning systems. A supervised learning problem is often formulated using an i.id. as- sumption that restricts model attention single relevant signal time when predicting. This contrasts with the human ability to actively use related samples as reference when making decisions. We hypothesize that the restriction single signal for each prediction in the standard i.i.d. framework contributes to well- known drawbacks of supervised learning: making overconfident predictions and vulnerability to overfitting, adversarial attacks, and out-of of-distribution data, To address these limitations, we propose a new supervised learning paradigm called self-joint learning that generalizes the standard approach by modeling the joint conditional distribution of two observed samples, where each sample is an im- models age and its label. Rather than assuming samples are independent, explicitly learn the sample-to-sample relation of conditional independence. Our framework naturally incorporate auxiliary unlabeled data to further improve the performance. Experiments on benchmark image datasets show our method offers significant improvement over standard supervised learning in terms of ac- curacy, robustness against adversarial attacks, out-of out-of-distribution detection, and overconfidence mitigation. Code: ggthab.com/nddn///ffoott---aaaaa
INTRODUCTION
Neural networks lay at the heart of deep learning success. However, problems such as overfitting (Weigend) 1994; Hawkins] 2004) and confidence miscalibration (Guo et al. (017) persist across the vast majority of supervised models. Model calibration requires the predictive confidence of classifier to be faithful to the ground truth, meaning the expected accuracy of samples with m% confidence should be almost m%. However, deep models tend to produce overconfident predic- tions (Guo et al.] 2017). which undermines utilization their confidence values. Consequences of miscalibration such susceptibility adversarial attacks (Szegedy et al. 2013 Goodfellow al.) 2014 Madry et al.] 2017 Akhhar & Mian 2018: Akhhar et al. 2021 Edraki et al. 2022) and poor performance on oft-s--distribution (OOD) examples (Liang al. 2017 Hendrycks al. 2018)
limit the utilization of deep learning systems in real-world applications.
Existing tools combat overfitting and miscalibration include artificially increasing training data through data augmentation (Shorten & Khoshgoftaar 2019), using loss penalties, utilizing stochas- ticity constrain a model's ability learn data idiosyncrasies (Larsen Hansen 1994; Wan et al. 2013 Gal & Ghahramani] 2016). and smoothing labels to moderate overconfident predic- tions SZzgedy et al.] [016)] Müller al., 2019). While increasing the number of samples during training using data augmentation has been explored, unexplored direction is to increase the num- ber of samples used when making a prediction. Even a well-trained model could benefit from having additional data available at the time of inference, similar to human expert that can still benefit from having a selection of reference examples when making decision about a single test example. We hypothesize the problems of overfitting and miscalibration could be further exacerbated by limita- tions of the commonly used model p(Y [X;
; 8) for a single sample-label pair (X, Y) and parameter 8.
Corresponding author

---

Published as a conference paper at ICLR 2022
This model form only allows other training pairs (X', Y") to provide reference information for the conditional probability [X; 8), and therefore the model prediction for X, implicitly via training and not at all during inference.
In this paper, we propose a new paradigm called self-joint supervision to construct models that make predictions using given test signal and randomly selected sample of reference signals (typically a subset of training or validation pairs). This work focuses on image signals but the framework applies broadly. During training, self-joint model learns to predict the joint distribution of pair of labels given their associated images. By training with a pair instead of one sample, the model explic- itly learns sample-to- -oappee relationships. In this work. we retain the standard i.i.d. assumption of supervised learning and our models actively learn the relationship of pairwise conditional indepen- dence (see Appendix A. for brief review of these concepts). Learning conditional independence is the key innovation of our work and the main difference from standard supervised learning, which has no explicit cross-sample modeling.
We apply Maximum Likelihood to learn self-joint models in the similar framework standard model. After training, the model infers individual labels by marginalization of the joint distribution output. We propose inference procedures to produce both a stochastic and a deterministic output. Figure I shows an intuitive visualization of the self-joint training and inference procedures. Ex- periments with stochastic self-joint models provide evidence that learning conditional independence among sample pairs can lead to models more resistant to adversarial attacks and OOD data. More- over, method can naturally be extended to incorporate auxiliary unannotated data during training by using marginal prediction to generate pseudo-labels. Our experiments suggest adding partially annotated data can further safeguard against OOD data.
Overall, our findings provide a novel approach to train more consistent and robust deep models and an alternative strategy to utilize auxiliary data. The main contributions of our work are summarized below.
ground truth
Y
marginalize
X
dog
frog
Bog
Trog
self-joint NN
Joss
input pair
A) Training self-joint model
B) Deterministic prediction
test image
average
marginalize
C) Stochastic prediction
ref. images
Figure 1: Self-joint paradigm effectively expands a c-class classification problem into cl-class classification problem. As a concrete example, assume we are facing ternary cat vs. dog vs. frog classification problem. Self-joint framework principally casts this into new 9-way classification problem [[eat, cat), (cat, dog). ..., (frog, dog), (frog, frog)), which represents all possible pairs of three labels for two inputs (in our implementation by the x matrix above. During inference, one can extract output probabilities for each input separately through marginalization, i.e. summing up the rows or columns of the matrix. Intuitively, the marginal prediction for a given test sample changes when we replace the second sample in the pair, this indicates the marginal predictions have weak confidence. However, if the model is consis- tent with its prediction across paired samples, the overall marginal prediction has high confidence. This property leads increased robustness.
concatenate inputs channel-wise as shown
(1) We introduce novel perspective of self-joimt learning where the model can accommodate sample dependency. By applying the model to pairs of i.i.d. data (pairs are i..dd), we learn rather than assume conditional independence between labels given two input images. (2) We show self-joint
2

---

Published as a conference paper at ICLR 2022
learning leads to substantial improvement in classification accuracy the network capacity scales to sizes that were previously found to be detrimental due to overfitting. (3) We construct a new type of stochastic models that can provide robust predictions for adversarial examples and detect OOD data. In particular, we explore the impact of our approach on model robustness against adversarial attacks and show it drastically improves adversarial robustness against small perturbations. (4) We propose a novel technique to inject priors into learning models with the help of auxiliary unlabelled data. Our experiments show that utilizing auxiliary data can improve the model's ability to recognize OOD data.
2 RELATED WORK
Dut-of-Distribution (OOD) Detection. OOD data play a vital role in real-world applications, where the model commonly encounters samples from novel classes. Ideally, the model should produce low confidence predictions with high level of uncertainty for data from unknown classes. However, several studies suggest that deep models frequently make confident predictions for OOD data (Kardan & Stanley 2016 2017: DeVries & Taylor, 2018). It is now an active area of research
Taylor 2018: Ren et al., 2019 [hen et al. 2020at Malinin & Gales| 2018 Kardan al. 2021). In recent studies, the majority of dedicated techniques for OOD detection assume access to part of OOD data (Liang et al. 2017]. which makes OOD detection new binary classification problem. However, in practice, this is unrealistic assumption. In a different approach, outlier exposure (Hendrycks et al.) 2018)) advocates applying different distributions of OOD data for train and test and achieve substantial improvement over stan- dard baseline. In this work we apply the same setup for evaluation of our OOD experiments with auxiliary data.
to mitigaae this drawback (Li Gal 2017;
Adversarial Examples. Adversarial examples, which use a deliberate minute perturbation of input samples that translates to significant modification of model's prediction, are yet another shortcoming that complicates adoption of deep models in security-critica applications such as face recognition and autonomous driving vehicle systems. In addition, these samples point out an intrinsic differ- ence between how inference occurs in humans and current deep models. Crafting a defense against adversarial attacks is an active area of research that witnessed a myriad of algorithms that were sub- sequently broken by more advanced attacks. A significant exception is adversarial training (Madry jet al. 2017 Kannan et al. 2018 Zhang et al., 2019), where the deep model is exposed to adversar- ial examples during the course training to induce degree of immunity against them. However, adversarially trained models suffer from performance degradation on clean (unperturbed) samples. Our proposed framework preserves classification performance on clean images, in addition to im- proving performance against adversarial examples by performing a robust prediction.
Stochastic Neural Networks. Several studies show that applying stochastic models, such as Bayesian neural networks (MacKay 1992, Barber & Bishop 1998 Hinton & van Camp 1993]
Hemández-Lobato & Adams| 20155] Gal & Ghahramani| 2016). can mitigate overfitting and im- prove resistance against OOD and adversarial data. Yet, the main challenge with these approaches is scaling these models up to deep models with millions of parameters. Another technique to construct stochastic deep neural networks is Monte-Carlo dropout (MC-dropout) proposed by Gal & Ghahra- mani (0016). This approach applics dropout (Srivastava et al., 2014) during both train and inference time. The Monte-Carlo sampling at inference time can produce reliable uncertainty estimations, therefore, MC-dropout has been successfully applied to detect OOD Li & Gal (2017) and adver- sarial examples |Feimman et al. (2017): Rawat et al.] (017)) Li & Gal (017). Similarly, self-joint models produce reliable predictions by performing a Monte-Carlo sampling during inference.
Mutual Information for Representation Learning. Paired data has been widely applied for rep- resentation learning by mutual information (Becker & Hinton, 1992. Dhillon et al. 2003; Wang al. 2010 Friedman al. 2013). In particular, paired data has been previously applied for co- clustering (Dhillon et al.] [200]] Wang et al., 2010). In deep networks, IMSAT (Hu et al. 2017). DeepINFOMAX (Hjelm et al.] 2018), and ICC (Ji al. 2019) pursue better features by maximizing information between different forms of representation of pair of variations of the same sample. Although these approaches learn from a pair of images, unlike self-joint paradigm, they do not learn a joint distribution between the two samples in the pair; They merely apply the same feature extrac- tor to each sample. Furthermore, these approaches maximize mutual information between different instances (augmentations) of the same sample.
Contrastive Learning for Representation Learning. There is a growing body of research, such as CPC (Henaff 2020) and SimCLRvv (Chen et al. 2020b), that tries to learn representations from the
3

---

Published as a conference paper at ICLR 2022
similarity between pair of samples based on the contrastive loss (Tian al. 2019 Li al. 2020
Chen et al. 2020b). These representations are later applied to a smaller set of samples for super- vised fine-tuning Given the abundance of unannotated data, recent studies (Goyal et al. 2021) show that this approach can surpass performance of supervised methods that have only access to a limited annotated dataset. Likewise contrastive learning, our framework learns better features based pair of samples. However, we learn an explicit joint distribution that can deal with both annotated and unannotated data, simultancously, and can lead more robust classifiers.
3 SELF-JOINT LEARNING FRAMEWORK
This section introduces our novel approach. We focus on the task of image classification, although our method applies broadly beyond image data and the classification task. We propose to learn stochastic classifier by training a model on the joint distribution of pairs of data samples. We leverage this framework to produce both robust stochastic predictions and fast deterministic predictions, and to incorporate auxiliary unlabelled data.
Throughout the paper, let (X,, Y;) be paired image random variable X, and class (label) random variable Y, and {(X,, Y)) J==1 be the training dataset. We consider the case where the state space of the predictors X, is a subset of IR"" and the state space of responses Y; is a subset of R where m is the number of image pixels times the number of color channels and is the number of classes. Finally, we
At times we overload the notation of Y; and treat it as a member of the set assume there is a true but unknown density q2, y) such that (X,, Y)) pairs are i.id. samples from q(r,y) 9) for i = I,..., n. We emphasize that in this work we always assume that the observations {(X;, Y)) truly i.i.d. from g. even when later introduce model that accommodate sample dependency.
3.1 REVISITING SUPERVISED LEARNING
The most common approach in supervised learning is to define conditional density p(Y|X;0) for a single pair (X, Y) and use the Maximum Likelihood Estimator (MLF)
OMLE((XA (YE arg max p( Yn X1, Xn:
(1)
6
arg max
II P (Y = arg max Σ logp(Y;|Xx;0))
(2)
This framework covers the most widely-used loss functions and tasks such as classification with cross-entropy and regression with square-error. The transition from (1) to (2) is justified not mathematical consequence but as a definition
p(Y X, X,; 0) II P(
chosen by the modeler based on the aisuuussnnttion about the dataset {(,,, Y) We note that this choice is not strictly necessary and that the modeler could alternate factorizations. In the extreme case, the modeler could simply not factorize. Instead, the modeler could define and attempt learning the fully joint conditional distribution p(Y XI,..., N,. 0) directly, which is valid but impractical approach even if the modeler still believes the data is i.i.d. Such a model would be difficult to learn, difficult use, and unlikely generalize well. However, the possibility of an intermediate factorization between i.i.d. and fully joint learning is an intriguing and unexplored direction that the main focus of this work.
The assumption 3 is motivated in part by the desire to learn the conditional distribution XXY; 0) for single pair [x,, Y;). In this work, we instead use a conditional distribution p(Y,, YjI Xy; for two pairs (X;, Yi) and (X,, Yj) as our base model. We call this the self-joint model. Al- though our base model could learn conditional dependency between Y; and Y, given X; and X,, we will exclusively train p(Y;, YjlX, X,, Xy; 5;;00 0) using conditionally independent samples Y; Y; X,) and Y,I(X,, (i(,,,X,)... X,). We note that this conditional independence relation trivially holds under the i.d. assumption. In the context of classification, conditional independence means that the diction matrix Xy; defined by the ideal classifier p(Y,, [X,, X,: should have the form h(X,, 0); 4.9.. where qi 4j IRC represent the true conditional distribution q( ((Y)X,,) and
pre-
4

---

Published as a conference paper at ICLR 2022
(Y,]X))) respectively (see Section 3.2 for full details). An essential difference between self-joint
learning and standard supervised learning is that our model can explicitly learn the sample-to poalle-to-saae to-samlle
relation of pairwise conditional independence, instead of assuming this via the factorization in (
We can once again use Maximum Likelihood to learn our self-joint model by defining
( 1))= = arg max p(Y1,
X ... X,; 0)
(4)
8
arg max
Σ log II
pY,, Xi, X,;
(5)
SES
(iJ)ES
arg max
Σ
lYg yJ X1, Xz:
(6)
SES (AJ)ES
where S is a set of arbitrary non-intersecting parings {i.j} for i,j {1,.... n and S is the set of all such sets. This has the mild requirement that the number of samples n is even. For given S, the factorization in (5) is again justified by the i.i.d. assumption across all observations, and we simply choose to cease factoring at pairs of two samples. Since all such factorizations are valid under the i.i.d. assumption, we choose to average the log likelihood over all possible factorizations when defining our learning objective, which under argmax is equivalent to summing. The ideal model will learn that I(a)) P(Y,, Y5|X; Xy; 0) remains constant across all S. In practice, we use a single sampled uniformly from S and a random subset S, of b pairs in to obtain a stochastic gradient
a
(s) = Σ
log XYI,
de
(K,))S,
This gradient unbiased for the gradient of the objective
66 as discussed in Appendix A.2
3.2 LEARNING AND PREDICTING WITH SELF-JOINT CLASSIFIERS
In this section we will apply the self-joint supervision method to the image classification task. Let p(Y,, Y,,X, X5;0) = h(X,, Xz:0) 0) be a classifier that maps two samples X,, Xj E R to pre- diction for their corresponding joint label Y,YT Tox where Y, Y, TR are the labels of X; and Xy. respectively. Since Y; is a probability density, ||Yal|ı = and < Yi, S 1 for each sealar component Yi, of Y, and the same applies to Yj. It is straightforward to show that the total loss
L(0; AX; Y.) =Σ 3ΣΣ
Σ 1(0; X,,
(8)
SES (1,))ES
where 10, Yi, Y,) = (Y:Y3) log (X,, X5;
(9)
with the dot product taken in the sense of vectors and log taken elementwise, is equivalent to the MLE learning objective in (6) for cross-entropy loss, and that the stochastic gradient can optimize this loss similar to 77). See Appendix A.3 for details of this equivalence between (6) and (8)
Although are learning joint model p(Y, Y" 8); still would like to make predictions for a single test image X via pX () as the standard i.i.d. framework. Note X and X' denote the first and second inputs of the model, respectively, and Y and Y' are their corresponding labels. This can naturally be accomplished by marginalization and Monte Carlo approximation:
x
p( = X|;
(10)
p(Y,Y' -X,,Y'=XXX,XX,, X,:) ;(
(11)
B
SE
where B denotes a random subset of (I. with |B] giving its cardinality, and q(x) is the marginal of the true joint distribution q(x,z) Our model can only approximate p(Y)X;;) instead of having a closed form solution because different samples B yield different predictors, but this estimate will be stable for large |B] due to the weak law of large numbers. In the case where [B] is moderately large, we empirically observe that predictive uncertainty estimated by either the variance of (X) or its entropy more robust than the uncertainty calibration of standard i.i.d. supervised
5

---

Published as a conference paper at ICLR 2022
model, which provides a compelling motivation for exploring the generalization properties of our framework.
Note that (11) has computational complexity of O(BB) and that p(Y|X; requires additional samples X; for estimation, which results in stochasticity. In our experiments, [ is typically less than equal to 50, which is reasonable but non-trivial additional cost. To address these concerns, we note that we can use the same for both input signals, yielding the deterministic classifier
P(Y XX;)) = Σ D
(12)
This is valid estimate for 10 with | provided that X is in-distribution.
Intuitively, in the Monte-Carlo inference in (TT). the joint label affected by the probabilistic label of both input samples. Generally, the two samples produce different levels of confidence and might belong to different classes, therefore, for a better approximation, we repeat our estimation of the prediction for the test sample several times. However, in (2) we know that the two sample inputs have the same label and seeing a replicate of a sample input does not add any new information about its label. Empirically, we observed that (12) often produces similar classification accuracy to
3.3
INCORPORATING AUXILIARY DATA
Interestingly, conditional independency provides a principled tool to incorporate auxiliary data in the self-joint learning framework. Recall that semi-supervised learning (Chapelle et al. 2009 Grand- valet et al. 2005 Lee et al.] 2013), self-supervised learning (Dosovitskiy et al.) [014]] Oord et al., 2018). and OOD detection frameworks also typically take advantage of auxiliary data. Similarly, self-joint framework can benefit from un-labelled data. To this end, we can learn models such that the predictions on training data remain conditionally independent given an auxiliary unlabelled im- age. Following this approach, we effectively train self-joint models that can handle OOD data.
To incorporate auxiliary unlabelled data, we still assume that all data are generated in i.i.d. pairs (X,, but for the auxiliary data we have unobserved labels which we denote by Y; 0.. The factorization in (6) still holds under this assumption, but we are unable to learn the model without numerical value if Y; = 0. A straightforward estimate (similar to pseudo-labels to replace one-hot encoding) is
F Σ PYY.Y, k|x,, Xy; k=1
(13)
which is the marginalization of the predictor matrix h(X,, ;;;0) 0). The same principle applies for missing Y, by summing over the rows. This approach can even be used when both Y; 0 and Y 0. When learning with auxiliary data, we replace any null Y, with the pseudo-label Y; in the objective function (6). Intuitively, auxiliary samples direct a self-joint model to preserve its predictions when it encounters such auxiliary samples. With the right choice of auxiliary data, can improve generalization of the model. For example, if auxiliary data share similar background as original data, the model learns to ignore those superficial features and focus on salient foreground characteristics of the samples.
EXPERIMENTS
In this section, we evaluate the proposed framework on four visual classification tasks, i.e., CIFAR- 10, CIFAR-100, 100, SVHN, and STL-100. The generalized self-joint training procedure is summarized in Algorithm and we use this framework across all experiments. To make general conclusions, we refrain from hyperparameter optimization for any specific dataset and apply almost the same configuration (data augmentation, training recipe, and model architecture) for all datasets (details in Appendix A.5) The overall results provide strong evidence that the framework dramatically improves classification performance, adversarial and OOD data robustness, and facilitates training of large neural networks.
4.1 REGULARIZATION EFFECT OF SELF-JOINT FRAMEWORK
To demonstrate the regularization effect of framework, trained several Wet Zagoruyko & Komodakis 2016 He et al., 2016) models with different capacities on CIFAR-10 10 dataset in standard and self-joint frameworks. In this experiment, we increase the capacity by fixing

---

Published as a conference paper at ICLR 2022
Algorithm I TTriiiin procedure for self-joint learning framework
Input Training D Prainn {(x,, 1-) auxiliary set 2 {[X,. ) loss function I
model p(Y, x';0) 11 0), learning rate number of classes batch size [B]
1. set D Drnin U UD aux
repeat
3.
B {Xx, Yi)) for i (1,..., [D]}
construct batch
4.
+ 0
D loss
for i 1 |B| step size
6.
j + i
Y; B
8.
Y, k== Ei=1 [K(X, xj;0)a.
Eq 13
marginalizing out the output space for X;
Y =
10.
Yj -E [h(X;, 0);0)
Eq
marginalizing out the output space for X, updating loss
11.
L L + +((h(Xi, Xz; 0), )
Eq
12.
9 + 0 aVoL
13.
until convergence
Classification Accuracy
05
Standard SL Self Joint
88-
3.0
10.0
12.5
15.0
17.5
Model Copacity
Figure 2: Visualization of overfitting effect for similar models trained in standard and self-joint framework CIFAR-10. Standard supervised models with high capacity, measured the axis using the width in standard WideResNet-34 suffer from overfitting as shown by the drop in accuracy of the blue line around 10. Self-joint learning can successfully train larger networks due to its regularization effect.
the depth (34 layers) and increasing the width (number of channels) of the models. All the models utilize the same training regime (hyperparameters, augmentations, etc) for training. Figure shows the different classification accuracies for models with different levels of capacity in each framework. In this plot, due to overfitting, increasing capacity after a certain level harms the model performance in the standard supervised learning framework. However, self-joint framework, due its strong regularization effect, is able to achieve increasingly higher levels of accuracy with higher capacity models.
4.2 MODEL ACCURACY AND CALIBRATION EXPERIMENTS
In this experiment, we compare trained models with similar architecture in the standard and self- joint frameworks. We train the models in full and low training-sample regimes, where in the latter we reduce training data to 1% of the total data. Table I depicts the classification accuracy for different models with these two setups on CIFAR-10/100, SVHN, and STL-10. The results suggest that proposed method can effectively train large models that are able to achieve higher accuracy than standard supervised learning models in both experiments. This indicates strong regularization Goodfellow et al. 2016) effect of the proposed framework.
Following common practice in model calibration, we report [Table negative log-likelihood (NLL) as indicator of the quality of predictive confidences. For more detail of this metric refer to (uuo ett al. 2017). Unlike common practice in model calibration literature, we do not run any post- processing step to calibrate models, but rather measure the model confidence values compare how different techniques affect predictive confidence. In this experiment it is evident that self-joint models outperform standard models in model calibration. Moreover, interestingly, observe while

---

Published as a conference paper at ICLR 2022
Table 1:
Classification accuracy (%) and model calibration measured by negative-log- (NLL) on different datasets. Higher accuracy and lower NLL are better.
Model
CIFAR-10
CIFAR-100
STL-10
SVHN
Standard
42.50:1:..3
6.67110.77
22. 22.195
82.48:12.32
Acc. with 1% training data
SJ (ours)
55.52.10.18
12.45.10.16
26.06.11.9
88,98110.80
Standard SJ (ours)
95.68=0003
79.14=0..1
73.95
97.48+00007
Acc. with full training data
97.03-10.04
81.16:10.03
87.69
97.59.1.11
Standard
0.26+#0.02
1.09::0.03
1.56+0.05
0.15
Model cal. on full training data
SJ (ours)
0.22:t0.01
0.85+00022
0.55++0.06
0.15+0.0
Table 2:
Adversarial robustness accuracy ()) of different models measured by the classification accuracy against adversarial examples generated by PGD with different levels of perturbation (e).
CIFAR-10
CIFAR-100
STL.100
SVHN
CIFAR-10
CIFAR-100
STL-10
SVHN
Model
2/256
4/256
Standard 33,2
9.52
21.22
63.04
22.06
5.85
14.0
46.33
Dropout
44.91
14.6
34.1
67.47
33.42
5.25
18.08
46.08
SJ (ours)
62.29
32.65
48.12
82.0
45.52
20.33
35.29
70.85
conventional supervised learning generally produce overconfident models, self-joint models tend to produce under-confident predictions. We attribute this phenomenon the capacity of the trained models, where increasing model depth usually enhances predictive confidence.
4.3 ADVERSARIAL ROBUSTNESS EXPERIMENTS
This section studies the adversarial robustness of self-joint models. Some early studies have pro- posed MC-dropout as defense against adversarial attacks (Feinman et al. 2017 Rawat et al. Li & Gal 2017). Although this defense has later been broken, its positive effect on adver- sarial robustness has been approved (Carlini & Wagner 2017). Therefore, we compare against MC-dropout models as our baseline. In these experiments we apply the gold standard of first-order attacks, PGD, in a white-box setting. To compute the gradients of stochastic model, we apply expec- tation over transformation (Athalye al. 2018). In this method, given stochastic classifier f fæ; p), its gradient is calculated by: V Vx = Ex[Vx](X; p)] RJ E Vxf(X; pt). where denotes model randomness associated with drawing sample (X,, 122
2017
to get the estimate x( in according to
the random indices (i,..... ( n)
[I,...,n.]
Table 2 shows the average performance over several runs of different models on 500 adversarial examples that generated from random test samples with different levels of perturbation (more results reported in Appendix A.55) We set the step size and the maximum number of iterations of PGD to 1/256 and respectively. The stochastic models tend outperform the rest of models for different levels of perturbation on every dataset. Nevertheless, is clearly evident that the stochastic self-joint models (average of 20 samples) are a notch above their stochastic dropout counterparts in adversarial robustness, e.g. about 15% improvement on CIFAR-100 and consistently outperform all the other baselines.
4.4 DETECTING OOD DATA
Detecting OOD data another important aspect of robustness. We apply two evaluation metrics to measure the OOD detection rate of model. The area under the receiver operating characteristic curve (AUROC) is widely used in OOD detection literature. The AUROC is the expected probability that an in-distribution data will have higher detection score than OOD data. We also report detection error, which is the highest detection rate possible that achieved by choosing the optimal threshold.
In this experiment, we include another neural network trained with outlier exposure (OE) (Hendrycks al. 2018), which is dedicated OOD detection. The OE utilizes auxiliary OOD dataset to improve OOD robustness of the model. It specifically tries to make classifier produce a uniform posterior OOD samples (for more detail refer to (Hendrycks et al. 2018)). We included
8

---

Published as a conference paper at ICLR 2022
Table Average AUROC (%)eecccio Error (%) for different models. The OOD datasets include CIFAR-10/100, SVHN, 10L-10, CelebA, and VOC.
Model/in-distnee distribbttinn
CIFAR-10 10
CIFAR-100
STL10
SVHN
Standard
79.46/24.21
74.84/29.01
68.97/33.9
96.09/7.89
Dropout
81.99/22.06
74.69/29.55
70.29/32.53
96.6/6.96
OE
90.23/12.94
81.24/21.71
70.74/32.55
99.99/0.03
SJ (ours)
85.41/19.38
77.03/27.7
82.26/23.6
98.91/3.94 94
SU (ours)
91.34/12.26
81.29/21.72
84.83/20.49
99.99/0.13
OE as a representative of a strong OOD detection baseline and method that incorporates additional auxiliary data. We also train models with auxiliary data as explained in section (33)), which obtain the best performance. For more information about these auxiliary datasets refer to Appendix AA.5]
The test set of SVHN, VOC-2012, CIFAR-10, CIFAR-100, CelebA, and STL datasets is chosen for OOD datasets, where 500 samples are randomly selected (five repetitions) for in- and out-of- distribution data to compare the detection rate of each model. Note that the majority of these datasets have little to no common classes, however STL-10 and CIFAR-IO share 80% of their classes and thus an ideal detection rate is far below 100% AUROC. We generally use maximum softmax probability (e.g. Hendrycks & Gimpel 2016)) as detection score, except for Monte-Carlo dropout models, where we apply entropy instead because found that entropy is slightly superior in this case. For Monte-Carlo approximation we always compute average of 50 samples.
Table shows the average results for all OOD datasets (detailed results are available in Ap- pendix[A.5) where model stochasticity plays a crucial role in boosting OOD detection rate. Another important observation is the vital role of auxiliary data in improving detection rate of stochastic self- joint models. Interestingly, self-joint with auxiliary data generally outperforms outlier exposure, which is a dedicated state-of-the-art OOD detection technique.
4.5 IMPORTANCE OF JOINT DISTRIBUTION MODELING
We conduct study to demonstrate the key role of joint distribution modeling in our framework. Here we train a model that jointly classifies two samples, simultaneously, but remove the joint output space. In partieular, unlike self-joint model, this model performs the two predictions inde- pendently, where, the model includes 2 x output units for a c-classification problem. From another perspective, it is ensemble of two classifiers with shared hidden units. During the test time, sample is fed to both inputs simultaneously and the average of the two predictions are considered as the output. To have fair comparison, we use the same WideResNet architecture as self-joint exper- iment except for the last layer that has 2 x outputs. We observe that not only this ensemble does not help the performance, but also it causes a 2% and 3.5% decline in the classification accuracy of CIFAR- and CIFAR-100 datasets, respectively; even an inferior accuracy than standard supervised models. Moreover, these model show overconfident predictions. These findings clearly demonstrate the superiority of self- jofnt models over mere ensembling of models with shared parameters (more experiments in Appendix A.5
CONCLUSION
In this study, we proposed an alternative way to train and make inference by deep supervised models. Our framework directly learns sample-to-sample relation of conditional independence by modeling the joint conditional distribution of two samples. Our approach has several advantages over the stan- dard supervised learning. First, induces a strong regularization effect that can successfully train large deep models to the higher levels of accuracy than the standard models. Second, it can lever- age extra set of unlaheled data, in conjunction with the training data, to improve the performance of the resulting model. Unlike self-supervised techniques, our approach works end-to-end, and, as a result learns features that are more suitable for the final learning task. Third, depending on the application, the trained models can infer in two modes: (I) fast deterministic inference similar to standard models, and (2) a stochastic inference mode that performs more robust predictions. We demonstrated that this stochastic inference results in more robust predictions against adversarial and OOD samples. We hope this study will be a stepping stone towards more reliable learning models and inspires new avenues for future research.

---

Published as a conference paper at ICLR 2022
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 1414400-14430, 2018.
Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances in adversarial attacks and defenses in computer vision: A survey. IEEE Access, 9:155161-155196, 2021.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference machine learning, pp. 284-293 PMLR, 2018.
David Barber and Christopher M Bishop. Ensemble learning in bayesian neural networks. Nato ASI Series Computer and Systems Sciences, 168:215-238, 1998.
Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):661-63,, 1992.
Zhaowei Cai, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Zhuowen Tu, and Stefano Soatto. Exponential moving average normalization for self self-super and semi-supervised learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 194-203, 2021.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Securily, pp. 3-14, 2017.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2b06)lbook reviews]. IEEE Transactions on Neural Networks, 20(3):542-42,, 2009.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-distribution detec- tion for neural networks. arXiv preprint 1iXXi::0033097111 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi- semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020b.
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv: 04060. 2018.
Inderjit S Dhillon, Subramanyam Mallela, and Dharmendra S Modha. Information-theorrticc co- clustering. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 89-98, 2003.
Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. Citeseer, 2014.
Marzieh Edraki, Nazmul Karim, Nazanin Rahnavard, Ajaml Mian, and Mubarak Shah. Odyssey: Creation, analysis and detection of trojan models. IEEE Transactions on Information Forensics and Security, 2021.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint 1rXiv:1703.004100 2017.
Nir Friedman, Ori Mosenzon, Noam Slonim, and Naftali Tishby. Multivariate information bottle- neck. arXiv preprint 13Xiv:1301.200 2013.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Ian Goodfellow, Y Bengio, and A Courville. Regularization for deep learning. Deep learning, pp- 216-261, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1411v:1412.6572 2014.
10

---

Published as a conference paper at ICLR 2022
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint ww3h:2203301188, 2021.
Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised leamnng by entropy minimization. In CAP, pp. 281-296, 2005.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp- 1321-1330, PMLR, 2017.
Douglas M Hawkins. The problem of overfitting. Journal of chemical information and computer sciences, 44(1):1-12, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference Machine Learning, pp- 11182-1192 PMLR, 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 110.0166 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 18f..:111100000 2.0660 2018.
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of bayesian neural networks. In International Conference on Machine Learning, pp. 1861- 1869. PMLR, 2015.
GE Hinton and Drew van Camp. Keeping neural networks simple by minimising the description length of weights. In Proceedings of COLT-93, pp. 5-13, 1993.
Devon Hjelm, Alex Fedorov, Samuel Maroii-Marchhldoo, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.0661 2018.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eichi Matsumoto, and Masashi Sugiyama. Learning
discrete representations via information maximizing self- augmented training. In International Conference on Machine Learning, pp. 1558-1567, PMLR, 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint 1rXiv:1803.05407, 2018.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9865-98744 2019.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv: mee::8803.0777 2018
Navid Kardan and Kenneth O Stanley. Fitted learning: Models with awareness of their limits. arXiv preprint Xrv:v: (tt9:6609.0000 0226,0226,, 2016.
Navid Kardan and Kenneth O Stanley. Mitigating fooling with competitive overcomplete output layer neural networks. In 2017 International Joint Conference Neural Networks (IJCN), pp- 125. IEEE, 2017.
Navid Kardan, Ankit Sharma, and Kenneth O Stanley. Towards consistent predictive confidence through fitted ensembles. arXiv preprint arXiv:2106. /2070, 2021.
Jan Larsen and Lars Kai Hansen. Generalization performance of regularized neural network models. In Proceedings of IEEE Workshop on Neural Networks for Signal Processing, pp. 42-51. IEEE, 1994.
11

---

Published as a conference paper at ICLR 2022
Dong- Hyyu Lee al. Pseudo-label: The simple and efficient supi-supe learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, 2013.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint FrXiv:2005.04966 2020.
Yingzhen Li and Yarin Gal. Dropout inference in bayesian neural networks with alpha-divergences. In International conference on machine learning, pp. 2052-2061. PMLR, 2017.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution
image detection in neural networks. arXiv preprint arXiv: 2017.
David JC MacKay, practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448-44) 472, 1992.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0603: 2017.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. arXiv preprint 180. 10501, 2018.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv:1906.02229, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03344. 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. arXiv preprint arXiv: 1xXiv:992.00700. 2019.
Ambrish Rawat, Martin Wistuba, and Mrria-Irina Nicolae. Adversarial phenomenon in the eyes of 2017.
bayesian deep learning. arXiv preprint
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark DePristo, Joshua V Dillon, and Balaji Lakshninarayanan. Likelihood ratios for -utto--distributiin detection. arXiv preprint arXiv: 1906. 02845, 2019.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (91):1929-1995 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 1818-2826. 2016.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve cemi-supervised deep learning results. arXiv preprint 1703:1700.0,780, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 05849, 2019.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. 1058-1066. PMLR, 2013.
12

---

Published as a conference paper at ICLR 2022
Pu Wang, Carlotta Domeniconi, and Kathryn Blackmond Laskey. Information bottleneck clustering. In Workshop TextMining SIAM DM, volume 10. Citeseer, 2010.
Andreas Weigend. On overfitting and the effective number of hidden units. In Proceedings of the 1993 connectionist models summer school, volume 1, pp. 335-342, 1994.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks. arXiv preprint
arXiv:1605.07146. 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference Machine Learning, pp. 7472-7482. PMLR, 2019.
A APPENDIX
A.1 INDEPENDENCE, CONDITIONAL INDEPENDENCE, AND I.L.D.
Two events, A and B, are independent, denoted by A IL B, if:
p(A, B) ((A)p(()) (B).
(14)
Furthermore, assuming p(C) > 0, A and are conditionally independent given C, denoted by A IL BJC, if:
p(A P(ACC)
(15)
Equivalently, Eq. 15 can be stated as:
B() = P(A(C))(BIC)
(16)
Intuitively, two events are independent if they are not related, thus occurrence of one does not affect the other. In Eq, [5] two events A and B given C are conditionally independent if occurrence of B does not add any information about A above what we know based on C. As a special case of this view:
rA, C(C = p(A,
(17)
Note that neither independence relationship entails conditional independence nor conditional inde- pendence entails independence.
The assumption that the training data are independent and identically distributed, or iii.d., is common most learning frameworks. A dataset (X,, Y: is called i.i.d. if (X,, Y.) g for some fixed density and the joint density of of the entire dataset satisfies
q' g((11, V1, Y2 .2,, Y,) q(1,, (X) Y2) ... g((, Yx).
The relation of pairwise conditional independence _1I Y,|X, Xj follows from the independence of the joint distribution of (X,, Yi) and (X,, Y,) for all i, q(Y,, Xx)a((X)XX))) ((Y))x))(XX)XX,)
qY,,, Xi,, X3) g(Y,, X,,
(())) This justifies the use of the joint label Y.Y, as the target for p(Y,, Y,,Y,,X,,, Xy; in the self-joint framework, where we are overloading the notation and treating pl (Y, Yj|X, Xy;0) as matrix rather than scalar in the classification example.
g(X,, X,)
A.2 UNBIASEDNESS OF SELF-JOINT STOCHASTIC GRADIENT
In this section we show that the stochastic gradient 7 is unbiased for the gradient of the objective
We note
θ
Σ Σ logp(Y, X,:0) x
log P(,, X,;
08
IS
|S|
SES (iJ)ES
SES
(t.))s
Es
Es
log log Y, X,;0)
where the summations are expectations with respect to uniform distributions on S and S respec- tively. The unbiasedness of (7) simply follows from the fact that each sample S and (i.j) come from the same uniform distributions, and the fact that the mean of unbiased estimators is also unbi- ased.
13

---

Published as a conference paper at ICLR 2022
A.3 EQUIVALENCE MAXIMUM LIKELIHOOD AND CROSS-ENTROPY LOSS FOR JELF-JOINT LEARNING
In this section show that the self-joint cross-entropy loss is equivalent to the Maximum Likelihood objective (6) in the case where the model ((X,X'; represents the conditional density X, p(Y, Y X. ()) This simply follows from observing that
,Y,,' = II ['X,X')))l 10--1111 -0)
which implies
log (X, Y(Y') log (xngh(XXXx
A.4 RELATIONSHIPP TO BAYESIAN NEURAL NETWORKS
A successful approach to estimate uncertainty is Bayesian neural network, where every parameter of the model has a distribution (MacKay) 1992, Barber & Bishop, 1998 Hinton & van Camp 1993 Herrández--oban
& Adams 2015 Gal Ghahramani 2016). From different perspective, these models can be thought as an ensemble of neural networks with shared parameters. Using maximum a posteriori method, these neural networks can readily produce uncertainty estimations. Furthermore, their principled training usually improves the overall performance of the model.
Despite these desirable properties, the application of Bayesian neural networks is very limited in real-world applications. This is mainly due to their excessive computational cost, both during in- ference and training phases, and their inability to appropriately scale up to large amount of data. Moreover, from technical point of view, choice of the right prior distribution in these networks is not always clear. An important exception MC-dropout (Gal & Ghahramani] 2016),, which applies dropout to construct ensemble of neural networks that approximate Bayesian neural network and can be readily applied to almost any conventional deep neural network.
Much like Bayesian neural network, self-joint neural network can be regarded as ensemble of neural networks with shared parameters. In this point of view, every reference sample implicitly generates one instance of parameters of the model, i.e. neural network. In other words,
Pz (Y f h( ()z,
(18)
where Z is drawn from the marginal data distribution g(x). By fixing the second input, X in Eq. 18 of the self-joint model, h, and changing the first input, Z. Eq 18 in effect generating an output distribution for X)) as well as generating distribution over every parameter in the neural network.
From this perspective, self-joint framework introduces different viable solution for producing scal- able Bayesian neural networks.
A.5 ADDITIONAL EXPERIMENTS
This section includes the experimental setup for our experiments, the comparison between perfor- mance of deterministic and Monte Carlo inference procedures for different models, measuring the independence eиror of self-joint models, additional adversarial robustness experiments, and the OOD detection performance for individual OOD datasets.
A.5.1 EXPERIMENTAL SETUP
We apply four common visual classification datasets i.e. CIFAR-10, CIFAR- STL-10, and SVHN- to train our models. Table 4 shows the characteristics of these datasets, along with their corresponding auxiliary dataset in our experiments. The auxiliary set for STL-10 dataset includes a subset of 5,000 samples from the extra set of annotated images provided with this dataset. To make more general conclusions, the same set of hyperparameters to all four datasets. In particular, we limit our augmentations to padding, random-Cros and random horizontal-Flippio (except for SVHN), where image first
adhere
unified experimental setting where try apply
14

---

Published as a conference paper at ICLR 2022
padded with four (12 for STL-10 to compensate for the higher image resolution of this dataset) pixels on every side. Then it is randomly cropped to the original size, and finally randomly horizontally flipped.
We implemented our code in Pytorch (Paszke et al. 2019). All datasets are normalized according to the mean and standard-deviation of their training set. All experiments apply WideResNet (with trainable parameters for BatchNorm layers) architecture, where models trained on CIFAR- 10/100 and SVHN (all datasets with 32 32 images) are always 64 layer WideResNet with width=10. The STL-10 model utilizes deeper 124 layer WideResNet with width= 5. Moreover, due to higher resolution of STL-10 images, we replaced the first convolutional layer WideResNet with kernels (stride= 2), and the final average pooling was replaced with an adaptive average pooling layer. These changes are made to make the STI-10 model as close as possible to other models. Thus, we did not attempt to optimize the architecture for the STI.-10 dataset. The networks make no distinction between two permutations of an input pair, i.e., the input pairs X, X, and X,, X; are assigned to the same class. We conduct experiments with shared parameters in the first convolutional layer for the two inputs on CIFAR-I0. However, does not lead to significant performance improvement. As result, in our final experiments learning rate always starts with 0.4 and it decays after every 10 epochs with a constant scale of 0.81. Note that we define epoch be roughly 390, 390, 196, and 572 iterations 100, STL-10, and SVHN, respectively. The batch size starts at 128 (256 for STL-10) and is doubled every 20 epochs. Maximum number of epochs is always 180, except for the regularization experi- ment, after which we did not observe any noticeable improvement in training loss in any experiment. We apply early stopping, where discontinue training if neither the validation accuracy nor its loss is improved for more than 20 epochs. We always keep 10% of test data for validation set and report the final test accuracy on the rest. The optimizer is always SGD with Nestrov (momentum=0.9) and weightDecay=e 1), Finally, we apply exponential moving average normalization (Cai et al. 22021 Tarvainen & Valpola| 2017 [zmailov et al., 2018), where momentum= 0.95 and the average model updated after every 100 iterations.
do not share the parameters of the first convolutional layer. The
CIFAR-10, CIFAR-
A.5.2 MONTE CARLO INFERENCE IN SELF-JOINT FRAMEWORK
The robust self-joint model primarily based on the Monte Carlo inference. Table reports the performance of the trained models using the deterministic inference procedure in Eq. 12 versus the Monte Carlo estimation in Eq. 11. Overall, the deterministic inference performs well. However, Monte Carlo inference generally improves with increasing the sample size. Table f the effect of the sample size on performance self-joint and Dropout models in Monte Carlo estimation. The results averaged
runs of each experiment.
A.5.3 MEASURING INDEPENDENCE EXPERIMENTS
Recall that self-joint modeling provides a flexible framework that learns sample independency stead of assuming it. To investigate if self-joint models are able to successfully learn conditional sample-independency relations we conducted an experiment. This experiment utilizes the test sam- ples from different datasets and measure the sample-independency for one pair according to the following independence error:
1xxx, D(YYY'' = x|x, 1 = x
Table 4: Overview of the datasets in our experiments. Note that always keep the first 10% test data from each class as validation set and report the test accuracy on the rest.
Dataset
Dimensionality
No. Classes
Training Set
Test Set
Auxiliary Set
CIFAR-10
3,072
10
50,000
10,000
CIFAR-100
CIFAR-100
3,072
100
50,000
10,000
CIFAR-I0
STL-10
2,654,208
10
5,000
8,000
STL-10 extra
SVHN
3,072
10
73,257
26,032
CIFAR- SIFAR-100
15

---

Published as a conference paper at ICLR 2022
Table 5: Classification accuracy of different models using deterministic versus Monte Carlo infer- ence. "MC' denotes the accuracy computed by Monte Carlo procedure, and 'ASJ' is representing the self-joint models that apply auxiliary data during training.
Model
CIFAR-10
CIFAR-100
STL-10
SVHN
Dropout
95.88
79.77+0.01
78.58+00.3
97.55=10.01
MC-Dropout
95.75:10.04
79.73:0.07
78.42::0.77
97.53+10.03
SJ (ours)
97.03:10.04
81.16:10.03
87.69
97.5910.1
MC-SJ (ours)
96.73+0.00
82.55:0.11
85.9442.31
97.88
ASJ (ours)
97.3=0.0
81.39::0.08
87.18+0.0
97.54+0.0
MC-ASJ (ours)
95.78+00.15
81.16+0..5
82.18+2.73
97.46+0000
Table 6: Classification accuracy of stochastic models with different sample sizes. Larger sample size results in higher accuracy for self-joint models.
Model
Sample Size
CIFAR-10
CIFAR-100
STL-10
SVHN
MC-Dropout
20
95.69=0.077
79.6810.16
78.33=1.09
97.54=0.03
MC-Dropout
40
95.7910.07
79.6310.01
78.3310.96
97.53
MC-Dropout
50
95.75
79.7310.07
78.42-0.97
97.53+1.03
MC-SJ (ours)
20
96.5
82.0340.16
85.21=2.63
97.79=0.02
MC-SJ (ours)
40
96.66=0.044
82.32+0.03
85.51=2.38
97.87=0.02
MC-SJ (ours)
50
96.73+1.00
82.
85.94+2.31
97.88
MC-ASJ (ours)
20
9..22
78.81
75.92+14.75
47.05
MC-ASJ (ours)
40
95.51
80.4440.39
80.41
97.46
MC-ASJ (ours)
50
95.78-0.15
81.1610.54
82.18+2.73
97.46=0.0
Furthermore, to limit computational cost, in this experiment we restrict sample pairs X X'. The results in Table 7 hiihhight that self-joint models are successfully learning conditional independency relations. Moreover, this table provides clear evidence that the independence relations are learned more precisely for in-distribution data than other unfamiliar datasets.
A.5.4 ADVERSARIAT. ROBUSTNESS EXPERIMENTS
In the main body of the paper reported adversarial robustness against PGD attack with two different setups. This section shows additional adversarial robustness experiments. First, we include additional setup for PGD, where the maximum perturbation is limited to 3 and the maximum iterations is set to 4. Following the same setting as the main paper, this experiment applies 20 steps for finding the adversarial examples using expectation Similarly, to produce robust predictions, 20 iterations Monte-Carke sampling is applied. We also repeat these experiments with different perturbations transformation that follows 40 iterations of Monte-Carlo sampling for defence. Finally, we include
transformation (Athalye et al.) 2018).
2, 4) for 40 steps of expectation over
Table 7: Independence error for different self-joint models trained CIFAR- SVHN, and STL-10. Low independence errors indicate that self-joint models could learn the independence re- lation successfully. In addition, lower errors the trained distribution than other datasets showcase the advantage of learning independency relations over assuming it.
test dataset
Model
SVHN
STL10
CIFAR-10
CIFAR-100
SVHN
0.0011+0.0
4.0072++00000
0.0074++.0000
0.0074++0.000
STL-10
0.00740.0034
0.0044+t0.00)
0.0083+0..0017
0.0075++0.001
CIFAR-10
0.0076+10.0066
0.0049=+0.0001
0.0023+1.0001
0.0084+0.0
CIFAR-IO0 100
0,000110..0
0.000110..0
0.000110.0
0.0:10.0
16

---

Published as a conference paper at ICLR 2022
Table 8: Adversarial robustness of different models measured by the classification accuracy(% + stdev) against adversarial examples generated by PGD for different levels of perturbation.
Model
CIFAR-10
CIFAR-100
STL-10
SVHN
Standard (c = 2/256)
33.2 4.86
9.52+ 1.83
21.22++ 3.53
63.04 3.43
Standard (c 3/256)
25.92+ 4.77
7.23+ 2.08
15.8 2.41
50.37+ 3.41
Standard (e 4/256)
22.06 3.8
5,85# 1.76
14.0 2.14
46.33 2.72
2/256, MC sample size=20, EOT steps=20
Dropout
44.91 1.55
14.6 2.16
34.1# 3.68
67.47 1.94
SJ (ours)
62.29 4.73
32.65 2.09
48.12 5.98
82.0# 1.93
3/256, MC sample size=20, EOT steps=20
Dropout
38.04 1.84
9.02 0.98
24.98 3.13 3 6.15
54.78+ 2.76
SJ (ours)
52.23 3.09
23.6+ 2.08
74.83+ 1.54
4/256, MC sample size=20,, BOT steps=20
Dropout
33.42 2.28
5.25 0.82
18.08 1.87
46.08 4.1
SJ (ours)
45.52 4.17
20.33 2.83
35.29 6.19
70.85 2.17
2/256, MC sample size=40, 40, EOT steps=40
Dropout
45,06=t 2.58
14.46 1.48
34.63 1.79
67.02 2.36
SJ (ours)
58.4+ 3.83
29.42: 2.04
44.27 4.35
80.73 2.41
3/256, MC sample size=40, EOT steps=40
Dropout
36.58 1.42
7.83 1.2
23.96 1.55
54.69 1.86
SJ (ours)
47.62 4.12
22.29 1.71
34.23# 5.69
75.12 1.27
4/256, MC sample size=40, EOT steps=40
Dropout
31.58 1.9
5.634 1.02
18.524 2.59
47.634 3.77
SJ (ours)
41.85 2.36
17.5+ 1.58
32.42+ 5.37
69.54 1.3
2/256, MC sample size=40, EOT steps=20
Dropout
45.67 2.02
14.98 1.68
35.17+ 3.17
66.85+ 2.02
SJ (ours)
61.541 3.4
32.79 1.35
47.98 6.18
83.1 1.87
3/256, MC sample 40ze=40,
EOT steps=20
Dropout
38.44 2.24
8.21 0.98
23.984
3.26
55.354 2.86
SJ (ours)
49.9 3.65
25.4 1.95
39.37 7.02
75.94 2.18
4/256, MC sample size=10. 10, BOT steps=20
Dropout
32.52 3.16
6.0 1.16
18.024 1.82
47.19 3.69
SJ (ours)
44.75 2.66
22. 19- 1.96
35.75 6.2
72.83+ 1.55
setting, where adversarial examples are generated by 20 steps of expectation over transformation, whereas the robust inference applies 40 iterations of Monte-Carlo sampling-
Table 8 shows the performance of different models (two models per method) on 500 adversarial examples that are generated from random test samples (five repetitions per model) for these settings. Note that the step size of PGD is always set to 1/256 in our adversarial experiments. These results reiterate that the stochastic self-joint model outperforms the rest models for different levels of perturbation every dataset. It is also evident that both MC-dropout and self-joint models gen- erally perform better with larger (Monte-Carlo) sample sizes when attack strength is unchanged. Nevertheless, this improvement is more remarkable for self-joint models.
17

---

Published as a conference paper at ICLR 2022
Table 9: AUROC (%) for different models trained on CIFAR-10/100, SVHN, and STL-10. Note that each Outlier Exposure model is explicitely trained on a OOD dataset indicated by (f) as OOD data. Each ASJ model is also utilizing the dataset highlighted by (f) as auxiliary data.
OOD dataset
Model
SVHN
STL10
CIFAR-10
CIFAR-100
CelebA
VOC
CIFAR-10
Standard
90.29:13.13
65.3542.15
86.79+0.92
76.8214.48
79.05
Dropout
1..77
65.94
89.77
80.4
81.45
OE
98.78=0057
68.5941.71
197.65-10.63
98.3910.97
87.76
SJ (ours)
93.3840.84
6..25
93.1610.59
86.6911.25
86.57=0.93
ASJ (ours)
98.71+0027
10.92
498.5
99.65#0.19
88.92
CIFAR- 100
Standard
81.7+1.15
7..5
78.44+1.03
58.88+2.116
76.62+1.116
Dropout
1..95+11.6
81.65+
80.23+1.4
57.95+33.42
76.97
OE
1..3
94.78:10.43
198.01=0.43
54.5412.99
81.48
SJ (ours)
8..76+11.22
82.6410.477
81.2440.911
60,86:2.32
19.67
ASJ (ours)
79.02+22.09
94.66110.3
196.4220.66
53.42+2.26
82.91
STL-10
Standard
70.48+3.72
68.87
68.32+2.27
67.54+2.23
69.63
Dropout
1..37
71.31
67.96:10.3
73.3
72.41
OF
7..744
66.45
7..56+1.755
71.1+
66.84+1.45
SJ (ours)
84.1747.885
82.6643.62
80.2444.22
83.9242.33
1.63
ASJ (ours)
86.1617..51
83.2542.32
84.6242.92
88.73#1.71
81.4:0097
SVHN
Standard
9..26
95.48
95.31+1.44
97.2+
96.2+t1.3
Dropout
96.83:10.87
96.56+0.94
96.46:10.67
97.65+0.37
96.59-0.8
OE
99.96110.06
100.0+0.00
$100.0
99.9810.05
100.0
SJ (ours)
99.03:10.16 16
98.88-10.38
98.6510..19
99.2310.31
98.77
ASJ (ours)
99.9910.01
99.97#0.04
19.098-1.066
100.010.0
99.9910.01
A.5.5 DETECTING OOD DATA
Robustness against OOD data is an important aspect of stochastic self-joint models. Here, we extend the results in the main body of the paper for each OOD dataset. The datasets that we consider for OOD include: SVHN, STL.-10, CIFAR-10, CIFAR-100 CelebA, and VOC. All these datasets con- tain natural color images. Note that we did not remove any overlapping class from these datasets. However, the majority of these datasets have little to no common classes, except for STL-10 10 and CIFAR-10, which share 80% of their classes. We report the area under the receiver operating char- acteristic curve, AUROC, and the detection error for each model against each OOD dataset.
We randomly select 500 samples from test set of in-distribution and OOD data compare the performance of each model (five repeats). Maximum softmax probability (e.g. Hendrycks & Gimpel (2016) is applied for detection score. However, apply entropy for Monte-Carlo dropout models, because we found it slightly superior for these models. Finally, the Monte-Carlo approximation always compute average of 50 samples.
Tables and 10 show the performance of different methods on OOD detection in AUROC and detection errors metrics, respectively. The tables suggest that outlier exposure (OE) (Hendrycks) et al. 2018) and self-joint models trained with auxiliary data perform almost always better than the rest. These results suggest learning to make predictions independent of OOD data can effectively reduce vulnerability to unseen OOD samples.
18

---

Published as a conference paper at ICLR 2022
Table 10: Detection Error(% for different models trained on CIFAR-10/100, SVHN, and STL-10. Note that each Outlier Exposure model is explicitely trained OOD dataset indicated by (t) as OOD data. Each ASJ model is also utilizing the dataset highlighted by (f) as auxiliary data.
OOD dataset
Model
SVHN
STL10
CIFAR-10
CIFAR-100
CelebA
VOC
CIFAR-10
Standard
14.67=3.5
36.03
17.78
27.98:1.559
24.57
Dropout
12.75+1.38
35.54+2.21
16.01
23.5+0.8
1..
OE
3.39=0.54
34.65#1.7
+5.87+0.76
2.85
17.93
SJ (ours)
11.91=1.15
34.92
12.54
18.5442.12
19.01
ASJ (ours)
0.94
33.46:+1.11
+5.06-10.44
2.06
15.77
CIFAR-100
Standard
24.02:1..9
26.5=0.92
26.71
39.37+
28.46
Dropout
28.04
25.12::.91
26.49
39.31
28.78
OE
24.63++0.89
11.51+0.46
+5.9+0.87
40.64+2.55
25.88
SJ (ours)
25.51
24.1=0.77
25.49-
37.06+11.61
26.37
ASJ (ours)
25.59+1.27
10.9=0.8
18.3+1.2
40.33+2.02
23.46+1.00
STL-10
Standard
32.56
34.24111009
33.91
35.21
33.61
Dropout
33.77
32.35+1.119
34.7440.85
29.78+1.21
32.02+1.31
OE
24.78:13.9
36.4942.25
33.410.95
32.19:0.75
35.88=0.92
SJ (ours)
22.39-1.226
23.6+3.39
25.5843.25
20.72+1.9
25.71
ASJ (ours)
17.94+4.06
21.86
21.24+2.4
16.69
24.73+1.3
SVHN
Standard
7,86=1.35
8.6:10.88
8.82+1.5
6.41
7.77
Dropout
7.21=1004
7.3810.91
7.1810.81
5.7740.64
7.25
0.06=0.05
0.0.1..0
10.05-10.05
0.03+0.07
0.03
SJ (ours)
3.69=0.59
3.85+0.67
4.81=0.61
3.05+0.66
4.32-0.46
ASJ (ours)
0.07=0.08
0.240.12
10.194:0.15
0.06+0.07
0.13#0.09
19


Reviews:
[{"review_id": "bgq04xhGww", "reviewer": "Reviewer_Ky9n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The main idea of modelling the conditional joint label distribution of a pair of data samples of the paper is straightforward. The proposal of the parameterisation of the conditional joint label distribution and strategies for making predictions in the inference phase is intuitive.\n\n2. The experiments cover a wide range of applications.\n\n3. The paper is well written and easy to follow.\n\nWeakness:\n\n1. I think it is analytically unjustified how or why explicitly modelling conditional independence can help deal with overfitting, adversarial examples, detecting OOD data, although the paper shows a considerable amount of experiments in various applications. For example, it is not clear where the robustness against adversarial attacks of the proposed method comes from. I would expect some theoretical or analytical study on this point. Also on this point, in Eq. (9), $Y_iY^T_j$ indicates the true labels of two samples, which is a one-hot matrix, without any label correlation. It is hard for me to understand why MLE in this case helps.\n\n2. Although the paper studies several applications, it is a bit hard to position the paper in the literature. Specifically, for each specific applicaltion, the paper mainly compares with the fundamental methods. But there is a rich literature in each application. For example, in adversarial machine leanring, there are many advanced methods that improve adversarial robustness without adversarial training, e.g. in [1] where an ensemble model is used. I'm wondering how the proposed method performs in the comparion with the state-of-the-art methods in their specific application.\n\n[1] Pang, Tianyu, et al. \"Improving adversarial robustness via promoting ensemble diversity.\" International Conference on Machine Learning. PMLR, 2019.\n\n3. It would be interesting to study the complexity (running speed) of the proposed method in both training and inference phases. As the proposed might need to sample multiple pairs of data to make prediction of one data sample, it is important to show how much addtional complexity would be added in the inference phase.\n ", "summary_of_the_paper": "This paper introduces a pairwise loss function for supervised learning, named self-joint learning. Instead of predicting the conditional distribution of the label given one data sample as in conventional supervised learning, the proposed self-joint learning framework predicts the conditional joint label distribution of a pair of data samples. The paper proposes the corresponding constructions of the conditional joint label distribution and several strategies for making predictions in the inference phase. The paper shows empirical studies on dealing with overfitting, adversarial robustness, and detecting OOD data.", "summary_of_the_review": "The paper studies an interesting idea and shows that the proposed idea improves some baselines in several applications. But I feel that it needs additional study on why the idea works and more comparison with more advanced methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "79tJyc2F8Dg", "reviewer": "Reviewer_PZCK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Simple idea that works very well\n2. Paper is easy to understand\n3. Formulation of the problem is well grounded theoretically\n4. Shows advantages over several different areas of deep learning\n\nAreas of improvements:\n1. While results are good, there is a lack of explanation on why the results are good. My sense is that the combinations of pairs of data span a much larger input space than to feed data one by one into the network. In a sense the amount of ‘independent’ data grows exponentially. This is a huge regulariser\n2. The data is not strictly independent since the authors feed different combinations of pairs. In any case, iid assumptions is not strictly enforce in many real world problems\n3. Icing on the cake would be having some theoretical proofs of properties of this network. Especially some regularising properties.\n\nQueries:\n1. How is the images being fed in? Side-by-side or concat channel wise? Fig1 seems to suggest side-by-side.", "summary_of_the_paper": "Paper propose a simple but effective method of learning. Feeding in a pair of data and learn the join conditional probabilities for predictions. The advantage is that the combination of data goes as the number of ways to partition the original data into pairs. This is a huge number.\n", "summary_of_the_review": "Good piece of work, can be enhance some more with more extensive analysis of the results. E.g. more explanations of the observations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4l7GAqeKj4I", "reviewer": "Reviewer_sxB6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The empirical results look convincing, and the Authors' interpretation of the method as regularising training sounds convincing. The term Y_i Y_j^T in Eq (9) means that maximising the MLE (8) fully would lead to a classifier which evaluates each sample independently. Hence, classifying two samples at a time matters only if MLE is not fully optimised. Hence, it's a regularisation scheme, as the Authors say. If the model architecture does not ensure the factorisation of the h() function by design, the model has to competing objectives to fulfill: get better at classifying samples correctly and get better at classifying them independently. Also, a training dataset containing N examples produces N^2 pairs of examples, which makes the training loop harder for the model. This means it's harder to overfit it.\n\nHowever, if the model architecture was big enough, it would probably overfit nevertheless, because it would be able to produce 100% conditionally independent label predictions and overfit each of them independently. This raises a question: could a similar effect could be achieved simply by training a smaller model, or stopping training earlier?\n\nSome parts of the paper are not clear:\n- In the Introduction, the Authors say \"By training with a pair instead of one sample, the model explicitly learns sample-to-sample relationships\". But if the labels are deterministic (as is most often the case), there are no such relationships (deterministic random variables are always independent). What \"relationships\" are learnt then?\n- When computing the gradient for training, are the pseudo-labels \\hat{Y} differentiated too?\n- How were the network architectures modified to support increased input and output dimensionality. Are the images concatenated together? The output layer must be larger (c^2 instead c logits), but what about the hidden layers?\n- How is the model capacity compared between standard and joint models in Figure 2? What does x = 10 mean on the X axis of the plot? Do both models have the same number of weights? Or the joint one has 2x as many?\n\nFinally, there are many small problems with writing and style, which should be corrected (e.g. \" using loss penalties and to use stochas-\nticity\", \" Even a well-trained model could benefit from having additional data available at the time of inference much, as a human expert can still benefit from having\", \"Although, these approaches learn...\", etc.)", "summary_of_the_paper": "The Authors propose a new way of training classifier models. Instead of classifying each i.i.d. example independently during training and inference, they jointly classify a pair (X1, X2) of them, returning a joint distribution of labels (Y1, Y2). The loss function is modified so that the model learns that labels Y1 and Y2 are conditionally independent given X1 and X2.\n\nThis approach is shown experimentally to lead to superior model performance for the same model capacity (measured as relative layer width). The Authors attribute it to the regularising effect of new training method. Additionally, joint training on pairs of examples allows the use of auxiliary unlabelled examples.\n", "summary_of_the_review": "The presented method is interesting and should be accepted for publication. However, the paper could be improved and there are some open questions which should be investigated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores a classification approach based on labeling pairs of inputs concurrently using a single network, rather than singletons. The authors test the approach on adversarial robustness (towards norm bounded perturbations), OOD detection next to basic standard accuracy calculations.

While the key idea is potentially interesting and the paper has received positive comments from the majority of reviewers, there were also some concerns that need to be addressed in a final manuscript:

* The paper does not motivate or explain theoretically why the joint classification framework is superior, beyond verbose arguments. These
arguments need to be better clarified and linked with the experimental evaluation.

* While the empirical results are perceived as positive by the reviewers, one reviewer has raised the concern about the comparisons. The adversarial robustness and OOD comparisons are indeed basic. The adversarial attack used here is quite a weak PGD attack with a small radius and low iteration budget. Possibly include stronger attacks. The OOD comparisons are with standard baselines only. Please include further comparisons.  

In its current form, the paper seems to be acceptable, and I strongly encourage the authors to improve both the theoretical justification, and empirical exploration in the final version.
Decision: Accept (Poster)