Year: 2021

Abstract:
Supervised learning is a fundamental framework used to train machine learning systems. A supervised learning problem is often formulated using an i.i.d. assumption that restricts model attention to a single relevant signal at a time when predicting. This contrasts with the human ability to actively use related samples as reference when making decisions. We hypothesize that the restriction to a single signal for each prediction in the standard i.i.d. framework contributes to wellknown drawbacks of supervised learning: making overconfident predictions and vulnerability to overfitting, adversarial attacks, and out-of-distribution data. To address these limitations, we propose a new supervised learning paradigm called self-joint learning that generalizes the standard approach by modeling the joint conditional distribution of two observed samples, where each sample is an image and its label. Rather than assuming samples are independent, our models explicitly learn the sample-to-sample relation of conditional independence. Our framework can naturally incorporate auxiliary unlabeled data to further improve the performance. Experiments on benchmark image datasets show our method offers significant improvement over standard supervised learning in terms of accuracy, robustness against adversarial attacks, out-of-distribution detection, and overconfidence mitigation.

heading: 1 INTRODUCTION
text: Neural networks lay at the heart of deep learning success. However, problems such as overfitting (Weigend, 1994; Hawkins, 2004) and confidence miscalibration (Guo et al., 2017) persist across the vast majority of supervised models. Model calibration requires the predictive confidence of a classifier to be faithful to the ground truth, meaning the expected accuracy of all samples with m% confidence should be almost m%. However, deep models tend to produce overconfident predictions (Guo et al., 2017), which undermines utilization of their confidence values. Consequences of miscalibration such as susceptibility to adversarial attacks (Szegedy et al., 2013; Goodfellow et al., 2014; Madry et al., 2017; Akhtar & Mian, 2018; Akhtar et al., 2021; Edraki et al., 2021) and poor performance on out-of-distribution (OOD) examples (Liang et al., 2017; Hendrycks et al., 2018) limit the utilization of deep learning systems in real-world applications.
Existing tools to combat overfitting and miscalibration include artificially increasing training data through data augmentation (Shorten & Khoshgoftaar, 2019), using loss penalties, utilizing stochasticity to constrain a model’s ability to learn data idiosyncrasies (Larsen & Hansen, 1994; Wan et al., 2013; Gal & Ghahramani, 2016), and smoothing labels to moderate overconfident predictions (Szegedy et al., 2016; Müller et al., 2019). While increasing the number of samples during training using data augmentation has been explored, an unexplored direction is to increase the number of samples used when making a prediction. Even a well-trained model could benefit from having additional data available at the time of inference, similar to a human expert that can still benefit from having a selection of reference examples when making a decision about a single test example. We hypothesize the problems of overfitting and miscalibration could be further exacerbated by limitations of the commonly used model p(Y |X; θ) for a single sample-label pair (X,Y ) and parameter θ. This model form only allows other training pairs (X ′, Y ′) to provide reference information for the conditional probability p(Y |X; θ), and therefore the model prediction for X , implicitly via training and not at all during inference.
In this paper, we propose a new paradigm called self-joint supervision to construct models that make predictions using a given test signal and a randomly selected sample of reference signals (typically
a subset of training or validation pairs). This work focuses on image signals but the framework applies broadly. During training, a self-joint model learns to predict the joint distribution of a pair of labels given their associated images. By training with a pair instead of one sample, the model explicitly learns sample-to-sample relationships. In this work, we retain the standard i.i.d. assumption of supervised learning and our models actively learn the relationship of pairwise conditional independence (see Appendix A.1 for a brief review of these concepts). Learning conditional independence is the key innovation of our work and the main difference from standard supervised learning, which has no explicit cross-sample modeling.
We apply Maximum Likelihood to learn self-joint models in the similar framework as a standard model. After training, the model infers individual labels by marginalization of the joint distribution output. We propose inference procedures to produce both a stochastic and a deterministic output. Figure 1 shows an intuitive visualization of the self-joint training and inference procedures. Experiments with stochastic self-joint models provide evidence that learning conditional independence among sample pairs can lead to models more resistant to adversarial attacks and OOD data. Moreover, our method can naturally be extended to incorporate auxiliary unannotated data during training by using marginal prediction to generate pseudo-labels. Our experiments suggest adding partially annotated data can further safeguard against OOD data.
Overall, our findings provide a novel approach to train more consistent and robust deep models and an alternative strategy to utilize auxiliary data. The main contributions of our work are summarized below.
1. We introduce a novel perspective of self-joint learning where the model can accommodate sample dependency. By applying the model to pairs of i.i.d. data (pairs are i.i.d.), we learn rather than assume conditional independence between labels given two input images.
2. We show self-joint learning leads to substantial improvement in classification accuracy as the network capacity scales to sizes that were previously found to be detrimental due to overfitting.
3. We construct a new type of stochastic models that can provide robust predictions for adversarial examples and detect OOD data. In particular, we explore the impact of our approach on model robustness against adversarial attacks and show it drastically improves adversarial robustness against small perturbations.
4. We propose a novel technique to inject priors into learning models with the help of auxiliary unlabelled data. Our experiments show that utilizing auxiliary data can improve the model’s ability to recognize OOD data.

heading: 2 RELATED WORK
text: Out-of-Distribution (OOD) Detection. OOD data play a vital role in real-world applications, where the model commonly encounters samples from novel classes. Ideally, the model should produce low confidence predictions with high level of uncertainty for data from unknown classes. However, several studies suggest that deep models frequently make confident predictions for OOD data (Kardan & Stanley, 2016; DeVries & Taylor, 2018). It is now an active area of research to mitigate this drawback (Li & Gal, 2017; DeVries & Taylor, 2018; Ren et al., 2019; Chen et al., 2020a; Malinin & Gales, 2018; Kardan et al., 2021). In recent studies, the majority of dedicated techniques for OOD detection assume access to part of OOD data (Liang et al., 2017), which makes OOD detection a new binary classification problem. However, in practice, this is an unrealistic assumption. In a different approach, outlier exposure (Hendrycks et al., 2018) advocates applying different distributions of OOD data for train and test and achieve substantial improvement over standard baseline. In this work we apply the same setup for evaluation of our OOD experiments with auxiliary data. Adversarial Examples. Adversarial examples, which use a deliberate minute perturbation of input samples that translates to significant modification of model’s prediction, are yet another shortcoming that complicates adoption of deep models in security-critical applications such as face recognition and autonomous driving vehicle systems. In addition, these samples point out an intrinsic difference between how inference occurs in humans and current deep models. Crafting a defense against adversarial attacks is an active area of research that witnessed a myriad of algorithms that were subsequently broken by more advanced attacks. A significant exception is adversarial training (Madry et al., 2017; Kannan et al., 2018; Zhang et al., 2019), where the deep model is exposed to adversarial examples during the course of training to induce a degree of immunity against them. However, adversarially trained models suffer from performance degradation on clean (unperturbed) samples. Our proposed framework preserves classification performance on clean images, in addition to improving performance against adversarial examples by performing a robust prediction. Stochastic Neural Networks. Several studies show that applying stochastic models, such as Bayesian neural networks (MacKay, 1992; Barber & Bishop, 1998; Hinton & van Camp, 1993; Hernández-Lobato & Adams, 2015; Gal & Ghahramani, 2016), can mitigate overfitting and improve resistance against OOD and adversarial data. Yet, the main challenge with these approaches is scaling these models up to deep models with millions of parameters. Another technique to construct stochastic deep neural networks is Monte-Carlo dropout (MC-dropout) proposed by Gal & Ghahramani (2016). This approach applies dropout (Srivastava et al., 2014) during both train and inference time. The Monte-Carlo sampling at inference time can produce reliable uncertainty estimations, therefore, MC-dropout has been successfully applied to detect OOD Li & Gal (2017) and adversarial examples Feinman et al. (2017); Rawat et al. (2017); Li & Gal (2017). Similarly, self-joint models produce reliable predictions by performing a Monte-Carlo sampling during inference. Mutual Information for Representation Learning. Paired data has been widely applied for representation learning by mutual information (Becker & Hinton, 1992; Dhillon et al., 2003; Wang et al., 2010; Friedman et al., 2013). In particular, paired data has been previously applied for coclustering (Dhillon et al., 2003; Wang et al., 2010). In deep networks, IMSAT (Hu et al., 2017), DeepINFOMAX (Hjelm et al., 2018), and ICC (Ji et al., 2019) pursue better features by maximizing information between different forms of representation of a pair of variations of the same sample. Although these approaches learn from a pair of images, unlike self-joint paradigm, they do not learn a joint distribution between the two samples in the pair; They merely apply the same feature extractor to each sample. Furthermore, these approaches maximize mutual information between different instances (augmentations) of the same sample. Contrastive Learning for Representation Learning. There is a growing body of research, such as CPC (Henaff, 2020) and SimCLRv2 (Chen et al., 2020b), that tries to learn representations from the similarity between a pair of samples based on the contrastive loss (Tian et al., 2019; Li et al., 2020;
Chen et al., 2020b). These representations are later applied to a smaller set of samples for supervised fine-tuning. Given the abundance of unannotated data, recent studies (Goyal et al., 2021) show that this approach can surpass performance of supervised methods that have only access to a limited annotated dataset. Likewise contrastive learning, our framework learns better features based on a pair of samples. However, we learn an explicit joint distribution that can deal with both annotated and unannotated data, simultaneously, and can lead to more robust classifiers.

heading: 3 SELF-JOINT LEARNING FRAMEWORK
text: This section introduces our novel approach. We focus on the task of image classification, although our method applies broadly beyond image data and the classification task. We propose to learn a stochastic classifier by training a model on the joint distribution of pairs of data samples. We leverage this framework to produce both robust stochastic predictions and fast deterministic predictions, and to incorporate auxiliary unlabelled data.
Throughout the paper, let (Xi, Yi) be a paired image random variable Xi and class (label) random variable Yi, and {(Xi, Yi)}ni=1 be the training dataset. We consider the case where the state space of the predictors Xi is a subset of Rm and the state space of responses Yi is a subset of Rc, where m is the number of image pixels times the number of color channels and c is the number of classes. At times we overload the notation of Yi and treat it as a member of the set {1, . . . , c}. Finally, we assume there is a true but unknown density q(x, y) such that (Xi, Yi) pairs are i.i.d. samples from q(x, y) for i = 1, . . . , n. We emphasize that in this work we always assume that the observations {(Xi, Yi)}ni=1 are truly i.i.d. from q, even when we later introduce a model that can accommodate sample dependency.

heading: 3.1 REVISITING SUPERVISED LEARNING
text: The most common approach in supervised learning is to define a conditional density p(Y |X; θ) for a single pair (X,Y ) and use the Maximum Likelihood Estimator (MLE)
θ̂MLE({Xi, Yi}ni=1) = arg max θ p(Y1, ..., Yn|X1, ..., Xn; θ) (1)
= arg max θ n∏ i=1 p(Yi|Xi; θ) = arg max θ n∑ i=1 log p(Yi|Xi; θ). (2)
This framework covers the most widely-used loss functions and tasks such as classification with cross-entropy and regression with square-error. The transition from (1) to (2) is justified not as a mathematical consequence but as a definition
p(Y1, . . . , Yn|X1, . . . , Xn; θ) := n∏ i=1 p(Yi|Xi; θ) (3)
chosen by the modeler based on the i.i.d.assumption about the dataset {(Xi, Yi)}ni=1. We note that this choice is not strictly necessary and that the modeler could use alternate factorizations. In the extreme case, the modeler could simply not factorize. Instead, the modeler could define and attempt learning the fully joint conditional distribution p(Y1, . . . , Yn|X1, . . . , Xn; θ) directly, which is a valid but impractical approach even if the modeler still believes the data is i.i.d. Such a model would be difficult to learn, difficult to use, and unlikely to generalize well. However, the possibility of an intermediate factorization between i.i.d. and fully joint learning is an intriguing and unexplored direction that is the main focus of this work.
The assumption (3) is motivated in part by the desire to learn the conditional distribution p(Yi|Xi; θ) for a single pair (Xi, Yi). In this work, we instead use a conditional distribution p(Yi, Yj |Xi, Xj ; θ) for two pairs (Xi, Yi) and (Xj , Yj) as our base model. We call this the self-joint model. Although our base model could learn conditional dependency between Yi and Yj given Xi and Xj , we will exclusively train p(Yi, Yj |Xi, Xj ; θ) using conditionally independent samples Yi|(Xi, Xj) and Yj |(Xj , Xj). We note that this conditional independence relation trivially holds under the i.i.d. assumption. In the context of classification, conditional independence means that the c × c prediction matrix h(Xi, Xj ; θ) defined by the ideal classifier p(Yi, Yj |Xi, Xj ; θ) should have the form h(Xi, Xj ; θ) = qiq ᵀ j , where qi, qj ∈ Rc represent the true conditional distribution q(Yi|Xi) and q(Yj |Xj), respectively (see Section 3.2 for full details). An essential difference between self-joint
learning and standard supervised learning is that our model can explicitly learn the sample-to-sample relation of pairwise conditional independence, instead of assuming this via the factorization in (3).
We can once again use Maximum Likelihood to learn our self-joint model by defining
θ̂SJ({Xi, Yi}ni=1) = arg max θ p(Y1, ..., Yn|X1, ..., Xn; θ) (4)
= arg max θ ∑ S∈S log ∏ (i,j)∈S p(Yi, Yj |Xi, Xj ; θ) (5)
= arg max θ ∑ S∈S ∑ (i,j)∈S log p(Yi, Yj |Xi, Xj ; θ), (6)
where S is a set of n2 arbitrary non-intersecting parings {i, j} for i, j ∈ {1, . . . , n} and S is the set of all such sets. This has the mild requirement that the number of samples n is even. For a given S, the factorization in (5) is again justified by the i.i.d. assumption across all observations, and we simply choose to cease factoring at pairs of two samples. Since all such factorizations are valid under the i.i.d. assumption, we choose to average the log likelihood over all possible factorizations when defining our learning objective, which under argmax is equivalent to summing. The ideal model will learn that ∏ (i,j)∈S p(Yi, Yj |Xi, Xj ; θ) remains constant across all S. In practice, we use a single S sampled uniformly from S and a random subset Sb of b pairs in S to obtain a stochastic gradient
∆̂SJ(θ) = ∑
(i,j)∈Sb
∂
∂θ log p(Yi, Yj |Xi, Xj ; θ). (7)
This gradient is unbiased for the gradient of the objective (6) as discussed in Appendix A.2.

heading: 3.2 LEARNING AND PREDICTING WITH SELF-JOINT CLASSIFIERS
text: In this section we will apply the self-joint supervision method to the image classification task. Let p(Yi, Yj |Xi, Xj ; θ) := h(Xi, Xj ; θ) be a classifier that maps two samples Xi, Xj ∈ Rm to a prediction for their corresponding joint label YiY ᵀ j ∈ Rc×c, where Yi, Yj ∈ Rc are the labels of Xi and Xj , respectively. Since Yi is a probability density, ||Yi||1 = 1 and 0 ≤ Yi,k ≤ 1 for each scalar component Yi,k of Yi, and the same applies to Yj . It is straightforward to show that the total loss
L(θ; {Xi, Yi}ni=1) = ∑ S∈S ∑ (i,j)∈S l(θ;Xi, Xj , Yi, Yj), (8)
where l(θ;Xi, Xj , Yi, Yj) = (YiY ᵀ j ) · log h(Xi, Xj ; θ) (9)
with the dot product taken in the sense of vectors and log taken elementwise, is equivalent to the MLE learning objective in (6) for cross-entropy loss, and that the stochastic gradient can optimize this loss similar to (7). See Appendix A.3 for details of this equivalence between (6) and (8).
Although we are learning a joint model p(Y, Y ′|X,X ′; θ), we still would like to make predictions for a single test image X via p(Y |X; θ) as in the standard i.i.d. framework. Note X and X ′ denote the first and second inputs of the model, respectively, and Y and Y ′ are their corresponding labels. This can naturally be accomplished by marginalization and Monte Carlo approximation:
p(Y |X; θ) = ∫ Rm c∑ k=1 p(Y, Y ′ = k|X,X ′; θ)q(X ′)dX ′ (10)
≈ 1 |B| ∑ i∈B c∑ k=1 p(Y, Y ′ = k|X,Xi; θ) = p̂(Y |X; θ), (11)
where B denotes a random subset of {1, . . . , n} with |B| giving its cardinality, and q(x) is the marginal of the true joint distribution q(x, y). Our model can only approximate p(Y |X; θ) instead of having a closed form solution because different samples B yield different predictors, but this estimate will be stable for large |B| due to the weak law of large numbers. In the case where |B| is moderately large, we empirically observe that predictive uncertainty estimated by either the variance of p̂(Y |X) or its entropy is more robust than the uncertainty calibration of a standard i.i.d. supervised model, which provides a compelling motivation for exploring the generalization properties of our framework.
Note that (11) has a computational complexity of O(|B|) and that p̂(Y |X; θ) requires additional samples Xi for estimation, which results in stochasticity. In our experiments, |B| is typically less than or equal to 50, which is a reasonable but non-trivial additional cost. To address these concerns, we note that we can use the same X for both input signals, yielding the deterministic classifier
p̃(Y |X; θ) = c∑
k=1
p(Y, Y ′ = k|X,X; θ). (12)
This is a valid estimate for (10) with |B| = 1 provided that X is in-distribution. Intuitively, in the Monte-Carlo inference in (11), the joint label is affected by the probabilistic label of both input samples. Generally, the two samples produce different levels of confidence and might belong to different classes, therefore, for a better approximation, we repeat our estimation of the prediction for the test sample several times. However, in (12) we know that the two sample inputs have the same label and seeing a replicate of a sample input does not add any new information about its label. Empirically, we observed that (12) often produces similar classification accuracy to (11).

heading: 3.3 INCORPORATING AUXILIARY DATA
text: Interestingly, conditional independency provides a principled tool to incorporate auxiliary data in the self-joint learning framework. Recall that semi-supervised learning (Chapelle et al., 2009; Grandvalet et al., 2005; Lee et al., 2013), self-supervised learning (Dosovitskiy et al., 2014; Oord et al., 2018), and OOD detection frameworks also typically take advantage of auxiliary data. Similarly, self-joint framework can benefit from un-labelled data. To this end, we can learn models such that the predictions on training data remain conditionally independent given an auxiliary unlabelled image. Following this approach, we effectively train self-joint models that can handle OOD data.
To incorporate auxiliary unlabelled data, we still assume that all data are generated in i.i.d. pairs (Xi, Yi), but for the auxiliary data we have unobserved labels which we denote by Yi = ∅. The factorization in (6) still holds under this assumption, but we are unable to learn the model without a numerical value if Yi = ∅. A straightforward estimate (similar to pseudo-labels to replace one-hot encoding) is
Ŷi = c∑ k=1 p(Yi, Yj = k|Xi, Xj ; θ), (13)
which is the marginalization of the predictor matrix h(Xi, Xj ; θ). The same principle applies for missing Yj by summing over the rows. This approach can even be used when both Yi = ∅ and Yj = ∅. When learning with auxiliary data, we replace any null Yi with the pseudo-label Ŷi in the objective function (6). Intuitively, auxiliary samples direct a self-joint model to preserve its predictions when it encounters such auxiliary samples. With the right choice of auxiliary data, it can improve generalization of the model. For example, if auxiliary data share similar background as original data, the model learns to ignore those superficial features and focus on salient foreground characteristics of the samples.

heading: 4 EXPERIMENTS
text: In this section, we evaluate the proposed framework on four visual classification tasks, i.e., CIFAR10, CIFAR-100, SVHN, and STL-10. The generalized self-joint training procedure is summarized in Algorithm 1 and we use this framework across all experiments. To make general conclusions, we refrain from hyperparameter optimization for any specific dataset and apply almost the same configuration (data augmentation, training recipe, and model architecture) for all datasets (details in the Supplementary Material). The overall results provide strong evidence that the new framework dramatically improves classification performance, adversarial and OOD data robustness, and facilitates training of large neural networks.

heading: 4.1 REGULARIZATION EFFECT OF SELF-JOINT FRAMEWORK
text: To demonstrate the regularization effect of our framework, we trained several wideResNet (Zagoruyko & Komodakis, 2016; He et al., 2016) models with different capacities on CIFAR-10 dataset in standard and self-joint frameworks. In this experiment, we increase the capacity by fixing the depth (34 layers) and increasing the width (number of channels) of the models. All the models utilize the same training regime (hyperparameters, augmentations, etc) for training. Figure 2 shows
the different classification accuracies for models with different levels of capacity in each framework. In this plot, due to overfitting, increasing capacity after a certain level harms the model performance in the standard supervised learning framework. However, self-joint framework, due to its strong regularization effect, is able to achieve increasingly higher levels of accuracy with higher capacity models.

heading: 4.2 MODEL ACCURACY AND CALIBRATION EXPERIMENTS
text: In this experiment, we compare trained models with similar architecture in the standard and selfjoint frameworks. We train the models in full and low training-sample regimes, where in the latter we reduce training data to 1% of the total data. Table 1 depicts the classification accuracy for different models with these two setups on CIFAR-10/100, SVHN, and STL-10. The results suggest that proposed method can effectively train large models that are able to achieve higher accuracy than standard supervised learning models in both experiments. This indicates strong regularization (Goodfellow et al., 2016) effect of the proposed framework.
Following common practice in model calibration, we report (Table 1) negative log-likelihood (NLL) as an indicator of the quality of predictive confidences. For more detail of this metric refer to (Guo et al., 2017). Unlike common practice in model calibration literature, we do not run any postprocessing step to calibrate models, but rather measure the raw model confidence values to compare how different techniques affect predictive confidence. In this experiment it is evident that self-joint models outperform standard models in model calibration. Moreover, interestingly, we observe while conventional supervised learning generally produce overconfident models, self-joint models tend to
produce under-confident predictions. We attribute this phenomenon to the capacity of the trained models, where increasing model depth usually enhances predictive confidence.

heading: 4.3 ADVERSARIAL ROBUSTNESS EXPERIMENTS
text: This section studies the adversarial robustness of self-joint models. Some early studies have proposed MC-dropout as a defense against adversarial attacks (Feinman et al., 2017; Rawat et al., 2017; Li & Gal, 2017). Although this defense has later been broken, its positive effect on adversarial robustness has been approved in (Carlini & Wagner, 2017). Therefore, we compare against MC-dropout models as our baseline. In these experiments we apply the gold standard of first-order attacks, PGD, in a white-box setting. To compute the gradients of stochastic model, we apply expectation over transformation (Athalye et al., 2018). In this method, given a stochastic classifier f(x; ρ), its gradient is calculated by: ∇XEρ[f(X; ρ)] = Eρ[∇Xf(X; ρ)] ≈ 1n ∑n i=1∇xf(X; ρi), where ρ denotes model randomness associated with drawing a sample {Xi1 , Xi2 , . . . , Xi|B|} according to the random indices {i1, . . . , i|B|} ⊂ {1, . . . , n} to get the estimate p̂(Y |X; θ) in (11). Table 2 shows the average performance over several runs of different models on 500 adversarial examples that are generated from random test samples with different levels of perturbation (more results reported in the Supplementary Material). We set the step size and the maximum number of iterations of PGD to 1/256 and 4, respectively. The stochastic models tend to outperform the rest of models for different levels of perturbation on every dataset. Nevertheless, it is clearly evident that the stochastic self-joint models (average of 20 samples) are a notch above their stochastic dropout counterparts in adversarial robustness, e.g. about 15% improvement on CIFAR-100, and consistently outperform all the other baselines.

heading: 4.4 DETECTING OOD DATA
text: Detecting OOD data is another important aspect of robustness. We apply two evaluation metrics to measure the OOD detection rate of a model. The area under the receiver operating characteristic curve (AUROC) is widely used in OOD detection literature. The AUROC is the expected probability that an in-distribution data will have a higher detection score than an OOD data. We also report detection error, which is the highest detection rate possible that is achieved by choosing the optimal threshold.
In this experiment, we include another neural network trained with outlier exposure (OE) (Hendrycks et al., 2018), which is a dedicated OOD detection. The OE utilizes an auxiliary OOD dataset to improve OOD robustness of the model. It specifically tries to make classifier produce a uniform posterior on OOD samples (for more detail refer to (Hendrycks et al., 2018)). We included OE as a representative of a strong OOD detection baseline and a method that incorporates
additional auxiliary data. We also train models with auxiliary data as explained in section (3.3), which obtain the best performance. For more information about these auxiliary datasets refer to the Supplementary Material.
The test set of SVHN, VOC-2012, CIFAR-10, CIFAR-100, CelebA, and STL-10 datasets is chosen for OOD datasets, where 500 samples are randomly selected (five repetitions) for in- and out-ofdistribution data to compare the detection rate of each model. Note that the majority of these datasets have little to no common classes, however STL-10 and CIFAR-10 share 80% of their classes and thus an ideal detection rate is far below 100% AUROC. We generally use maximum softmax probability (e.g. Hendrycks & Gimpel (2016)) as detection score, except for Monte-Carlo dropout models, where we apply entropy instead because we found that entropy is slightly superior in this case. For Monte-Carlo approximation we always compute average of 50 samples.
Table 3 shows the average results for all OOD datasets (detailed results are available in the Supplementary Material), where model stochasticity plays a crucial role in boosting OOD detection rate. Another important observation is the vital role of auxiliary data in improving detection rate of stochastic self-joint models. Interestingly, self-joint with auxiliary data generally outperforms outlier exposure, which is a dedicated state-of-the-art OOD detection technique.

heading: 4.5 IMPORTANCE OF JOINT DISTRIBUTION MODELING
text: We conduct a study to demonstrate the key role of joint distribution modeling in our framework. Here we train a model that jointly classifies two samples, simultaneously, but remove the joint output space. In particular, unlike self-joint model, this model performs the two predictions independently, where, the model includes 2× c output units for a c-classification problem. From another perspective, it is an ensemble of two classifiers with shared hidden units. During the test time, a sample is fed to both inputs simultaneously and the average of the two predictions are considered as the output. To have a fair comparison, we use the same WideResNet architecture as self-joint experiment except for the last layer that has 2 × c outputs. We observe that not only this ensemble does not help the performance, but also it causes a 2% and 3.5% decline in the classification accuracy of CIFAR-10 and CIFAR-100 datasets, respectively; even an inferior accuracy than standard supervised models. Moreover, these model show overconfident predictions. These findings clearly demonstrate the superiority of self-joint models over mere ensembling of models with shared parameters (more experiments in the Supplementary Material).

heading: 5 CONCLUSION
text: In this study, we proposed an alternative way to train and make inference by deep supervised models. Our framework directly learns sample-to-sample relation of conditional independence by modeling the joint conditional distribution of two samples. Our approach has several advantages over the standard supervised learning. First, it induces a strong regularization effect that can successfully train large deep models to the higher levels of accuracy than the standard models. Second, it can leverage extra set of unlabeled data, in conjunction with the training data, to improve the performance of the resulting model. Unlike self-supervised techniques, our approach works end-to-end, and, as a result learns features that are more suitable for the final learning task. Third, depending on the application, the trained models can infer in two modes: (1) a fast deterministic inference similar to standard models, and (2) a stochastic inference mode that performs more robust predictions. We demonstrated that this stochastic inference results in more robust predictions against adversarial and OOD samples. We hope this study will be a stepping stone towards more reliable learning models and inspires new avenues for future research.

heading: A APPENDIX
text: A.1 INDEPENDENCE, CONDITIONAL INDEPENDENCE, AND I.I.D.
Two events, A and B, are independent, denoted by A ⊥⊥ B, if: p(A,B) = p(A)p(B). (14)
Furthermore, assuming p(C) > 0, A and B are conditionally independent given C, denoted by A ⊥⊥ B|C, if:
p(A|B,C) = p(A|C). (15) Equivalently, Eq. 15 can be stated as:
p(A,B|C) = p(A|C)p(B|C). (16) Intuitively, two events are independent if they are not related, thus occurrence of one does not affect the other. In Eq. 15, two events A and B given C are conditionally independent if occurrence of B does not add any information about A above what we know based on C. As a special case of this view:
p(A,B|C,C) = p(A,B|C). (17) Note that neither independence relationship entails conditional independence nor conditional independence entails independence.
The assumption that the training data are independent and identically distributed, or i.i.d., is common to most learning frameworks. A dataset {Xi, Yi}ni=1 is called i.i.d. if (Xi, Yi) ∼ q for some fixed density q and the joint density q′ of the entire dataset satisfies q′(X1, Y1, X2, Y2 . . . , Xn, Yn) = q(X1, Y1)q(X2, Y2) · · · q(Xn, Yn). The relation of pairwise conditional independence Yi ⊥⊥ Yj |Xi, Xj follows from the independence of the joint distribution of (Xi, Yi) and (Xj , Yj) for all i, j:
q(Yi, Yj |Xi, Xj) = q(Yi, Yj , Xi, Xj) q(Xi, Xj) = q(Yi, Xi)q(Yj , Xj) q(Xi)q(Xj) = q(Yi|Xi)q(Yj |Xj).
This justifies the use of the joint label YiY ᵀ j as the target for p(Yi, Yj |Xi, Xj ; θ) in the self-joint framework, where we are overloading the notation and treating p(Yi, Yj |Xi, Xj ; θ) as a matrix rather than scalar in the classification example.
A.2 UNBIASEDNESS OF SELF-JOINT STOCHASTIC GRADIENT
In this section we show that the stochastic gradient (7) is unbiased for the gradient of the objective (6). We note
∂
∂θ ∑ S∈S ∑ (i,j)∈S log p(Yi, Yj |Xi, Xj ; θ) ∝ 1 |S | ∑ S∈S 1 |S| ∑ (i,j)∈S ∂ ∂θ log p(Yi, Yj |Xi, Xj ; θ)
= ES [ ES [ ∂
∂θ log p(Yi, Yj |Xi, Xj ; θ) ]] where the summations are expectations with respect to uniform distributions on S and S respectively. The unbiasedness of (7) simply follows from the fact that each sample S and (i, j) come from the same uniform distributions, and the fact that the mean of unbiased estimators is also unbiased.
A.3 EQUIVALENCE OF MAXIMUM LIKELIHOOD AND CROSS-ENTROPY LOSS FOR SELF-JOINT LEARNING
In this section we show that the self-joint cross-entropy loss (8) is equivalent to the Maximum Likelihood objective (6) in the case where the model h(X,X ′; θ) represents the conditional density p(Y, Y ′|X,X ′; θ). This simply follows from observing that
p(Y, Y ′|X,X ′; θ) = ∏ k,k′ [h(X,X ′; θ)] 1{Yk=1,Y ′k′=1} k,k′
which implies log p(Y, Y ′|X,X ′; θ) = Y (Y ′)t · log h(X,X ′; θ).
A.4 RELATIONSHIP TO BAYESIAN NEURAL NETWORKS A successful approach to estimate uncertainty is Bayesian neural network, where every parameter of the model has a distribution (MacKay, 1992; Barber & Bishop, 1998; Hinton & van Camp, 1993; Hernández-Lobato & Adams, 2015; Gal & Ghahramani, 2016). From a different perspective, these models can be thought as an ensemble of neural networks with shared parameters. Using maximum a posteriori method, these neural networks can readily produce uncertainty estimations. Furthermore, their principled training usually improves the overall performance of the model.
Despite these desirable properties, the application of Bayesian neural networks is very limited in real-world applications. This is mainly due to their excessive computational cost, both during inference and training phases, and their inability to appropriately scale up to large amount of data. Moreover, from a technical point of view, choice of the right prior distribution in these networks is not always clear. An important exception is MC-dropout (Gal & Ghahramani, 2016), which applies dropout to construct an ensemble of neural networks that approximate a Bayesian neural network and can be readily applied to almost any conventional deep neural network.
Much like a Bayesian neural network, a self-joint neural network can be regarded as an ensemble of neural networks with shared parameters. In this point of view, every reference sample implicitly generates one instance of parameters of the model, i.e. a neural network. In other words,
pZ(Y |X) = ∫ h(Z,X)dY ′, (18)
where Z is drawn from the marginal data distribution q(x). By fixing the second input, X in Eq. 18, of the self-joint model, h, and changing the first input, Z, Eq.18 is in effect generating an output distribution for p(Y |X), as well as generating a distribution over every parameter in the neural network.
From this perspective, self-joint framework introduces a different viable solution for producing scalable Bayesian neural networks.

Review summaries:
Paper propose a simple but effective method of learning. Feeding in a pair of data and learn the join conditional probabilities for predictions. The advantage is that the combination of data goes as the number of ways to partition the original data into pairs. This is a huge number. 

This paper introduces a pairwise loss function for supervised learning, named self-joint learning. Instead of predicting the conditional distribution of the label given one data sample as in conventional supervised learning, the proposed self-joint learning framework predicts the conditional joint label distribution of a pair of data samples. The paper proposes the corresponding constructions of the conditional joint label distribution and several strategies for making predictions in the inference phase. The paper shows empirical studies on dealing with overfitting, adversarial robustness, and detecting OOD data.

The Authors propose a new way of training classifier models. Instead of classifying each i.i.d. example independently during training and inference, they jointly classify a pair (X1, X2) of them, returning a joint distribution of labels (Y1, Y2). The loss function is modified so that the model learns that labels Y1 and Y2 are conditionally independent given X1 and X2.  This approach is shown experimentally to lead to superior model performance for the same model capacity (measured as relative layer width). The Authors attribute it to the regularising effect of new training method. Additionally, joint training on pairs of examples allows the use of auxiliary unlabelled examples.