Self-Joint Supervised Learning

1 INTRODUCTION . Neural networks lay at the heart of deep learning success . However , problems such as overfitting ( Weigend , 1994 ; Hawkins , 2004 ) and confidence miscalibration ( Guo et al. , 2017 ) persist across the vast majority of supervised models . Model calibration requires the predictive confidence of a classifier to be faithful to the ground truth , meaning the expected accuracy of all samples with m % confidence should be almost m % . However , deep models tend to produce overconfident predictions ( Guo et al. , 2017 ) , which undermines utilization of their confidence values . Consequences of miscalibration such as susceptibility to adversarial attacks ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Madry et al. , 2017 ; Akhtar & Mian , 2018 ; Akhtar et al. , 2021 ; Edraki et al. , 2021 ) and poor performance on out-of-distribution ( OOD ) examples ( Liang et al. , 2017 ; Hendrycks et al. , 2018 ) limit the utilization of deep learning systems in real-world applications . Existing tools to combat overfitting and miscalibration include artificially increasing training data through data augmentation ( Shorten & Khoshgoftaar , 2019 ) , using loss penalties , utilizing stochasticity to constrain a model ’ s ability to learn data idiosyncrasies ( Larsen & Hansen , 1994 ; Wan et al. , 2013 ; Gal & Ghahramani , 2016 ) , and smoothing labels to moderate overconfident predictions ( Szegedy et al. , 2016 ; Müller et al. , 2019 ) . While increasing the number of samples during training using data augmentation has been explored , an unexplored direction is to increase the number of samples used when making a prediction . Even a well-trained model could benefit from having additional data available at the time of inference , similar to a human expert that can still benefit from having a selection of reference examples when making a decision about a single test example . We hypothesize the problems of overfitting and miscalibration could be further exacerbated by limitations of the commonly used model p ( Y |X ; θ ) for a single sample-label pair ( X , Y ) and parameter θ . This model form only allows other training pairs ( X ′ , Y ′ ) to provide reference information for the conditional probability p ( Y |X ; θ ) , and therefore the model prediction for X , implicitly via training and not at all during inference . In this paper , we propose a new paradigm called self-joint supervision to construct models that make predictions using a given test signal and a randomly selected sample of reference signals ( typically a subset of training or validation pairs ) . This work focuses on image signals but the framework applies broadly . During training , a self-joint model learns to predict the joint distribution of a pair of labels given their associated images . By training with a pair instead of one sample , the model explicitly learns sample-to-sample relationships . In this work , we retain the standard i.i.d . assumption of supervised learning and our models actively learn the relationship of pairwise conditional independence ( see Appendix A.1 for a brief review of these concepts ) . Learning conditional independence is the key innovation of our work and the main difference from standard supervised learning , which has no explicit cross-sample modeling . We apply Maximum Likelihood to learn self-joint models in the similar framework as a standard model . After training , the model infers individual labels by marginalization of the joint distribution output . We propose inference procedures to produce both a stochastic and a deterministic output . Figure 1 shows an intuitive visualization of the self-joint training and inference procedures . Experiments with stochastic self-joint models provide evidence that learning conditional independence among sample pairs can lead to models more resistant to adversarial attacks and OOD data . Moreover , our method can naturally be extended to incorporate auxiliary unannotated data during training by using marginal prediction to generate pseudo-labels . Our experiments suggest adding partially annotated data can further safeguard against OOD data . Overall , our findings provide a novel approach to train more consistent and robust deep models and an alternative strategy to utilize auxiliary data . The main contributions of our work are summarized below . 1 . We introduce a novel perspective of self-joint learning where the model can accommodate sample dependency . By applying the model to pairs of i.i.d . data ( pairs are i.i.d . ) , we learn rather than assume conditional independence between labels given two input images . 2 . We show self-joint learning leads to substantial improvement in classification accuracy as the network capacity scales to sizes that were previously found to be detrimental due to overfitting . 3 . We construct a new type of stochastic models that can provide robust predictions for adversarial examples and detect OOD data . In particular , we explore the impact of our approach on model robustness against adversarial attacks and show it drastically improves adversarial robustness against small perturbations . 4 . We propose a novel technique to inject priors into learning models with the help of auxiliary unlabelled data . Our experiments show that utilizing auxiliary data can improve the model ’ s ability to recognize OOD data . 2 RELATED WORK . Out-of-Distribution ( OOD ) Detection . OOD data play a vital role in real-world applications , where the model commonly encounters samples from novel classes . Ideally , the model should produce low confidence predictions with high level of uncertainty for data from unknown classes . However , several studies suggest that deep models frequently make confident predictions for OOD data ( Kardan & Stanley , 2016 ; DeVries & Taylor , 2018 ) . It is now an active area of research to mitigate this drawback ( Li & Gal , 2017 ; DeVries & Taylor , 2018 ; Ren et al. , 2019 ; Chen et al. , 2020a ; Malinin & Gales , 2018 ; Kardan et al. , 2021 ) . In recent studies , the majority of dedicated techniques for OOD detection assume access to part of OOD data ( Liang et al. , 2017 ) , which makes OOD detection a new binary classification problem . However , in practice , this is an unrealistic assumption . In a different approach , outlier exposure ( Hendrycks et al. , 2018 ) advocates applying different distributions of OOD data for train and test and achieve substantial improvement over standard baseline . In this work we apply the same setup for evaluation of our OOD experiments with auxiliary data . Adversarial Examples . Adversarial examples , which use a deliberate minute perturbation of input samples that translates to significant modification of model ’ s prediction , are yet another shortcoming that complicates adoption of deep models in security-critical applications such as face recognition and autonomous driving vehicle systems . In addition , these samples point out an intrinsic difference between how inference occurs in humans and current deep models . Crafting a defense against adversarial attacks is an active area of research that witnessed a myriad of algorithms that were subsequently broken by more advanced attacks . A significant exception is adversarial training ( Madry et al. , 2017 ; Kannan et al. , 2018 ; Zhang et al. , 2019 ) , where the deep model is exposed to adversarial examples during the course of training to induce a degree of immunity against them . However , adversarially trained models suffer from performance degradation on clean ( unperturbed ) samples . Our proposed framework preserves classification performance on clean images , in addition to improving performance against adversarial examples by performing a robust prediction . Stochastic Neural Networks . Several studies show that applying stochastic models , such as Bayesian neural networks ( MacKay , 1992 ; Barber & Bishop , 1998 ; Hinton & van Camp , 1993 ; Hernández-Lobato & Adams , 2015 ; Gal & Ghahramani , 2016 ) , can mitigate overfitting and improve resistance against OOD and adversarial data . Yet , the main challenge with these approaches is scaling these models up to deep models with millions of parameters . Another technique to construct stochastic deep neural networks is Monte-Carlo dropout ( MC-dropout ) proposed by Gal & Ghahramani ( 2016 ) . This approach applies dropout ( Srivastava et al. , 2014 ) during both train and inference time . The Monte-Carlo sampling at inference time can produce reliable uncertainty estimations , therefore , MC-dropout has been successfully applied to detect OOD Li & Gal ( 2017 ) and adversarial examples Feinman et al . ( 2017 ) ; Rawat et al . ( 2017 ) ; Li & Gal ( 2017 ) . Similarly , self-joint models produce reliable predictions by performing a Monte-Carlo sampling during inference . Mutual Information for Representation Learning . Paired data has been widely applied for representation learning by mutual information ( Becker & Hinton , 1992 ; Dhillon et al. , 2003 ; Wang et al. , 2010 ; Friedman et al. , 2013 ) . In particular , paired data has been previously applied for coclustering ( Dhillon et al. , 2003 ; Wang et al. , 2010 ) . In deep networks , IMSAT ( Hu et al. , 2017 ) , DeepINFOMAX ( Hjelm et al. , 2018 ) , and ICC ( Ji et al. , 2019 ) pursue better features by maximizing information between different forms of representation of a pair of variations of the same sample . Although these approaches learn from a pair of images , unlike self-joint paradigm , they do not learn a joint distribution between the two samples in the pair ; They merely apply the same feature extractor to each sample . Furthermore , these approaches maximize mutual information between different instances ( augmentations ) of the same sample . Contrastive Learning for Representation Learning . There is a growing body of research , such as CPC ( Henaff , 2020 ) and SimCLRv2 ( Chen et al. , 2020b ) , that tries to learn representations from the similarity between a pair of samples based on the contrastive loss ( Tian et al. , 2019 ; Li et al. , 2020 ; Chen et al. , 2020b ) . These representations are later applied to a smaller set of samples for supervised fine-tuning . Given the abundance of unannotated data , recent studies ( Goyal et al. , 2021 ) show that this approach can surpass performance of supervised methods that have only access to a limited annotated dataset . Likewise contrastive learning , our framework learns better features based on a pair of samples . However , we learn an explicit joint distribution that can deal with both annotated and unannotated data , simultaneously , and can lead to more robust classifiers . 3 SELF-JOINT LEARNING FRAMEWORK . This section introduces our novel approach . We focus on the task of image classification , although our method applies broadly beyond image data and the classification task . We propose to learn a stochastic classifier by training a model on the joint distribution of pairs of data samples . We leverage this framework to produce both robust stochastic predictions and fast deterministic predictions , and to incorporate auxiliary unlabelled data . Throughout the paper , let ( Xi , Yi ) be a paired image random variable Xi and class ( label ) random variable Yi , and { ( Xi , Yi ) } ni=1 be the training dataset . We consider the case where the state space of the predictors Xi is a subset of Rm and the state space of responses Yi is a subset of Rc , where m is the number of image pixels times the number of color channels and c is the number of classes . At times we overload the notation of Yi and treat it as a member of the set { 1 , . . . , c } . Finally , we assume there is a true but unknown density q ( x , y ) such that ( Xi , Yi ) pairs are i.i.d . samples from q ( x , y ) for i = 1 , . . . , n. We emphasize that in this work we always assume that the observations { ( Xi , Yi ) } ni=1 are truly i.i.d . from q , even when we later introduce a model that can accommodate sample dependency .

Summary: The Authors propose a new way of training classifier models. Instead of classifying each i.i.d. example independently during training and inference, they jointly classify a pair (X1, X2) of them, returning a joint distribution of labels (Y1, Y2). The loss function is modified so that the model learns that labels Y1 and Y2 are conditionally independent given X1 and X2.  This approach is shown experimentally to lead to superior model performance for the same model capacity (measured as relative layer width). The Authors attribute it to the regularising effect of new training method. Additionally, joint training on pairs of examples allows the use of auxiliary unlabelled examples.