Combining Diverse Feature Priors

1 INTRODUCTION . The driving force behind deep learning ’ s success is its ability to automatically discover predictive features in complex high-dimensional datasets . In fact , these features can generalize beyond the specific task at hand , thus enabling models to transfer to other ( yet similar ) tasks ( Donahue et al. , 2014 ) . At the same time , the set of features that the model learns has a large impact on how well it will perform on unseen inputs , especially in the presence of distribution shift ( Ponce et al. , 2006 ; Torralba & Efros , 2011 ; Sagawa et al. , 2020 ) or spurious correlations ( Heinze-Deml & Meinshausen , 2017 ; Beery et al. , 2018 ; Meinshausen , 2018 ) . Motivated by this , recent work focuses on encouraging specific modes of behavior by preventing the models from relying on certain features . Examples include suppressing texture features ( Geirhos et al. , 2019 ; Wang et al. , 2019 ) , avoiding ` p-non-robust features ( Tsipras et al. , 2019 ; Engstrom et al. , 2019 ) , or utilizing different parts of the frequency spectrum ( Yin et al. , 2019 ) . At a high level , these methods can be thought of as ways of imposing a feature prior on the learning process , so as to bias the model towards acquiring features that generalize better . This makes the choice of the feature prior to impose a key design decision . The goal of this work is thus to explore the underlying design space of feature priors and , specifically , to understand : How can we effectively harness the diversity of feature priors ? OUR CONTRIBUTIONS . In this paper , we cast diverse feature priors as different perspectives on the data and study how they can complement each other . In particular , we aim to understand whether training with distinct priors result in models with non-overlapping failure modes and how such models can be combined to improve generalization . This is particularly relevant in settings where the data is unreliable— e.g , when the training data contains a spurious correlation . From this perspective , we focus our study on two priors that arise naturally in the context of image classification , shape and texture , and investigate the following : Feature diversity . We demonstrate that training models with diverse feature priors results in them making mistakes on different parts of the data distribution , even if they perform similarly in terms of overall accuracy . Further , one can harness this diversity to build model ensembles that are more accurate than those based on combining models which have the same feature prior . Combining feature priors on unlabeled data . When learning from unlabeled data , the choice of feature prior can be especially important . For strategies such as self-training , sub-optimal prediction rules learned from sparse labeled data can be reinforced when pseudo-labeling the unlabeled data . We show that , in such settings , we can leverage the diversity of feature priors to address these issues . By jointly training models with different feature priors on the unlabeled data through the framework of co-training Blum & Mitchell ( 1998 ) , we find that the models can correct each other ’ s mistakes to learn prediction rules that generalize better . Learning in the presence of spurious correlations . Finally , we want to understand whether combining diverse priors during training , as described above , can prevent models from relying on correlations that are spurious , i.e. , correlations that do not hold on the actual distribution of interest . To model such scenarios , we consider a setting where a spurious correlation is present in the training data but we also have access to ( unlabeled ) data where this correlation does not hold . In this setting , we find that co-training models with diverse feature priors can actually steer them away from such correlations and thus enable them to generalize to the underlying distribution . Overall , our findings highlight the potential of incorporating distinct feature priors into the training process . We believe that further work along this direction will lead us to models that generalize more reliably . 2 BACKGROUND : FEATURE PRIORS IN COMPUTER VISION . When learning from structurally complex data , such as images , relying on raw input features alone ( e.g. , pixels ) is not particularly useful . There has thus been a long line of work on extracting input patterns that can be more effective for prediction . While early approaches , such as SIFT ( Lowe , 1999 ) and HOG ( Dalal & Triggs , 2005 ) , leveraged hand-crafted features , these have been by now largely replaced by features that are automatically learned in an end-to-end fashion ( Krizhevsky , 2009 ; Ciregan et al. , 2012 ; Krizhevsky et al. , 2012 ) . Nevertheless , even when features are learned , model designers still tune their models to better suit a particular task via changes in the architecture or training methodology . Such modifications can be thought of as imposing feature priors , i.e. , priors that bias a model towards a particular set of features . One prominent example here are convolutional neural networks , which are biased towards learning a hierarchy of localized features Fukushima ( 1980 ) ; LeCun et al . ( 1989 ) . Indeed , such a convolutional prior can be quite powerful : it is sufficient to enable many image synthesis tasks without any training Ulyanov et al . ( 2017 ) . More recently , there has been work exploring the impact of explicitly restricting the set of features utilized by the model . For instance , Geirhos et al . ( 2019 ) demonstrate that training models on stylized inputs ( and hence suppressing texture information ) can improve model robustness to common corruptions . In a similar vein , Wang et al . ( 2019 ) penalize the predictive power of local features to learn shape-biased models that generalize better between image styles . A parallel line of work focuses on training models to be robust to small , worst-case input perturbations using , for example , adversarial training Goodfellow et al . ( 2015 ) ; Madry et al . ( 2018 ) or randomized smoothing ( Lecuyer et al. , 2019 ; Cohen et al. , 2019 ) . Such training biases these models away from non-robust features ( Tsipras et al. , 2019 ; Ilyas et al. , 2019 ; Engstrom et al. , 2019 ) , which tends to result in them being more aligned with human perception ( Tsipras et al. , 2019 ; Kaur et al. , 2019 ) , more resilient to certain input corruptions ( Ford et al. , 2019 ; Kireev et al. , 2021 ) , and better suited for transfer to downstream tasks Utrera et al . ( 2020 ) ; Salman et al . ( 2020 ) . 3 FEATURE PRIORS AS DIFFERENT PERSPECTIVES . As we discussed , the choice of feature prior can have a large effect on what features a model relies on and , by extension , on how well it generalizes to unseen inputs . In fact , one can view such priors as distinct perspectives on the data , capturing different information about the input . In this section , we provide evidence to support this view ; specifically , we examine a case study on a pair of feature priors that arise naturally in the context of image classification : shape and texture . 3.1 TRAINING SHAPE- AND TEXTURE-BIASED MODELS . In order to train shape- and texture-biased models , we either pre-process the model input or modify the model architecture as follows : Shape-biased models . To suppress texture information in the images , we pre-process our inputs by applying an edge detection algorithm . We consider two such canonical algorithms : the Canny algorithm Ding & Goshtasby ( 2001 ) which produces a binary edge mask , and the Sobel algorithm Sobel & Feldman ( 1968 ) which provide a softer edge detection , hence retaining some texture information ( see Figures 1b and 1c ) . Texture-biased models . To prevent the model from relying on the global structure of the image , we utilize a variant of the BagNet architecture Brendel & Bethge ( 2019 ) . This architecture deliberately limits the receptive field of the model , thus forcing it to rely on local features ( see Figure 1d ) . We visualize all of these priors in Figure 1 and provide implementation details in Appendix A . 3.2 DIVERSITY OF FEATURE-BIASED MODELS . After training models with shape and texture biases as outlined above , we evaluate whether these models indeed capture complementary information about the input . Specifically , we train models on a small subset ( 100 examples per class ) of the CIFAR-10 ( Krizhevsky , 2009 ) and STL-10 ( Coates et al. , 2011 ) datasets , and measure the correlation between which test examples they correctly classify . We find that pairs consisting of a shape-biased model and a texture-biased model ( i.e. , Canny and BagNet , or Sobel and BagNet ) indeed have the least correlated predictions—cf . Table 2 . In other words , the mistakes that these models make are more diverse than those made by identical models trained from different random initializations . At the same time , different shape-biased models ( Sobel and Canny ) are relatively well-correlated with each other , which corroborates the fact that models trained on similar features of the input are likely to make similar mistakes . Model ensembles . Having shown that training models with these feature priors results in diverse prediction rules , we examine if we can now combine them to improve our generalization . The canonical approach for doing so is to incorporate these models into an ensemble . We find that the diversity of models trained with different feature priors indeed directly translates into an improved performance when combining them into an ensemble—cf . Table 3 . In fact , we find that the performance of the ensemble is tightly connected to prediction similarity of its constituents ( as measured in Table 2 ) , i.e. , more diverse ensembles tend to perform better . For instance , the best ensemble for the STL-10 dataset is the one combining a shape-biased ( Canny ) and a texture-biased model ( BagNet ) which were the models with the least aligned predictions . 4 COMBINING DIVERSE PRIORS ON UNLABELED DATA . In the previous section , we saw that training models with different feature priors ( e.g. , shape- and texture-biased models ) can lead to prediction rules with less overlapping failure modes—which , in turn , can lead to more effective model ensembles . However , ensembles only combine model predictions post hoc and thus can not take advantage of diversity during the training process . In this section , we instead focus on utilizing diversity during training . Specifically , we will leverage the diversity introduced through feature priors in the context of self-training Lee et al . ( 2013 ) —a framework commonly used when the labeled data is insufficient to learn a well-generalizing model . This framework utilizes unlabeled data , which are then pseudo-labeled using an existing model and used for further training . While such methods can often improve the overall model performance , they suffer from a significant drawback : models tend to reinforce suboptimal prediction rules even when these rules do not generalize to the underlying distribution Arazo et al . ( 2020 ) . Our goal here is thus to leverage diverse feature priors to address this exact shortcoming . Specifically , we will jointly train models with different priors on the unlabeled data through the framework of co-training Blum & Mitchell ( 1998 ) . Since these models capture complementary information about the input ( cf . Table 2 ) , we expect them to correct each other ’ s mistakes and improve their prediction rules . As we will see in this section , this approach can indeed have a significant impact on the performance of the resulting model , outperforming ensembles that combine such models only at evaluation time—see summary in Figure 4 . Setup . We base our analysis on the CIFAR-10 and STL-10 datasets . Specifically , we treat a small fraction of the training set as labeled examples ( 100 examples per class ) , another fraction as our validation set for tuning hyperparameters ( 10 % of the total training examples ) , and the rest as unlabeled data . We report our results on the standard test set of each dataset . ( See Appendix A for experimental details , and Appendix B.6 for experiments with varying levels of labeled data . )

Summary: The paper presents multiple techniques for training models with different feature priors (i.e. inclinations to focus on different aspects of the training data) and combining them, either post hoc via ensembles or by allowing the models to provide augmented pseudo-labelled training data to each other via co-training. When using simple ensembling techniques, ensembles with a diversity of feature priors are show to perform better than ensembles where the individual models have similar feature priors. Co-training is shown to boost performance substantially when models with diverse feature priors supply pseudo-labels to each other. The problem domain is image classification. The feature priors concern shape and texture. Different preprocessing and or architecture constraint techniques are used for different models so as to predispose them to focus on shape but not texture or vice versa.