Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs

1 INTRODUCTION . Our ability to understand deep learning systems lags considerably behind our ability to obtain practical outcomes with them . A breadth of approaches have been developed in attempts to better understand deep learning systems and render them more comprehensible to humans ( Yosinski et al. , 2015 ; Bau et al. , 2017 ; Olah et al. , 2018 ; Hooker et al. , 2019 ) . Many of these approaches examine the properties of single neurons and treat them as representative of the networks in which they ’ re embedded ( Erhan et al. , 2009 ; Zeiler and Fergus , 2014 ; Karpathy et al. , 2016 ; Amjad et al. , 2018 ; Lillian et al. , 2018 ; Dhamdhere et al. , 2019 ; Olah et al. , 2020 ) . The selectivity of individual units ( i.e . the variability in a neuron ’ s responses across data classes or dimensions ) is one property that has been of particular interest to researchers trying to better understand deep neural networks ( DNNs ) ( Zhou et al. , 2015 ; Olah et al. , 2017 ; Morcos et al. , 2018b ; Zhou et al. , 2018 ; Meyes et al. , 2019 ; Na et al. , 2019 ; Zhou et al. , 2019 ; Rafegas et al. , 2019 ; Bau et al. , 2020 ) . This focus on individual neurons makes intuitive sense , as the tractable , semantic nature of selectivity is extremely alluring ; some measure of selectivity in individual units is often provided as an explanation of `` what '' a network is `` doing '' . One notable study highlighted a neuron selective for sentiment in an LSTM network trained on a word prediction task ( Radford et al. , 2017 ) . Another attributed visualizable , semantic features to the activity of individual neurons across GoogLeNet trained on ImageNet ( Olah et al. , 2017 ) . Both of these examples influenced many subsequent studies , demonstrating the widespread , intuitive appeal of `` selectivity '' ( Amjad et al. , 2018 ; Meyes et al. , 2019 ; Morcos et al. , 2018b ; Zhou et al. , 2015 ; 2018 ; Bau et al. , 2017 ; Karpathy et al. , 2016 ; Na et al. , 2019 ; Radford et al. , 2017 ; Rafegas et al. , 2019 ; Morcos et al. , 2018b ; Olah et al. , 2017 ; 2018 ; 2020 ) . ∗Work performed as part of the Facebook AI Residency Finding intuitive ways of representing the workings of DNNs is essential for making them understandable and accountable , but we must ensure that our approaches are based on meaningful properties of the system . Recent studies have begun to address this issue by investigating the relationships between selectivity and measures of network function such as generalization and robustness to perturbation ( Morcos et al. , 2018b ; Zhou et al. , 2018 ; Dalvi et al. , 2019 ) . Selectivity has also been used as the basis for targeted modulation of neural network function through individual units ( Bau et al. , 2019a ; b ) . However there is also growing evidence from experiments in both deep learning ( Fong and Vedaldi , 2018 ; Morcos et al. , 2018b ; Gale et al. , 2019 ; Donnelly and Roegiest , 2019 ) and neuroscience ( Leavitt et al. , 2017 ; Zylberberg , 2018 ; Insanally et al. , 2019 ) that single unit selectivity may not be as important as once thought . Previous studies examining the functional role of selectivity in DNNs have often measured how selectivity mediates the effects of ablating single units , or used indirect , correlational approaches that modulate selectivity indirectly ( e.g . batch norm ) ( Morcos et al. , 2018b ; Zhou et al. , 2018 ; Lillian et al. , 2018 ; Meyes et al. , 2019 ; Kanda et al. , 2020 ) . But single unit ablation in trained networks has two critical limitations : it can not address whether the presence of selectivity is beneficial , nor whether networks need to learn selectivity to function properly . It can only address the effect of removing a neuron from a network whose training process assumed the presence of that neuron . And even then , the observed effect might be misleading . For example , a property that is critical to network function may be replicated across multiple neurons . This redundancy means that ablating any one of these neurons would show little effect , and could thus lead to the erroneous conclusion that the examined property has little impact on network function . We were motivated by these issues to pursue a series of experiments investigating the causal importance of class selectivity in artificial neural networks . To do so , we introduced a term to the loss function that allows us to directly regularize for or against class selectivity , giving us a single knob to control class selectivity in the network . The selectivity regularizer sidesteps the limitations of single unit ablation and other indirect techniques , allowing us to conduct a series of experiments evaluating the causal impact of class selectivity on DNN performance . Our findings are as follows : • Performance can be improved by reducing class selectivity , suggesting that naturally-learned levels of class selectivity can be detrimental . Reducing class selectivity could improve test accuracy by over 2 % in ResNet18 and 1 % in ResNet50 trained on Tiny ImageNet . • Even when class selectivity isn ’ t detrimental to network function , it remains largely unnecessary . We reduced the mean class selectivity of units in ResNet20 trained on CIFAR10 by a factor of ∼2.5 with no impact on test accuracy , and by a factor of ∼20—nearly to a mean of 0—with only a 2 % change in test accuracy . • Our regularizer does not simply cause networks to preserve class-selectivity by rotating it off of unit-aligned axes ( i.e . by distributing selectivity linearly across units ) , but rather seems to suppress selectivity more generally , even when optimizing for high-selectivity basis sets . This demonstrates the viability of low-selectivity representations distributed across units . • We show that regularizing to increase class selectivity , even by small amounts , has significant negative effects on performance . Trained networks seem to be perched precariously at a performance cliff with regard to class selectivity . These results indicate that the levels of class selectivity learned by individual units in the absence of explicit regularization are at the limit of what will impair the network . Our findings collectively demonstrate that class selectivity in individual units is neither necessary nor sufficient for convolutional neural networks ( CNNs ) to perform image classification tasks , and in some cases can actually be detrimental . This alludes to the possibility of class selectivity regularization as a technique for improving CNN performance . More generally , our results encourage caution when focusing on the properties of single units as representative of the mechanisms by which CNNs function , and emphasize the importance of analyses that examine properties across neurons ( i.e . distributed representations ) . Most importantly , our results are a reminder to verify that the properties we do focus on are actually relevant to CNN function . 2 RELATED WORK . 2.1 SELECTIVITY IN DEEP LEARNING . Examining some form of selectivity in individual units constitutes the bedrock of many approaches to understanding DNNs . Sometimes the goal is simply to visualize selectivity , which has been pursued using a breadth of methods . These include identifying the input sample ( s ) ( e.g . images ) or sample subregions that maximally activate a given neuron ( Zhou et al. , 2015 ; Rafegas et al. , 2019 ) , and numerous optimization-based techniques for generating samples that maximize unit activations ( Erhan et al. , 2009 ; Zeiler and Fergus , 2014 ; Simonyan et al. , 2014 ; Yosinski et al. , 2015 ; Nguyen et al. , 2016 ; Olah et al. , 2017 ; 2018 ) . While the different methods for quantifying single unit selectivity are often conceptually quite similar ( measuring how variable are a neuron ’ s responses across different classes of data samples ) , they have been applied across a broad range of contexts ( Amjad et al. , 2018 ; Meyes et al. , 2019 ; Morcos et al. , 2018b ; Zhou et al. , 2015 ; 2018 ; Bau et al. , 2017 ; Karpathy et al. , 2016 ; Na et al. , 2019 ; Radford et al. , 2017 ; Rafegas et al. , 2019 ) . For example , Bau et al . ( 2017 ) quantified single unit selectivity for `` concepts '' ( as annotated by humans ) in networks trained for object and scene recognition . Olah et al . ( 2018 ; 2020 ) have pursued a research program examining single unit selectivity as a building block for understanding DNNs . And single units in models trained to solve natural language processing tasks have been found to exhibit selectivity for syntactical and semantic features ( Karpathy et al. , 2016 ; Na et al. , 2019 ) , of which the `` sentiment-selective neuron '' reported by Radford et al . ( 2017 ) is a particularly recognized example . The relationship between individual unit selectivity and various measures of DNN performance have been examined in prior studies , but the conclusions have not been concordant . Morcos et al . ( 2018b ) , using single unit ablation and other techniques , found that a network ’ s test set generalization is negatively correlated ( or uncorrelated ) with the class selectivity of its units , a finding replicated by Kanda et al . ( 2020 ) . In contrast , though Amjad et al . ( 2018 ) confirmed these results for single unit ablation , they also performed cumulative ablation analyses which suggested that selectivity is beneficial , suggesting that redundancy across units may make it difficult to interpret single unit ablation studies . In a follow-up study , Zhou et al . ( 2018 ) found that ablating class-selective units impairs classification accuracy for specific classes ( though interestingly , not always the same class the unit was selective for ) , but a compensatory increase in accuracy for other classes can often leave overall accuracy unaffected . Ukita ( 2018 ) found that orientation selectivity in individual units is correlated with generalization performance in convolutional neural networks ( CNNs ) , and that ablating highly orientation-selective units impairs classification accuracy more than ablating units with low orientation-selectivity . But while orientation selectivity and class selectivity can both be considered types of feature selectivity , orientation selectivity is far less abstract and focuses on specific properties of the image ( e.g. , oriented edges ) rather than semantically meaningful concepts and classes . Nevertheless , this study still demonstrates the importance of some types of selectivity . Results are also variable for models trained on NLP tasks . Dalvi et al . ( 2019 ) found that ablating units selective for linguistic features causes greater performance deficits than ablating less-selective units , while Donnelly and Roegiest ( 2019 ) found that ablating the `` sentiment neuron '' of Radford et al . ( 2017 ) has equivocal effects on performance . These findings seem challenging to reconcile . All of these studies examining class selectivity in single units are hamstrung by their reliance on single unit ablation , which could account for their conflicting results . As discussed earlier , single unit ablation can only address whether class selectivity affects performance in trained networks , and not whether individual units to need to learn class selectivity for optimal network function . And even then , the conclusions obtained from single neuron ablation analyses can be misleading due to redundancy across units ( Amjad et al. , 2018 ; Meyes et al. , 2019 ) .

Summary: This paper examines the impact of forcing units in a CNN to be more or less “class-selective” – i.e. respond preferentially to one image class compared to another.  The approach taken is to include a regularizer in the loss that directly penalizes or encourages class selectivity in individual units. They report that penalizing class selectivity at intermediate layers has little-to-no effect on classification performance, and in some cases mildly improves performance. They authors conclude that class selectivity is not an essential component of successful performance in CNNs, and that methods which use class selectivity to interpret CNNs should be approached with caution.