Causally Focused Convolutional Networks Through Minimal Human Guidance

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) are more popular than any other techniques in image classification . The ability to automatically extract required features is one key factor behind the phenomenal success of these models . Image classification being used in critical application areas such as medicine , surveillance , and many others , CNNs could make a huge impact in these domains . However , when implementing artificial intelligence based systems in such domains , attributing the success of the application to accuracy alone is not sufficient . In such cases , these systems are expected to be justifiable as the decisions made by them may have huge impact on various factors with high risks . Recently , it has been observed that CNNs are very much efficient to find correlation between features and labels and often extract features greedily following that principle ( Shwartz-Ziv & Tishby , 2017 ; Tishby & Zaslavsky , 2015 ; Chaitin , 2015 ; Blier & Ollivier , 2018 ) . In this process , often it may happen that these models learn correlations ( Shen et al. , 2017 ) which may not be justifiable from human perspective . In order to eliminate the effect of non causal correlations , CNNs need to be trained on huge datasets which may not be always possible in various domains like medicine . Thus learning the correct features efficiently from less data becomes an important problem in these areas . Let us illustrate this using an example . Suppose we have a dataset with images of cows on grasslands and aeroplanes in blue-sky . It has been observed that grass is extracted as a feature for cow and sky is extracted as a feature for aeroplane ( see Fig . 1 ) . We have used Grad-Cam ( Selvaraju et al. , 2017 ) to generate the heatmaps to visualize the features extracted by the CNNs . These heatmaps reveal that the model is using irrelevant features for classification . Possible solutions to overcome this issue would be to add more images to the dataset or to re-balance it in order to remove the data bias . Contrarily , our objective is to utilize the available data efficiently and to make the models learn features which can be causal from human perspective . It is evident that , CNNs are not guaranteed to extract such causal features . With this point of view , we propose to take guidance from humans on what they think is causal for a few samples in a class . We capture this guidance in the form of activation masks which are basically binary matrices with 1s on the causal parts of the images ( see Fig . 2 ) . Once we have the user guidance , our plan is to tweak the learning process of the CNNs though these guidance to focus them on extracting the causal features . We achieve that by modifying the learning objective of the CNNs and the backpropagation algorithm then takes care of updating the model parameters accordingly . This simple modification in the training procedure helps avoid the learning of spurious correlations between features and labels and focus just on the causal ones . We have experimentally observed that , this concept is working quite well on a wide range of cases and has proved to be extremely useful in the case of medical datasets . The main contributions of our work are summarized below : 1 . We propose a technique to focus the CNNs in learning causal features with the help of minimal human guidance . 2 . We demonstrate that our method not just improves learning of causal features but also helps in learning efficiently with less data . Additionally , we also show that the features learnt using our method are more robust to various types of image perturbations . 2 RELATED WORK . CNN and Interpretability Convolutional Neural Networks ( LeCun et al. , 1999 ) have boosted the progress in the field of computer vision since their inception . Manually designed architectures like LeNet ( LeCun et al. , 1989 ) , AlexNet ( Krizhevsky et al. , 2012 ) , VGG16 ( Simonyan & Zisserman , 2014 ) and many more have been proposed in the literature . In order to simplify the CNN architectures by retaining the spatial structure throughout the network , Springenberg et al . ( 2014 ) proposed the all convolutional nets , which eliminate the fully connected layers in these networks . To interpret the decisions of the CNNs , tools like Grad-Cam ( Selvaraju et al. , 2017 ) are widely used in practice , as they provide a way to extract the class discriminatory features learnt by the model . Correlation and Causality Research on the topics of correlation and causality has been gaining popularity among the researches in the recent years . Work by Shen et al . ( 2018 ) has recently shed some light on the correlational behavior of CNNs in image classification . Few other works like Arjovsky et al . ( 2019 ) , try to understand the causal relation between the input images and the corresponding labels , i.e . studying whether the relation is causal , anticausal or agnostic in nature . In our work , we just rely on the fact that there exist few features in the images which are the cause for the label and we expect the model to correctly identify such causal features only . The importance of causality , specifically in field of medicine is studied by Castro et al . ( 2020 ) and Liu et al . ( 2019 ) , highlighting the challenges for causality in computer aided diagnosis . Learning Causal Features The very recent work by Xiao et al . ( 2020 ) , studies the influence of the image background on object recognition . They show that non-trivial accuracy can be achieved by relying just on the background features in the images . A similar study was done in the case of medical images by Maguolo & Nanni ( 2021 ) , where they showed that CNN models provided diagnosis for the chest x-ray images even when the lung regions were removed from the input images . Not much work has been done on improving the learning of causal features , especially in the case of small datasets . The closest work that we found to our method is the self-supervised method called Guided Attention Inference Network ( GAIN ) ( Li et al. , 2019 ) which is proposed to improve the priors for the task of weakly supervised image segmentation . The authors present an extended version of this method , called the GAINpext , which uses an additional parameter sharing network with the GAIN architecture for pixel level supervision , that brings up the similarity with our work . We use this model as a baseline in our experiments . 3 PROPOSED METHODOLOGY : CAUSALLY FOCUSED CONVOLUTIONAL . NETWORKS ( CFCN ) In this section we describe the proposed method dubbed , Causally Focused Convolutional Networks ( CFCN ) . In CFCN , we force the model to break the spurious correlation between the label and any feature , and focus only on the causal features . In order to achieve this , we resort to minimal human guidance through activation masks . 3.1 NOTATIONS . Consider an input image X of size m1×m2 , its ground truth one-hot label y and the corresponding input activation mask A . Let , C be the number of classes in the dataset . The image classification model outputs class probabilities ŷ and the set of feature maps { Âf } Ff=1 generated by the last convolutional layer after application of relu activation , where F is the number of filters in this convolutional layer . The input activation masks and the feature map outputs are resized to a common shape n1 × n2 . Let c denote the index of the true class of the input image . A ◦ B denotes the element-wise product of two matrices A and B of same size . All the notations are summarized in Tab . 1 . 3.2 ACTIVATION MASK FOR HUMAN GUIDANCE . Activation masks are binary images where the causal regions are indicated using 1s and those belonging to the context regions are indicated using 0s . Some datasets like the Brain MRI ( Cheng et al. , 2015 ) , readily provide binary masks which can be directly used for our purpose . In few other cases , we may have pixel level labels which provide fine grained annotations exactly covering the regions of the objects of interest or bounding boxes annotations , that provide relatively coarse regions which may also contain few context features in them . Such annotations can be used to generate the activation masks as described in Appendix A . However , the method should not be dependent on the availability of masks or not . So we devise a simple technique to generate masks automatically . A typical step by step procedure for activation mask generation is shown in Fig . 2 . For a small subset of training images , the user has to roughly select the area of the objects of interest , which is then converted into a binary mask as shown in the figure . This can be automated using a python script . 3.3 CAUSAL FOCUS THROUGH ACTIVATION MASKS . In general CNNs are composed of several convolutional layers and pooling layers to extract features . The features extracted at the last layer then get passed to a Feed-forward Neural Network ( FNN ) to assign labels . Better the quality of features better will be the performance of the classifier . CNN layers along with the FNN layers are trained end to end by optimizing the categorical cross entropy loss . It has been observed that CNNs learn to extract features greedily and often end optimizing the correlation between labels and features . This process does not ensure that these models will always extract features which are causal from a human point of view . To mitigate this issue , we propose a mechanism to guide CNNs to focus on causal ( in a human eye ) features through additional minimal human input in the form of activation masks . During the model training we provide input images , their labels and activation masks . For the subset of input images which do not have the activation masks , we provide dummy masks with all values as 1s . The forward pass through the network for a single input image X , with true label y and input activation mask A generates the class probabilities ŷ and the feature map outputs { Âf } Ff=1 from the last convolutional layer , where F is the number of filters in this layer . Using this notation , we propose to optimize the following loss to train the CNN . L = − C∑ i=1 yilog ( ŷi ) ︸ ︷︷ ︸ Lcl + α 1− 1F F∑ f=1 n1∑ j=1 n2∑ k=1 ( A ◦ Âf ) j , k n1∑ j=1 n2∑ k=1 ( Âf ) j , k + , ︸ ︷︷ ︸ Lcf ( 1 ) where ≥ 0 is a small quantitiy to avoid accidental divide by zero error . α ≥ 0 is the trade-off parameter between the traditional categorical cross entropy loss ( Lcl ) and the proposed causallyfocus loss ( Lcf ) . Greater the value of α greater is the weightage on the causally-focus loss . Apart from the traditional CNNs , we also applied our causal feature learning method to the all convolutional nets proposed by Springenberg et al . ( 2014 ) . The number of filters in the last convolutional layer of these nets , is equal to the number of classes with each feature map output corresponding to each class , thus highlighting only that class specific features . We then calculate the causally-focus loss only with respect to the feature map Ac corresponding to the true class of the image and the input activation mask A as follows : L = − C∑ i=1 yilog ( ŷi ) ︸ ︷︷ ︸ Lcl +α 1− n1∑ j=1 n2∑ k=1 ( A ◦ Âc ) j , k n1∑ j=1 n2∑ k=1 ( Âc ) j , k + , ︸ ︷︷ ︸ Lcf ( 2 ) where c is the index corresponding to the actual class of the image , i.e . yc = 1 . This formulation has the ability to preserve the spatial structure of the data which is otherwise not maintained by the fully connected layers . Secondly , in Eq . 1 , we are calculating the causally-focus loss on all the feature map outputs which can be more time consuming . A detailed analysis of the loss function is presented in the Appendix B . The proposed approach CFCN is depicted in Fig . 3 .

Summary: This paper proposes a user-guided training for CNN to focus more on casual features which are based on human perspective and termed as casually focused convolutional neural network. Human guidance is used to make a rough binary segmentation mask of foreground objects/pixels.    For an input image, mask and label are both utilized to compute final loss. Final loss is composed of 1) activation loss which is computed using the binary mask and last convolution layer features and 2) cross-entropy loss which is calculated using predicted probability and class label of the input image. Images for which binary mask is not available, dummy mask is created as all foreground pixel. Two types of networks, all convolutional layers (CFCN-C) and fully-connected classifiers (CFCN-F) are utilized with the proposed methodology.  Experiments are performed on a total of 4 datasets of natural and medical images. Accuracy and visual activations are compared with baseline CNN, GAIN [1], and both CFCN-C and CFCN-F.