Causally Focused Convolutional Networks Through Minimal Human Guidance
Authors: Rimmon Saloman Bhosale
Mrinal Das
Venue: ICLR 2022 Submitted (2022)
Keywords: Causal Features
Convolutional Networks
Interpretability
Minimal Guidance
Computer Vision
Deep Learning

Abstract: Convolutional Neural Networks (CNNs) are the state of the art in image classification mainly due to their ability to automatically extract features from the images and in turn, achieve accuracy higher than any method in history. However, the flip side is, they are correlational models which aggressively learn features that highly correlate with the labels. Such features may not be causally related to the labels as per human cognition. For example, in a subset of images, cows can be on grassland, but classifying an image as cow based on the presence of grassland is incorrect. To marginalize out the effect of all possible contextual features we need to gather a huge training dataset, which is not always possible. Moreover, this prohibits the model to justify the decision. This issue has some serious implications in certain domains such as medicine, where the amount of data can be limited but the model is expected to justify its decisions. In order to mitigate this issue, our proposal is to focus CNN to extract features that are causal from a human perspective. We propose a mechanism to accept guidance from humans in the form of activation masks to modify the learning process of CNN. The amount of additional guidance can be small and can be easily formed. Through detailed analysis, we show that this method not only improves the learning of causal features but also helps in learning efficiently with less data. We demonstrate the effectiveness of our method against multiple datasets using quantitative as well as qualitative results.

Under review as a conference paper at ICLR 2022
CAUSALLY FOCUSED CONVOLUTIONAL NETWORKS THROUGH MINIMAL HUMAN GUIDANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional Neural Networks (CNNs) are the state of the art in image classifica- tion mainly due to their ability to automatically extract features from the images and in turn, achieve accuracy higher than any method in history. However, the flip side is, they are correlational models which aggressively learn features that highly correlate with the labels. Such features may not be causally related to the labels as per human cognition. For example, in a subset of images, cows can be on grassland, but classifying image as cow based the presence of grassland is incorrect. To marginalize out the effect of all possible contextual features we need gather huge training dataset, which is not always possible. Moreover, this pro- hibits the model to justify the decision. This issue has some serious implications in certain domains such medicine, where the amount of data can be limited but the model is expected to justify its decisions. In order to mitigate this issue, our proposal is to focus CNN to extract features that are causal from a human per- spective. We propose mechanism to accept guidance from humans in the form of activation masks to modify the learning process of CNN. The amount of addi- tional guidance can be small and can be easily formed. Through detailed analysis, we show that this method not only improves the learning of causal features but also helps in learning efficiently with less data. We demonstrate the effectiveness of our method against multiple datasets using quantitative as well as qualitative results.
INTRODUCTION
Convolutional Neural Networks (CNNs) are more popular than any other techniques in image clas- sification. The ability to automatically extract required features is one key factor behind the phe- nomenal success of these models. Image classification being used in critical application areas such as medicine, surveillance, and many others, CNNs could make a huge impact in these domains. However, when implementing artificial intelligence based systems in such domains, attributing the success of the application to accuracy alone is not sufficient. In such cases, these systems are ex- pected to be justifiable the decisions made by them may have huge impact on various factors with high risks.
Recently, it has been observed that CNNs are very much efficient find correlation between features and labels and often extract features greedily following that principle (Shwartz-Ziv & Tishby 2017: Tishby & Zaslavsky 2015 Chaitin 2015 Blier & Ollivier 2018). In this process, often it may happen that these models learn correlations (Shen et al.) (017)) which may not be justifiable from human perspective. In order eliminate the effect of non causal correlations, CNNs need to be trained on huge datasets which may not be always possible in various domains like medicine. Thus learning the correct features efficiently from less data becomes an important problem these areas.
Let us illustrate this using an example. Suppose we have dataset with images of cows on grasslands and aeroplanes in blue-sky. has been observed that grass is extracted as feature for cow and sky is extracted as a feature for aeroplane (see Fig. D. We have used Grad-Cam (Selvaraju et al. (017) to generate the heatmaps to visualize the features extracted by the CNNs. These heatmaps reveal that the model is using irrelevant features for classification. Possible solutions overcome this issue would be to add more images to the dataset or to re-balance it in order to remove the data bias. Contrarily, our objective is utilize the available data efficiently and make the models learn

---

Under review as a conference paper at ICLR 2022
Input
CNN
low
high
Figure 1: CNNs need not learn causal features. The examples are taken from the Pascal VOC 2012 dataset (first row) for aeroplane-cove classification in a biased dataset. In the second row we see the features learnt by the CNN highlighted using the Grad-Cam method. It is evident that the model has learnt features such as the sky and grassland for classifying aeroplane and cow respectively.
features which can be causal from human perspective. It is evident that, CNNs are not guaranteed to extract such causal featurex. With this point view, we propose to take guidance from humans on what they think is causal for few samples in class. We capture this guidance in the form of activation masks which are basically binary matrices with Is on the causal parts of the images (see Fig. 2 Once we have the user guidance, our plan is to tweak the learning process of the CNNs though these guidance to focus them on extracting the causal features. We achieve that by modifying the learning objective of the CNNs and the backpropagation algorithm then takes care of updating the model parameters accordingly. This simple modification in the training procedure helps avoid the learning of spurious correlations between features and labels and focus just the causal ones. We have experimentally observed that, this concept is working quite well on a wide range of cases and has proved to extremely useful in the case of medical datasets.
The main contributions of our work are summarized below:
1. We propose technique focus the CNNs in learning causal features with the help of minimal human guidance.
2. We demonstrate that our method not just improves learning of causal features but also helps in learning efficiently with less data. Additionally, we also show that the features learnt using our method are more robust to various types of image perturbations.
RELATED WORK
CNN and Interpretability Convolutional Neural Networks (LeCun et al. 1999) have boosted the progress in the field of computer vision since their inception. Manually designed architectures like LeNet (LeCun et al. 1989). AlexNet (Krizhevsky et al.) 2012). VGG16 (Simonyan & Zisserman, 2014) and many more have been proposed in the literature. In order to simplify the CNN architec- tures by retaining the spatial structure throughout the network, (Springenberg et al. (2014) proposed the all convolutional nets, which eliminate the fully connected layers these networks. To interpret the decisions of the CNNs, tools like Grad-Cam (Selvaraju et al., 2017) are widely used in practice, as they provide way extract the class discriminatory features learnt by the model.
Correlation and Causality Research on the topics of correlation and causality has been gaining popularity among the researches in the recent years. Work by [Shen et al.) (2018) has recently shed some light on the correlational behavior of CNNs in image classification. Few other works like Arjovsky et al. (2019). try to understand the causal relation between the input images and the cor- responding labels, i.e. studying whether the relation is causal, anticausal or agnostic in nature. In our work, we just rely on the fact that there exist few features in the images which are the cause for the label and we expect the model correctly identify such causal features only. The importance of causality, specifically in field of medicine is studied by Castro et al. (2020) and Liu et al. (2019), highlighting the challenges for causality in computer aided diagnosis.
2

---

Under review as a conference paper at ICLR 2022
Table 1: This table summarizes list of all the notations used in the paper.
Symbol
Description
Symbol
Description
Input image
C
Number of classes
y
One-hot class label
F
Number of feature map outputs Index true class
A
Input activation mask
C
Predicted class probabilities
kj X k2
Size of the filter
(A
Set of feature map outputs A single filter in the convolution layer
m1
X m2
Size of the input image Size of the activation masks and the feature maps
W
11 X 72
Ld
Classification loss
OF
Weight for the causally-focus loss
Lef
Causally-food loss
A very small constant value
Input Marked Boundary
Selected Area
Binary Mask
Figure 2: Activation mask creation: In the case of lack of annotated data, one can create activation mask for guidance very easily (this can be implemented in Python). The order is from left to right.
Learning Causal Features The very recent work by Xiao al. (2020), studies the influence of the image background on object recognition. They show that non trivill accuracy can be achieved by relying just on the background features in the images. similar study was done in the case of med- ical images by Maguolo & Nanni (2021), where they showed that CNN models provided diagnosis for the chest x-ray images even when the lung regions were removed from the input images. Not much work has been done on improving the learning of causal features, especially in the case of small datasets. The closest work that we found to our method is the self-supervised method called Guided Attention Inference Network (GAIN) (Li al. 2019) which is proposed to improve the pri- ors for the task of weakly supervised image segmentation, The authors present an extended version of this method, called the GAINP ext which uses additional parameter sharing network with the GAIN architecture for pixel level supervision, that brings up the similarity with our work. We use this model as a baseline in our experiments.
PROPOSED METHODOLOGY: CAUSALLY FOCUSED CONVOLUTIONAL. NETWORKS (CFCN)
In this section we describe the proposed method dubbed, Causally Focused Convolutional Networks (CFCN), In CFCN, we force the model to break the spurious correlation between the label and any feature, and focus only the causal features. In order to achieve this, resort to minimal human guidance through activation masks.
3.1 NOTATIONS
Consider an input image X of size my X m2, its ground truth one-hot label y and the corresponding input activation mask A. Let, C be the number of classes in the dataset. The image classifica- tion model outputs class probabilities and the set of feature maps (Åf) generated by the last convolutional layer after application of relu activation, where F' is the number of filters in this con- volutional layer. The input activation masks and the feature map outputs are resized to common shape 72- Let denote the index of the true class of the input image. A B denotes the element-we wise product of two matrices A and B same size. All the notations are summarized in Tab.
3

---

Under review as a conference paper at ICLR 2022
Feature Mass
Frace Pheppruu Sereoaiin
Prediction
"Deg"
Feature Extractor
Classifter
Feature
Categorical Loss
Activation
Loss
Combing Losses
Update Parameters
(images, labels, activation masks)
Feature Extraction
Feature
Output
Classifier
Layer
Mechanism
Layers
Layers
Figure 3: Illustration of the methodology: Along with images and labels, activation masks also input for guidance. Dotted lines denote the flow present only during the training phase.
3.2 ACTIVATION MASK FOR HUMAN GUIDANCE
Activation masks are binary images where the causal regions are indicated using Is and those be- longing to the context regions are indicated using Os. Some datasets like the Brain MRI (Cheng et al. 2015), readily provide binary masks which can be directly used for our purpose. In few other cases, may have pixel level labels which provide fine grained annotations exactly covering the gions of the objects of interest or bounding boxes annotations, that provide relatively coarse regions which may also contain few context features in them. Such annotations be used generate the activation masks as described in Appendix A
However, the method should not be dependent on the availability of masks or not. So we devise simple technique to generate masks automatically. A typical step by step procedure for activation mask generation is shown in Fig. For small subset of training images, the user has to roughly select the area of the objects of interest, which is then converted into binary mask as shown in the figure. This can be automated using a python script.
3.3 CAUSAL Focus THROUGH ACTIVATION MASKS
In general CNNs are composed of several convolutional layers and pooling layers to extract features. The features extracted at the last layer then get passed to a reed-forward Neural Network (FNN) to assign labels. Better the quality of features better will be the performance of the classifier. CNN layers along with the FNN layers are trained end to end by optimizing the categorical cross entropy loss.
has been observed that CNNs learn to extract features greedily and often end optimizing the correlation between labels and features. This process does not ensure that these models will always extract features which are causal from human point of view. To mitigate this issue, we propose a mechanism to guide CNNs to focus causal (in a human eye) features through additional minimal human input in the form of activation masks.
During the model training provide input images, their labels and activation masks. For the subset of input images which do not have the activation masks, we provide dummy masks with all values 1s. The forward pass through the network for single input image X, with label y and input activation mask A generates the class probabilities ý and the feature map outputs {A,)F= from the last convolutional layer, where F is the number of filters in this layer. Using this notation,
4

---

Under review as a conference paper at ICLR 2022
Table 2: Brief summary of the datasets used in the experiments.
Dataset
Size
Significance
Oxford IiIT Pets
7349
Large annotated dataset.
Aeroplane-Com Cow
718
Small and biased dataset created from "aeroplane" and 'cow' classes in Pascal
VOC 2012 dataset (Everingham et al..
Brain MRI
3064
Small medical dataset. Small medical dataset.
IDRiD
82
propose to optimize the following loss to train the CNN.
(Ao Ar)at
C
yilog(ūi)
a
n2 (A)) +
(1)
Det
Les
where c 2 O small quantitiy to avoid accidental divide by zero error. a 2 0 is the trade-off parameter between the traditional categorical cross entropy loss (Let) and the proposed causally- focus loss (Laf). Greater the value of a greater is the weightage on the causally-focus loss.
Apart from the traditional CNNs, we also applied causal feature learning method the all convo- lutional nets proposed by Springenberg et al. (2014). The number of filters in the last convolutional layer of these nets, equal the number of classes with each feature map output corresponding to each class, thus highlighting only that class specific features. We then calculate the causally- loss only with respect to the feature map A, corresponding the true class of the image and the input activation mask A as follows:
ng
(A Ac)s,N
C yolog (0:) +a 1
(Ac)i, +
Let
Les
where e is the index corresponding to the actual class of the image, i.c. ye = 1. This formulation has the ability to preserve the spatial structure of the data which is otherwise not maintained by the fully connected layers. Secondly, in Eq. [ we are calculating the causally-focus loss on all the feature map outputs which can be more time consuming- A detailed analysis of the loss function is presented in the Appendix B The proposed approach CFCN is depicted in Fig.
EXPERIMENTS
4.1 DATASET
We have particularly selected the following four datasets Oxford IIIT Pets (Parkh et al. 2012),
Aeroplane-Cow: Brain MRI (Cheng al. (015) and IDRiD (Porwal al.) (018)- for performance comparison with the baseline. These datasets, help in demonstrating the effectiveness of our method across different challenges such as biased data, small dataset size and feature extraction in medical images. The detailed description of these datasets is given in Tab.
4.2 BASELINES
In our experiments, we use the CNN traditionally trained using just the classification loss as our first baseline. As another related method that uses additional pixel level guidance, we use the extended version of GAIN (Li et al. 2019) as our second baseline. In comparison to these, we compare two that uses fully connected layers for classification (CFCN-F) and another that
of our models: uses the all convolutional nets (CFCN-C). We use the same architecture for feature extraction in all the methods for given dataset.

---

Under review as a conference paper at ICLR 2022
Input
Wilk WAN.NNIA
CFCN
This figure shows the counterpart heatmaps from CFCN for the images in Fig. [ We see that unlike CNN, CFCN is able focus on the causal features belonging to the aeroplanes and the cows present in the images.
Figure 4: Demonstrating the effectiveness of proposed CFCN:
Table 3: Quantitative results comparing proposed CFCN variants with the state of the We see that in the case of small datasets, CFCN models outperform the baselines in almost every metric. In the case of Oxford IIIT Pets dataset, CNN is performing slightly better than other methods but must be noted that high accuracy may also come from the spurious correlations in the features and the labels. This fact also evident as shown the visual results (Fig. 5
Metrics
Metrics
Models
Acc.
F1
ROC
Acc.
F1
ROC
Oxford IIII Pets
Aeroplane- Cow
(Small Dataset)
CNN
0.94
0.94
0.98
0.78
0.74
0.83
GAIN
0.86
0.84
0.92
0.72
0.70
0.74
CFCN-F
0.86
0.85
0.97
0.76
0.75
0.83
CFCN-C
0.90
0.88
0.96
0.82
0.81
0.86
Brain MRI
IDRiD
(Small Dataset)
(Small Dataset)
CNN
0.86
0.85
0.97
0.83
0.80
0.24
GAIN
0.66
0.56
0.83
0.80
0.84
0.82
CFCN-F
0.88
0.87
0.96
0.92
0.94
0.80
CFCN-C
0.88
0.87
0.97
0.88
0.90
0.71
4.3 EXPERIMENTAL SETTING
All experiments GPU. The codes are implemented in Python 3.7 with Tensorflow v2.2. We have used the 'matplotli'' library in Python to generate all the plots and used the 'polyfit' function available in 'numpy" library to regress curve in the plots wherever necessary. As preprocessing step, we normalize the input images in the range [0.1]. We evaluate the performance of all the models using five quantitative metrics: accuracy, macro fl-score and AU-ROC. Further we present the qualitative results using Grad-Cam for heatmap visualization. A detailed description of the experimental setup for each dataset is given in the Appendix
GPU system with 16 GB RAM and a single GeForce RTX 2080
4.4 RESULTS: BEYOND CNNS TO DETECT CAUSAL FEATURES
Recall from Fig. that CNNs need not learn causal features as optimizes over correlation. Whereas, with additional human guidance. CFCNs are able ignore the spurious correlations present in the data and learn just the causal features in the images as shown in Fig. 4
Tab. 3 compares the performance of various models several metrics. It can be seen that both CFCN variants perform quiet well. Furthermore, we see that, in the case of very small datasets (Aeroplane-Cow: Brain MRI and IDRiD), models outperform the baselines in almost every met- ric. Further, to validate the correctness of our models in terms of feature learning, we present the comparison of the Grad-Cam heatmaps generated by these models in Fig. 5 We observe that CFCNs

---

Under review as a conference paper at ICLR 2022
Input
GAIN
CFCN-F
Figure 5: Comparison of models qualitative results: CFCN variants learn causal features
whereas the features learnt by baselines are not causal.
Oxford IIIT Pets Dataset
Brain MRI Dataset
Metric Value
Metric Value
Accuracy
Accuracy
Classification Loss
Classification Loss
Causally-Focus toss
Causally Focus Loss
Epoch Number
Epoch Number
Figure 6: Convergence: Causally-focur loss does affect the convergence in training CFCNs.
are not just accurate but also rely on the class discriminatory causal features in the images for classi- fication rather than using any context information. We share few additional results in the Appendix D
The Convergence of CFCN. We also empirically verify that adding the causally-focu loss does not affect the training of the underlying CNN. Fig. shows steady convergence based on both the loss functions.
4.5 RESULTS: EFFECT OF VARYING THE WEIGHTAGE FOR CAUSALLY-FOCUS Loss
As we have seen that, CFCN is able to detect causal features and also perform better or quite similar the baselines, we want to investigate the sensitivity of the setup of human guidance.
The trade-off parameter a is an important quantity as it trades-off between the classification loss and causally-focus loss (see Eq. [ and Eq. (2). We expect that as the value of increases, the model should focus more and more on the causal features. We see the expected behaviour in Fig. Further, we have observed that beyond a certain threshold, the activations become very small there-by affecting the classification performance of the models.
7
4.6 RESULTS: LEARN MORE WITH LESS DATA DUE TO CAUSALITY
Through this experiment we demonstrate that our method helps not just in learning causal features but also helps in learning faster with less data. The basic idea is, with more guidance, reduce the

---

Under review as a conference paper at ICLR 2022
Input
0
0.2
0.5
1.0
2.0
Figure 7: Illustration of effect of the trade-off parameter (a) in the Loss: We that, with increase in the value of a, the model is able to focus more on the diseriiiscrm causal feature in the input images.
0.5
AM Percent
Test Accuracy
Combined Loss
00.00%
85%
08.57%
0.75
14.28%
8.00
645
Train Dataset Size
Train Dataset
Figure 8: Illustration of the effect of human guidance in the form of activation mask: As increase the amount of human guidance in the form of activation masks accuracy increases (left), and loss reduces greater rate with the increase in the training dataset size (in æ-axis, unit equals to 1000 samples). One observation is that, with additional guidance the models are able to learn efficiently with less data.
uncertainties in the model for learning causal features as the model explicitly guided to focus on the features which are effective to discriminate.
To demonstrate this, we trained the CNN models with same architectures by varying the training dataset size in steps of 1000, from 1000 to 7000 training samples. In each case, we train models with 0, 200, 600 and 1000 activation masks as input. So basically we train 28 (7 (7)4) models in total. For given dataset size expect the models with more activation masks input to have higher accuracy than those with less number of activation masks as input. We share the results in Fig. B in the form of two plots. To understand the average behavior of the models we present the curves which we regressed through the points representing the metric values for each model.
The first plot shows comparison models on the basis of test accuracy. We see that the curve for the models with higher number of activation masks as input dominates over the curves of the models with less number of activation masks as input in almost all the train dataset size cases. This shows that the model is able to learn more with the same amount of data when it is provided with additional information as described in our approach.
The second plot considers the average of categorical cross entropy loss and the causally-focus loss as a metric for model comparison. This metric captures the correctness of the models both in terms of classification and causal feature extraction. We see that the models with more activation masks as input have lower loss than the models with less number of activation masks as input. We can also see that the lowest loss attained using 7000 training samples by the models with no activation masks as input is also attained by the model with just 200 activation masks as input using just 3000 to 4000 training samples. This confirms that, with more guidance the models are able to learn faster from less amount of data.
8

---

Under review as a conference paper at ICLR 2022
Input
EB
CNN
CFCN
aa)
Vary Background
Input
CNNN
CFCN
(b) Vary Brightness
Figure 9: Robust feature learning: The above figures show the robustness of feature learning in CFCNs against changes in the background and the brightness the input images.
4.7 RESULTS: ROBUSTNESS AGAINST ADVERSARIAL PERTURBATION OF IMAGES
Finally, we present another interesting experiment highlighting the robustness of the features learnt by the CFCN models in comparison to that of traditionally trained CNNs with the same architecture and identical experimental setup. We vary the background for the objects of interest and check how the two models perform in terms of visual results. We expect that in comparison to the traditional CNNs, the features learnt by CFCNs would be more stable across such image perturbations. As shown in Fig. a). of the images. This further confirms the effectiveness of our method in learning causal features rather than relying of the context features for classification.
that the features learnt by CFCN are more robust to change in the context
Similarly, we also conducted another experiment in this direction by varying the brightness of the input images. As shown in Fig. 9)), we see that the features learnt by CFCNs more robust in comparison to that of CNNs. for different brightness values of the input images.
5 CONCLUSION
In this paper we demonstrate that, through minimal human guidance it is possible to go beyond the traditional CNN architectures avoid overestimating spurious correlations between labels and contextual features. Using such guidance in the form of activation masks, the proposed model CFCN is able detect causal features in the images. This ability turn helps CFCN not only to achieve higher accuracy with less data but also to be robust against several adversarial changes in images.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization.
arXiv preprint arXiv:1907 0000002999, 2019.
Léonard Blier and Yann Ollivier. The description length of deep learning models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett
(eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL htttps::/ hew.reeoooo...n................. newrips...p.... https/scpppppee/////////// 3b712de 32333 -aapaasseeeeeeeepppp-e.e pdf
Daniel C Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communications, 1(1):1-10, 2020.
Gregory Chaitin. On the intelligibilityy of the universe and the notions of simplicity, complexity, and irreducibility. Akademie Verlag, 2015.
Jun Cheng, Wei Huang, Shuangliang Cao, Ru Yang. Wei Yang, Zhaoqiang Yun, Zhijian Wang, and Qianjin Feng. Enhanced performance of brain tumor classification via tumor region augmentation and partition. PloS one, 10(10):c0140381, 2015.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.
The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/chan http://hhlaaaaasooooooo222220///
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Yann LeCun, Berhhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, (4)::41-555) 1989.
Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient- based learning. In Shape, contour and grouping in computer vision, pp. 319-345. Springer, 1999.
Kunpeng Li, Ziyan Wu, Kuan- Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Guided attention inference network. IEEE transactions on pattern analysis and machine intelligence, 42(12)0996--0000 2019.
Yuanyuan Liu, Zhouxuan Li, Qiyang Ge, Nan Lin, and Momiao Xiong. Deep feature selection and causal analysis of alzheimer's 's disease. Frontiers in neuroscience, 13:1198, 2019.
Gianluca Maguolo and Loris Nanni. A critic evaluation of methods for covid-19 automatic detection from x-ray images. Information Fusion, 76:1-7, 2021.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 20/2 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sa- hasrabuddhe, and Fabrice Meriaudeau. Indian diabetic retinopathy image dataset (idrid), 2018.
URL httthhts:://aat //dx http:///w..doo.o//....//.0.//
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618-626, 2017.
doi: 10. 10.100//CC..201...44
Zheyan Shen, Peng Cui, Kun Kuang, and Bo Li. image classification: Correlation vs causality. arXiv preprint arXiv: 07656, 2017.
Zheyan Shen, Peng Cui, Kun Kuang, Bo Li, and Peixuan Chen. Causally regularized learning with agnostic data selection bias. In Proceedings of the 26th ACM international conference on Multimedia, pp. 41 1-499, 2018.
10

---

Under review as a conference paper at ICLR 2022
Ravid Shwartz-Ziv Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint
2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 14X2:664/266806, 2014.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint wrXiv:2006.9994, 2020.

---

Under review as a conference paper at ICLR 2022
A ACTIVATION MASK GENERATION
For manual activation mask generation, we have used the impl_interactions library available in Python. As shown in Fig2] a user has to select the regions interest the given image which is then automatically converted into binary activation mask. These masks can be then stored along with the datasets for later use.
Further, in order to utilize the available annotations for activation mask generation, the pixel level labels provided the image segmentation datasets or the bounding box annotations provided in the object localization datasets. When using pixel level annotations, the pixel values representing the objects of interest can be set to Is and rest other pixels can be set to Os. Similarly, when using bounding box annotations, the pixels withing the bounding box regions need be to Is while all other pixels outside the box regions need to be set to Os. Few examples of such conversions are shown Fig. 10
Input
Available
Masks
Generated
Masks
Figure 10: Obtaining the activation masks: The following table shows some examples of ob- taining activation masks from different annotations available for some aligned tasks such as image segmentation and object localization. First row in the figure shows the input images, second row shows their corresponding available annotations and third row shows the activation masks generated from these annotations. As in the case of Brain MRI dataset, binary masks are readily available, which can be directly used for our purpose.
B ANALYSING THE LOss FUNCTION
We present the Eq. again for the purpose analysis.
Σ
(A
45)
F
Σ gilog(is)
L =
+ a
1
F
(Ar)ss +c
f=1
Teel
The first term in the loss represents the standard categorical cross entropy loss (La).. The second term consists of the loss weight a and the causally-focus loss (Iee)). this section we elaborate more the second loss.
Intuitively, the causally-focus loss calculates the percentage of the activation values present in output feature maps which are active in the context region of the images. More the activations present in the context regions, more will be the loss. In the optimal case, the model will have learnt only the causal features, in which case the loss will be equal to 0. which is the lowest. Contrarily, the worst case, only the features outside the causal regions marked by the activation map will be active, making the loss equal 1, which is the highest.
12

---

Under review as a conference paper at ICLR 2022
When using the CFCN for multi-label classification, the loss needs some modifications. An updated version of the loss for multi-label classification is given by Eq. 3 Basically, in the case of multi- label classification, each image will have 'C" activation mask, one each corresponding one of the C' classes in the network. Thus, it also requires the last convolutional layer to have number of filters to be equal to the number of classes in the dataset. This requirement is already satisfied in the case of the 'all convolutional nets'. Each filter corresponds to class in the dataset. We calculate the causally-focur loss with respect to each feature map output and the corresponding activation mask for that class, as depicted in Eq.
n2 [ Ar)a.
Σ
L= Σ y:oog(úi)
1
(3)
Σ (Ar)i, +e
C
DETAILED EXPERIMENTAL SETUP
Table 4: Detailed experimental setup for each dataset.
Sr.
Dataset
Filters in conv. layers
DO
BS
LR
Epochs
1.
Oxford IIIT Pets
128,128,64, 64,36..3......
0.3
64
le-4
500
2.
Aeroplane- Cow
128, 128, 64,64,32 64.64,64.32..2..22
0.3
16
le-4
200
1.5
3.
Brain MRI
32
le-5
150
3
4. IDRiD
32,32,16,16.88
0.3
5
le-4
200
0.8
In this section we describe the detailed experimental setup used for training different models on each dataset. We have implemented all the models using Tensorflow v2.2. In all the experiments, we used custom CNN architectures for each dataset. Further, for each dataset, all the experiments were run by maintaining an identical experimental setup for all the models. Input images from all the models except the IDRiD dataset were resized to the shape 96x96 while those belonging to the IDRiD dataset were resized the shape 250 175. The number of filters used in each convolutional layers of these architectures is given in Tab. 4 Every convolutional layer is followed by batch normalization layer and later by dropout layer (except for the Brain MRI dataset). We use dO) value of 0.3. The last dropout layer is followed by a convolutional layer with 1x1 filter size and 16 filters in the case of CFCN-F models and "C" filters in the case of CFCN-C, where "C" is the number of classes in the given dataset. As I IDRiD is multi-label classification dataset, the Nimes convolutional layer in this case has filters, for both CFCN-F and CFCN-C. The 1x convolutional layer is then followed by flatten layer in the of CFCN-F, while for the CFCN- models, we use global average pooling. In the case of CFCN-F the flatten layer is followed by the output layer which is fully connected layer with C neurons in it. In CFCN-C, global average pooling layer acts as the output layer. We have used 'relu' activation in all the intermediate layers. For output layer, we use 'softmax" activation in the case multi-class classification and 'sigmoid' in the case of multi-label classification. Details about the batch size (BS), learning rate (LR), number of epochs and the value for loss weight parameter introduced in Eq. and 2 for each dataset given in Tab.
For the Oxford !!IT Pets dataset and the IDRiD dataset, we do not down-sample the input images throughout the network. For the deroplane-Cov and Brain MRI datasets, we down-sample the images by using stride of every second and every third convolutional layer in the respective dataset case. While calculating the causally focus loss, we can resize the input activation masks from input size to the size of the output feature maps. Conversely, we can also resize the output feature maps to the input size. In case of the Oxford IIIT Pets and IDRiD, the input activation masks and the output feature maps have the same size as the input (ie. (m1 X m2) = (n1 x n2))), so no resize operation is required. For the Aeroplane- Cow dataset, resize the output activation mask to the input size while for Brain MRI dataset, we resize the input activation masks to the size of output feature maps.
13

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL RESULTS
D.1 BENEFITS IN VISUAL EXPLANATION
In this subsection, we present the benefits of using the all convolutional nets. As discusses earlier, the all convolutional nets are known for their architectural simplicity and their ability to retain the spatial structure of the images throughout the network. Additionally, the last convolutional layer of these nets generate feature maps, each of which corresponds to one particular class in the dataset. Thus, each feature map retains the features in the input images which are similar to those of the images belonging to the class represented by that feature map. This fact is clearly highlighted in Fig. We see that, for the first image of digit '0', the most similar digits are the digit '2' and digit '6". Similarly, for the second image of imput digit '9', the closest digits are '1', '4' and '7'.
Due, to this explanations similar to the Grad-Cam are obtained in just the single forward pass through the network, thereby eliminating the need of back-propagating through the network for gradient calculations. When using CFCN with the all convolutional nets, the additional benefit is that, each feature map output highlights only the class-specific causal features present in the images, thereby enhancing the explainability of these models.
Feature Maps
(a) Feature maps for input digit '0'
Feature Maps
(b) Feature maps for input digit '9'
Figure 11: Enhanced visual explanations: This figure demonstrates the benefits of all convolu- tional nets in terms of visual explanations. The feature map outputs for the input digits (a) 6 and (b) 9 are given. The images represent the classes 0 to 9 row major order from left right.
14

---

Under review as a conference paper at ICLR 2022
D.2 ADDITIONAL TRAINING PLOTS
In Fig. 12 we present the training plots for the remaining two dataset: Aeroplane-Cow and IDRiD. These plots also confirm empirically that the modified loss in CFCN attains convergence after specified number of epochs.
Aeroplane-Cow Dataset
IDRiD Dataset
Metric Value
Metric Value
Accuracy
Accuracy
Classification
Classification
Causally Facus Loss
Causally- Saally-Focuu Focus Loss
wo
125
150
200
300
125
150
175
200
Epoch Number
Epoch Number
Figure 12: Convergence: Causally-focus loss does not affect the convergence in training CNNs. Demonstration using the training plots for Acroplane-Cow and IDRiD datasets.
D.3 COMPARISON THE CAUSALLY-FOCUS LOsS ON THE BENCHMARKS
Normalized Causally Focus Loss
10-
0.8
0.6
04-
['Oxford
['Arro Cow']
02
['Brain MRF]
[DDRDD']
ONN
CAN
OTCN
CTORT
Method
Figure 13: Comparing the causally-focus loss: In the above plot, we compare the values of the causally-focus loss for different models on the benchmark datasets. We observe that, our models perform better than baselines in all the dataset cases.
In Fig. we present the comparison of the causally-fooe loss for different models on different datasets. The æ-axis represents the models while the y-axis represents the loss values. For better visualization, we have performed min-max normalization over the causally-focus losses of the mod- els for given dataset. We observe that the causally-focus loss is the lowest mostly in the case of CFCN-F models. Also, the CFCN-C models perform more or less similar to these models. This further confirms that our method is successful in eliminating the spurious correlations based on the context features, thereby providing reliable predictions.
15

---

Under review as a conference paper at ICLR 2022
D.4 CAUSAL FEATURE LEARNING IN CFCN
In Fig. 14, we present few additional results comparing the heatmaps generated by the traditionally trained CNN and the CFCN the Bike-Helmets dataset. This dataset consists of about 764 images in total, of which we have used 500 samples for testing.
Input
CNN
CFON
Input
CNNN
CFCN
Figure 14: Additional qualitative results presenting the Grad-Cam heatmaps generated for the Bike- Helmets dataset provided by Make-ML.
16

---

Under review as a conference paper at ICLR 2022
D.5
VARY BACKGROUND
Input
CNNN
CFCN
Input
CNN
CFCN
9
Figure 15: Additional visual results comparing the robustness of features learnt by CNN and CFCN for the vary-background experiment. The results show that CFCN consistently outperforms CNN in terms of robust feature learning.
we present few more results comparing the traditionally trained CNNs and the CFCN models for the sacy-background experiment. We observe that CFCN are able to extract the class discriminatory causal features in the images in all the cases and it is least affected by the changes in the context regions of the input images.
In Fig. I5
17


Reviews:
[{"review_id": "qjD9KBeSAiq", "reviewer": "Reviewer_HtjX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1> The proposed method is very simple to understand and adapt. \n\n2> A small percent of images with binary mask labels can be used to make this work as shown in experiments.\n\n3> The proposed method can be beneficial for application areas such as medical imaging where “why” matters equally as “what”.\n  \nWeakness:\n\n1> Method requires some user intervention which might be possible for simple applications as natural images but not possible without expertise in some applications such as medical. \n\n2> Sometimes relative features are also important such as the correlation between background and foreground. For example, a foreground with a completely irrelevant background does not make sense (a cat flying in outer space).\n\n3> Authors have experimented with near-to-good masks but in reality, masks could be noisy (partially or completely wrong). No study is done.\n\n4> The proposed method is closely related to explainable artificial intelligence (XAI) methods but there is no comparison with XAI method. \n\n5> Authors claim for the proposed method to work better on the small dataset: A small dataset with less number of classes is not very difficult for a model to learn. The model difficulty associated with dataset difficulty such as a high number of classes with a small number of images in each class, intra-class variations, inter-class similarity, etc. are factors of dataset difficulty, not just small dataset. I believe the degraded performance on the Oxford pet dataset is because of dataset difficulty with a large number of classes and similarity between different classes. \n\nComments and suggestions:\n\n1> Authors Could try with soft masks instead of hard binary masks for background relation factor and compare the results.\n\n2> Also some noisy masks ablation studies will be interesting to see. \n\n3> It is not clear from the paper that if all the classes (at least a few images per class) in the dataset have binary masks for training or some classes have zeros masked training. It will be good to see if some classes are left with zero masks, how they perform during testing and how they adapt knowledge from other-classes mask training. \n\n3> Intrinsic XAI models can be good candidates to check model performance and activation/heatmaps for better comparison. \n\n4> Authors should try the proposed method on a difficult dataset such as any fine-grained dataset with a large number of classes (e.g., Stanford-cats, cub-200). However, this is rare in medical imaging which authors claim to be the main objective of the proposed method. Still, it would be better to see the method’s low points (if any in this regard) for clear future use.  \n\n5> It is interesting that how model focus (heatmap) are reduced in the smallest area (which might be responsible for a particular class) irrespective of providing full foreground as binary mask (fig 4 and 5). Any comments on this?  \n\n6> How is proposed method is different from the extension version of GAIN [1] (pixel wise and bbox masks) is not explained anywhere. \n\n[1] Li et al. Guided attention inference network. IEEE TPAMI, 42(12):2996–3010, 2019.\n", "summary_of_the_paper": "This paper proposes a user-guided training for CNN to focus more on casual features which are based on human perspective and termed as casually focused convolutional neural network. Human guidance is used to make a rough binary segmentation mask of foreground objects/pixels.  \n\nFor an input image, mask and label are both utilized to compute final loss. Final loss is composed of 1) activation loss which is computed using the binary mask and last convolution layer features and 2) cross-entropy loss which is calculated using predicted probability and class label of the input image. Images for which binary mask is not available, dummy mask is created as all foreground pixel. Two types of networks, all convolutional layers (CFCN-C) and fully-connected classifiers (CFCN-F) are utilized with the proposed methodology.\n\nExperiments are performed on a total of 4 datasets of natural and medical images. Accuracy and visual activations are compared with baseline CNN, GAIN [1], and both CFCN-C and CFCN-F.\n", "summary_of_the_review": "1> Difference between GAIN and the proposed method is not clear. What are the technical improvement over GAIN and how are they working? \n\n2> Experimental section is weak and could be improved as per suggestions. Also, there is no comparison with intrinsic XAI methods. Proper evaluation can make things more clear.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mVMRVsyP2Je", "reviewer": "Reviewer_nnUS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**General comments:**\n\nThe masking approach is very similar to binary segmentation (e.g. foreground vs background). How well the model would work if the current \"masking loss\" would be substituted by a binary segmentation loss? Such approaches have been extensively studied in the literature, see e.g. https://arxiv.org/pdf/2004.07054.pdf\n\nIt is unclear why the authors decided to use grad-cam given a large body of explainability work (see e.g. https://medium.com/analytics-vidhya/cnns-explainability-papers-review-5ed380577c64)?\n\nThe limitations of Grad-Cam are not discussed either. Is Grad-Cam the best tool to detect causal features?\n\nWould the approach work with multiple objects?\n\nThe limitations to the model are not discussed. Adding discussion of limitations would make the paper stronger.\n\nWould similar approach work with non-CNN based models, e.g. ViT?\n\n\n**Specific comments:**\n\n_Introduction:_\n\n\"... these systems are expected to be justifiable...\" Could the authors clarify this statement? Do the authors mean the decision to be explainable or interpretable? Is it always the case that this expectation is in place for all scenarios discussed in this paragraph?\n\n\"... and often extract features greedily following that principle..\" What do the authors mean by greedily in this context? It is also a bit unclear to which principle the authors are referring to.\n\n\"In order to eliminate the effect of non causal correlations, CNNs need to be trained on huge datasets...\" Could the authors provide a citation for this statement? Can we always remove non causal correlations just by scaling up the dataset?\n\n\"... minimal human guidance.\" Is providing a binary mask really a minimal human guidance? Would it be easier to just provide yes no answers, e.g. is this gradCAM visualization respecting the causal relations for his object?\n\n\n_Methodology:_\n\nFig. 3: It is a bit unclear at which point the binary mask are applied in the CNN pipeline. The figure could be improved by depicting explicitly at which point the multiplication of features with binary mask is performed.\n\nFrom the current description, it is unclear how the model works at inference time. Do we still need binary masks at inference time?\n\nEq. 1 lacks explanation. In particular, why this particular loss is beneficial for the task? Could we design the loss differently? What makes this particular formulation important?\n\n_Experiments:_\n\nIt would be nice to see the effect of how exact the mask has to be in order to notice the described effect, eg. how precisely the object has to be segmented to provide meaningful causal features?\n\nFig. 4: It is interesting to notice that the highlighted features do not capture the whole plane, only some parts of the plane seems to be causally relevant. Adding a discussion on why some pixels are more relevant than others would be beneficial. \n\nTab 3. There is no measure of std or variance.\n\nFig 6. The convergence claim (the modified loss does not affect convergence) is not supported by the plot. To support the claim, the convergence plot of vanilla CNN should be depicted. It would also be nice to state which dataset split has been used to obtain them.\n\nFig 8. It is unclear what AM stands for. This figure also indicates that the ablation was run on the test set, it would be more principled to run all ablations on the validation set.\n", "summary_of_the_paper": "The paper proposes to alternate the loss of classification CNNs by adding additional loss term that focuses the model's attention on the object present in the image rather than on the background. To do so, the model is provided with an additional input (a binary mask) that is used to guide the learning of the model. The approach is validated on four datasets showing some benefits in terms of classification performance on small-scale datasets while decreasing the classification performance on larger scale datasets.", "summary_of_the_review": "The paper tackles an interesting problem of making the model more robust to spurios correlations. Although the reviewer finds the problem interesting and relevant for the ICLR community, the reviewer thinks that the paper in its current state is below the acceptance threshold. \nThe methodological contribution is rather minor. The approach requires collections of binary masks that might not be available for all datasets. It is also a bit unclear what the community learns from this paper -- it is expected to have more object centered gradCAM activations by using object binary masks. Moreover, the presentation of the paper could be significantly improved. For details see main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AV98zSjDPWZ", "reviewer": "Reviewer_teVu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this work, the authors design a causally focused CNN framework through human guidance. By utilizing the auxiliary guidance, the proposed method can particularly focus on the information useful for the classification tasks. Extensive experiments on several datasets have indicated the effectiveness of the proposed method.\n\nStrength:\n+ The proposed method can achieve appealing performance compared with normal classification framework without guidance.\n+ Several feature visualization results have been presented to indicate the effective of the proposed method.\n\nWeakness:\n- There lack discussions on the potential guidance which can be used to improve the model performance on learning causal features. It seems that only segmentation masks are introduced in the manuscript.\n- The potential effect of the proposed method is questionable. As introduced in Section 4.3, the coarse mask boundaries are obtained by a simple regression algorithm. However, the region of interest (ROIs) in the images are sometimes hard to segment. For example, segmenting Multiple Sclerosis lesions in brain MRI is challenging, even a well-trained CNN framework cannot always produce accurate segmentation masks. Under this situation, can the proposed framework still work?\n- The overall methodology design lacks novelty. The causally-focus loss in Equation 1 is similar to the Dice loss, which is widely used for medical image segmentation, such as:\nF. Milletari, N. Navab, S.A. Ahmadi, “V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation”, 3DV2016, pp565-571, 2016.\n- For each dataset, there lacks an introduction on the specific classification task, and the data split details.\n- Introducing an auxiliary segmentation task learning might induce computational cost. Given the unstable performance gain in Table 3, it is questionable whether the proposed framework is worthwhile to use in real practice.\n\nQuestions:\n(1) What if replacing the causally-focus loss in Equation 1 with the normal cross-entropy loss and dice loss?\n(2) Compared with the model without the auxiliary segmentation task learning, how much auxiliary computational cost will the proposed method bring?\n", "summary_of_the_paper": "This work aims at improving the classification tasks via segmentation learning. The methodology and experiments are problematic. ", "summary_of_the_review": "As discussed in the review comments, I think this manuscript lacks technique novelty, has potential limits to more challenging tasks, and has less practical usages. In addition, the overall manuscript is not clearly written. Therefore, I recommend rejecting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although this paper is on an interesting topic, there is a consensus that this paper is below the bar for acceptance. My advice is to take take criticisms of the reviewers seriously, add the extra experiments, rewrite the paper and then submit it to a different conference. If the authors feel that the reviewers misunderstood their paper, please remember that the level to which they were able to understand it is also a function of how the paper is written.
Decision: Reject