Learning Reliable Rules by Re-generating Deep Features

Authors: Yuhe Jiang, Zexin Xue, Xujie Si
Primary area: interpretability and explainable AI
Keywords: Interpretable ML, Neuro-symbolic AI, SATNet, Logical Reasoning

Abstract:
Improving the interpretability and reliability of deep learning models is essential for advancing machine learning applications, though it remains a significant challenge. One promising approach is the integration of logical reasoning into deep learning systems. Previous works have demonstrated that SATNet, a differentiable MaxSAT solver, can learn interpretable and reliable rules from input-output examples in puzzle domains. In this work, we propose *Visual SATNet* (Vi-SATNet), an extended version of SATNet capable of learning logical reasoning rules in more general and complex domains, such as the feature space of real-life images. We find that, given a pre-trained deep convolutional neural network (CNN) architecture, a Vi-SATNet layer can be integrated and trained efficiently to learn a set of reasoning rules on the deep features, guiding the classifier’s decision. Vi-SATNets are trained to perform feature re-generation tasks for a given image dataset, where the re-generated features maintain high accuracy when used for image classification, proving their quality. In our experiment on the Imagenette dataset with a pre-trained VGG19 model, masking out 10\% to 80\% of the features results in classification accuracy ranging from 98.50\% to 93.92\% with Vi-SATNet re-generation, compared to 97.07\% to 9.83\% without re-generation. Furthermore, we introduce a visualization method to illustrate the rules learned by Vi-SATNets, thereby enhancing the interpretability of the pre-trained CNN model.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730564074838

thread_type:
official_review

<thread_object>
creator: reviewer_lbbh

<content>
summary:
In this paper, the author proposed Visual SATNet (Vi-SATNet) which targets learning logic reasoning rules in a general domain. Specifically, the author trains and evaluates Vi-SATNet on deep feature re-generation and the experiment results show the effectiveness of the proposed methods.

strengths:
1. The experiment results are convincing, showing the effectiveness of the proposed method.
2. The author provides detailed definitions and algorithm processes to demonstrate their ideas and contributions.

weaknesses:
1. The author proposes a method to learn logical reasoning rules in a more general domain. However, different from the original SATNet, which targets solving Sudoku, a task has a clear logic process, how can we understand the logical reasoning for the real-life images? The author should provide more explanation and analysis.
2. In Table 1, the author obtained the best results with regeneration with a 30% mask ratio. However, without regeneration, the model's performance will drop with a higher maks ratio.  Why does the model have the best performance with a 30% mask ratio? The author should provide more analysis.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638843714

thread_type:
official_review

<thread_object>
creator: reviewer_hqxe

<content>
summary:
This manuscript introduces Visual SATNet (Vi-SATNet), an innovative extension of SATNet aimed at learning logical rules within complex feature spaces, particularly those generated by convolutional neural networks (CNNs). By employing Vi-SATNet for feature regeneration, the study illustrates its capacity to improve interpretability in CNNs during image classification tasks. Experiments conducted on datasets such as MNIST and Imagenette reveal promising classification outcomes across different levels of feature masking.

strengths:
1.	Vi-SATNet extends the capabilities of the original SATNet by enabling the learning of logical reasoning rules within the feature spaces of convolutional neural networks (CNNs). This generalization allows it to operate effectively on complex datasets beyond simple logical puzzles.
2.	The study presents a new method for feature regeneration that leverages learned logical rules derived from deep features, thereby enhancing the interpretability and reliability of convolutional neural networks (CNNs).
3.	Vi-SATNet can be seamlessly integrated as a drop-in layer within existing CNN architectures, requiring no retraining or fine-tuning.

weaknesses:
1.	The manuscript effectively outlines the architecture of Vi-SATNet and establishes an evaluation framework through feature regeneration tasks, employing cosine similarity and Vi-C agreement as measurable metrics for assessing feature quality. However, it could enhance clarity regarding the Vi-SATNet training process, especially concerning hyperparameter selection.
2.	The manuscript describes visualization through minimal significant feature sets (MSFs) and their corresponding receptive fields. However, a more in-depth explanation of how MSFs correlate with specific learned rules would be advantageous. Without clear criteria for evaluating rule quality, the reader may find it challenging to interpret the rules’ significance. 
3.	To enhance the study's rigor, it is recommended to include additional datasets, such as CIFAR-10 or other complex real-world datasets.
4.     Some related works are needed to discuss in the manuscript, such as [1-2].

[1] Wei J, Garrette D, Linzen T, et al. Frequency effects on syntactic rule learning in transformers[J]. arXiv preprint arXiv:2109.07020, 2021.
[2] Zhang W, Mo S, Liu X, et al. Learning robust rule representations for abstract reasoning via internal inferences[J]. Advances in Neural Information Processing Systems, 2022, 35: 33550-33562.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714598181

thread_type:
official_review

<thread_object>
creator: reviewer_ckfz

<content>
summary:
This paper extends SATNet by enabling it to learn logical rules from the complex feature space of real-life images, allowing it to regenerate masked features while maintaining high classification accuracy. 
Experimental results on the features from some pre-trained models show that the learned reasoning rules allow Vi-SATNet to re-generate missing feature vectors accurately.
Finally, the authors present a visualization technique to illustrate the rules learned by Vi-SATNet models.

strengths:
Experiments on MNIST and Imagenette datasets demonstrate the effectiveness of Vi-SATNet in feature regeneration and classification, showing impressive results even under high masking ratios.

The visualizations of the learned rules offer an intuitive understanding of feature dependencies.

weaknesses:
The visualization of the learned rules is not very clear. Whether in the foreground or background, MSF tends to pay more attention to the position around the target, which is a rather trivial result.

questions:
As mentioned in the weaknesses section, the visualization of the learned rules indicates that the target-related MSF pays more attention to its surrounding features, which is a trivial finding. Since CNN features are derived from convolutions with neighboring pixels, the features at each position are inherently closely related to those in the surrounding area. Could you provide some more meaningful rules from different perspectives?

The features in the paper appear to be derived from the last layer of the convolutional module in the model. What would happen if features from other convolutional layers were used? Could different layers with varying depths extract distinct rules?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732554982916

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Dear reviewers, please kindly let us know if you have further questions, concerns or suggestions

comment:
Dear reviewers, please kindly let us know if you have further questions, concerns or suggestions regarding to our work, and we would be very happy to answer them.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The proposed Vi-SATNet extends the capabilities of the original SATNet by enabling the learning of logical reasoning rules within the feature spaces of convolutional neural networks (CNNs). The study presents a new method for feature regeneration that leverages learned logical rules derived from deep features, thereby enhancing the interpretability and reliability of convolutional neural networks (CNNs).However, the explaniation of the logical reasoning for real-life image is still unclear. And the experimental results as well as the analysis are limited.