Title: Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lens
Venue: NeurIPS 2025
Accepted: yes
Keywords: Concept-based models

Neuro-Symbolic AI

Identifiability

Shortcuts
Abstract: Concept-based Models are neural networks that learn a concept extractor to map inputs to high-level concepts and an inference layer to translate these into predictions. Ensuring these modules produce interpretable concepts and behave reliably in out-of-distribution is crucial, yet the conditions for achieving this remain unclear. We study this problem by establishing a novel connection between Concept-based Models and reasoning shortcuts (RSs), a common issue where models achieve high accuracy by learning low-quality concepts, even when the inference layer is fixed and provided upfront. Specifically, we extend RSs to the more complex setting of Concept-based Models and derive theoretical conditions for identifying both the concepts and the inference layer. Our empirical results highlight the impact of RSs and show that existing methods, even combined with multiple natural mitigation strategies, often fail to meet these conditions in practice.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates joint reasoning shortcuts (JRS) in the context of concept-based models. JRS describes shortcuts emerging from the two-level architecture of CBMs, where the concept extractor, the inference layer or both together can be subject to shortcut learning. The paper analyses these JRS from a theoretical perspective, formally describing situations under which these shortcuts can or cannot occur. Further, the paper illustrates that common mitigation strategies are limited in effect when mitigating JRS, highlighting the importance of the topic.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- The paper covers an important problem of concept-based models: joint reasoning shortcuts, which are even more difficult to handle than regular (reasoning) shortcuts.\n- It introduces a theoretical framework that allows for analysis of these JRS and to describe situations where they occur or can be avoided.\n\nWeaknesses:\n- While the theoretical framework is important to analyse these shortcuts from a formal perspective, the given assumptions are unlikely to hold in practice.\n- The clarity of motivation and setup can be improved.\n\nQUESTIONS:\nRecommendations to improve the clarity of the beginning of the paper:\n- For the introduction and section 2, the current setting is often unclear (for example what type of supervision is considered or what parts of the model are learned). I would suggest to make this more clear in the beginning, especially for the examples up to 2.2.\n- Further, I would suggest to stick to one running example (and not mixing mnist with the pedestrians)\n\nI am confused by the intended semantics:\n- To my understanding, this allows $\\alpha$ and $\\beta$ to permute the concepts with respect to the ground-truth concepts as long as this inversion is corrected.\n- I understand this in the context of a learned $\\beta$ and no ground-truth concepts available. But it does not make sense to me in the context of a fixed inference layer, as this does not have the possibility to permute the concepts anymore. Is it then implied that in this case there is no permutation learned by $\\alpha$?\n\nCan the authors discuss the practicability of assumptions 3.1 and 3.2 in practice? In particular the assumption that the label can be predicted via without ambiguity from the ground-truth concepts seems unrealistic, as it assumes a perfect and complete ground-truth concept set. (Or, if stated differently: when the existence of such a concept set is assumed, it seems unlikely that assumption 3.1 holds and these concepts are always recoverable).\n\nFurther, can the authors provide a brief intuition how the problem of JRS is affected if the CBM is trained independently or if the concept extractor is pretrained in a general setup, as often done in recent CBMNs [1, 2]?\n\nQuestions about the experimental setup and evaluation:\n- How is the CBNM trained if there is no concept supervision? \n- Could the authors explain the evaluation of the learned inference layer? It seems confusing to use $\\pi$ on the concepts if the inference layer is learned, assuming that the inference layer itself is not permutation invariant, as then it gets a different input permutation than usual during its training.\n\n\nMinor:\n- Caption of figure 2: introduce the abbreviation (JRS) before using it\n- Typo in line 264 (reporhhts)\n\n[1] Oikarinen, Tuomas, et al. \"Label-free concept bottleneck models.\" _arXiv preprint arXiv:2304.06129_ (2023).\n\n[2] Yang, Yue, et al. \"Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2023.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes CBMs with respect to reasoning shortcuts: a CBM that learns a classifier of high accuracy without uncovering the true concepts or classifier function over the concepts. For this, the paper establishes a theory inspired by NeSy-CBMs. Using this theory the paper presents results for identifying both the true concepts and the true inference layer. In an empirical analysis, the paper demonstrated the impact of reasoning shortcuts on different datasets ans shows the impact of different mitigation strategies.\n\nSTRENGTHS AND WEAKNESSES:\n**Note:** I have not checked the correctness of the proofs in detail due to the overall length of the paper+appendix.\n\n### Strengths:\n\n- **Originality:** The idea to analyze the learning abilities of CBMs from a mathematical perspective wrt whether the true generative process can be recovered is great. The results provide new insights about how difficult it is in theory to identify this said process. Moreover, the paper clearly motivates the existing work with respect to state of the art, and shows the limitations in learning CBMs even if respective penalties are applied.\n\n- **Significance:** The presented theory and results are of interested for the community that works on the design of interpretable models, especially CBM researchers. Additionally, the paper addresses a difficult task, IMHO.\n\n### Weaknesses:\n\n- **Clarity:** As already said, the paper addresses a difficult task and, in such a situation, it is to some extent okay if the paper is not easy to understand and follow. However, the understandability of the paper should be improved. First of all, a lot of information relevant to key claims (e.g., the transferability of the results to a real task is in Appendix B) are in the appendix making the paper not self-contained. Second, the mathematical part is not easy to understand because of too short descriptions and because of using mathematical notations as isolated descriptors instead of accompanying symbols for clarity of a description (see examples below). My impression is that the paper clarity and also **quality** suffered from the page limit of the venue. I think the paper would be better suited in a journal where the page limit is not so strict. In this case, I think that the paper could become a really strong submission by having the paper better organized and content less scattered across main part and appendix. Regarding the usage of mathematical notations, ideally, mathematical notation should support the written text to add clarity but it shouldn't be a replacement for text. For instance:\n   - L117: \"the maps $\\alpha$ and $\\beta$\" better \"the ground-truth distribution $\\alpha$ that generates the data ...\"; I'm not claiming that this is a good description but I hope it makes my statement clearer.\n   - L83: \"$\\mathcal{G} = \\mathcal{C}$\" better \"where we assume equal discrete $\\mathcal{C}$ and ground-truth concept sets $\\mathcal{G}$, i.e., $\\mathcal{G} = \\mathcal{C}$\" [...just realizing that it is not easy for me to find clear descriptive phrases for the mathematical entities, which is not a good indicator for the paper's clarity].\n\n   The reason why I'm stressing this point is because it complicates clarity as one has to memorize all the notations in order to understand the paragraphs. Moreover, in some cases it leaves room for different interpretations if a mathematical notation is not clear (e.g., L116, $\\mathbb{1}${$\\mathbf{C} = \\mathbf{c}$}, isn't $\\mathbf{C}$ a set and $\\mathbf{c}$ an element of $\\mathbf{C}$? How does comparison work?). Moreover, sometimes mathematical notations are not even clear at all (e.g., cf. L46 and L63: $\\mathbf{C}$ vs $\\mathbf{c}$). Additionally, Figure 1 is not self-explanatory. Even after reading the related text in the corresponding section, it requires some time to understand the illustration.\n\n- **Quality:** The paper nicely presents the assumptions related to the theory. However, when it comes to Definition 3.3 that is the backbone of all the theory, it is not rigorous in explaining the importance and the connection to the disentanglement assumption. It should be clearly stated that the definition only makes sense under this assumption. Then it should be discussed how realistic this assumption is in reality (element-wise, invertible functions and bijection from ground-truth to learned concepts); relating to the **limitations** of the paper. This discussion is crucial as it explains whether the paper analyzes an aspect of practical relevance. Hence, this should be stated clearly. The same is true for Definition 3.4. It should be discussed how realistic this definition is and what happens if it doesn't hold.\n\nQUESTIONS:\n1. L7: What are low-quality concepts? Isn't the definition of a concept quality always human-centric and, hence, notoriously difficult to describe mathematically?\n2. The paper references causality papers. One know paper in this field is \"Towards Causal Representation Learning\" by Schölkopf et al. A major statement in this paper is the fact that statistical ML can only generalize in the iid and not in the ood regime. To generalize in the ood regime, it is required to identify causal connections between variables as each ood shift from the iid training data can be viewed as an intervention (in the causal sense). In this light, how conflicting are statements that CBMs can deal with ood (L20) even if they are identifying correlations (statistical machines)?\n3. L46: What exactly is meant by discrete concepts?\n4. How is Eq. (1) related to the usual factorization of CBMs $P(X,C,Y) = P(X)P(Y|C)P(C|X)$\n5. L81: IMHO this assume that $\\mathcal{G}$ is unique. How realistic is this assumption considering that in reality a set of different objects can be uniquely described by different sets of concepts (e.g., assume different level of abstractions)?\n6. Table 2 results: All the methods in the table are deemed to be interpretable. Consequently, it should be possible to analyze the hypothesized simplicity bias. What rule has the SENN learned to solve the problem and how does this result relate to the established theory?\n7. L341: \"Our work partailly fills this gap\" Why partially? Again, the **limitations** should be clearly stated.\n8. L357: Why is contrastive learning helpful? Once more, using the interpretability of the methods should provide an answer.\n\n\n**Some general notes not related to my assessment:**\n\n- L24: What question? There is no question in the sentences before.\n- Colored links and references are unusual and not really needed IMHO\n- L32: shortcuts [...], more than reasoning shortcuts; it sounds like \"more shortcuts than shortcuts\"\n- L44: a **proven** fact cannot fail **empirically**\n- L99: A set is compared with an element. This doesn't make sense.\n- L264: reporhhts\n- L319: The \"ood\" generalization works because we induce expert knowledge pushing the assumed ood data into iid (see my comment above).\n- Maybe it would be good to have one running example instead of MNIST and autonomous driving.\n\nLIMITATIONS:\nThe discussions of limitations must be improved: How realistic are the assumptions and what are the consequences? See further details in my comments before.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the issue of Joint Reasoning Shortcuts (JRSs) in CBMs and NeSy-CBMs. Through theoretical analysis and empirical validation, the authors reveal how JRSs lead models to learn low-quality concepts and inference layers, thereby compromising interpretability and OOD performance. The paper introduces the notion of \"Intended Semantics\" and derives theoretical conditions under which JRSs can be avoided. Experiments demonstrate the limitations of existing mitigation strategies, particularly in unsupervised settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper is the first to establish a theoretical framework for JRSs, which extends RS, and proposes sufficient conditions to avoid JRSs (e.g., Theorem 3.9).\n2. The theoretical analysis and experimental evaluations in this paper are both highly comprehensive.\n\nWeaknesses:\nThere are some questions regarding the content of Theorem 3.9. If I understand correctly, the theorem states that when the number of deterministic JRSs is zero, the system possesses the intended semantics. While this result aligns with intuition, I believe it is more important to provide a theoretical explanation of the conditions/assumptions under which the number of deterministic JRSs can be reduced.\n\nAdditionally, most of the experiments in this paper are based on variants of MNIST, which is somewhat limited. Furthermore, it would be valuable to compare with other NeSy algorithms (I noticed that only DPL was mentioned in the paper), such as LTN, among others.\n\nQUESTIONS:\nRegarding the issue of the number of deterministic JRSs raised in the weaknesses, I suggest further analyzing how different mitigation strategies might affect this quantity.\n\nConcerning the necessity of studying CBM shortcuts, I have several reservations:\n- Intermediate layers in standard vision neural networks typically lack interpretable semantic features, and there is already extensive research on neural network shortcuts [1]. What distinguishes this work?\n- Enforcing intermediate layers to output interpretable concepts may be redundant (though it does enhance interpretability). In practical applications, constructing meaningful intermediate concepts is highly challenging. However, within the NeSy framework, the presence of intermediate concepts is justified due to the support of a knowledge base. So why not use current sota vision models?\n\n[1] Shortcut Learning in Deep Neural Networks. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to connect reasoning shortcuts from neurosymbolic AI  with concept-based models, where models can achieve high performance without learning the right semantics. It shows that maximum likelihood training alone is insufficient to ensure intended semantics and formalize the conditions for when a CBM can acquire high-quality concepts and inference layer. It also explores and empirically tests potential mitigation strategies. The results show promise on benchmark datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n+ Studying concept and inference layer quality in CBMs from a reasoning shortcuts point of view is novel, to the best of my knowledge. \n+ The presented theory and bounds seemed fairly complete (although I do have some concerns about them in the weaknesses) and the provided examples made things more intuitive.\n\nWeaknesses: \n- The formalization of reasoning shortcuts for CBMs is interesting, however, considering the fact that a lot of the results build on existing works like [1, 2] (which the paper also mentions in the related work), makes the degree of novelty of the work questionable. Additionally, most of the practical solutions (Sec 4), have also already been proposed in [1].\n- I found the paper slightly lacking in clarity. In particular, in Sec 3, where CBMs are modeled as a pair of functions, the concept extractor is defined as mapping onto a simplex, yet in words it is said that it maps to a conditional distribution. Putting this mismatch aside, in standard CBMs [3], the concepts are predicted independently. There is no distribution over them. So it is not clear how vanilla CBM (as widely used) fits into this approach. \n- I also found it difficult to differentiate between NeSyCBMs and general NeSy models - although NeSyCBMs were defined, they weren’t formalized mathematically. Several of the references and baselines were standard NeSy models, like DeepProbLog. \n\nReferences:\n[1] Marconato, Emanuele, et al. \"Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts.\" Advances in Neural Information Processing Systems 36. 2023.\n[2] Yang, Xiao-Wen, et al. \"Analysis for abductive learning and neural-symbolic reasoning shortcuts.\" Forty-first International Conference on Machine Learning. 2024.\n[3] Koh, Pang Wei, et al. \"Concept bottleneck models.\" International conference on machine learning. PMLR, 2020.\n[4] Alvarez Melis, David, and Tommi Jaakkola. \"Towards robust interpretability with self-explaining neural networks.\" Advances in neural information processing systems 31 (2018).\n\nQUESTIONS:\n* The paper seems to give a concept-based lens to a neurosymbolic models, rather than the other way around. It claims that a CBM is a NeSyCBM with a learnable inference layer. Is my understanding correct? If so, I don’t fully agree with this, as CBMs and NeSy models are fundamentally meant for different tasks (classification and reasoning respectively). Could you please exactly state what sort of CBM is being referred to throughout?\n* Is there a difference between NeSy models as NeSyCBMs? If so, could you please give an example of a NeSy model that isn’t a NeSyCBM? \n* Could you explain the reasoning behind the choice of baselines? There are only two CBM-based baselines [3, 4]. All the others are NeSy ones. Also, if the CBNMs [3] have no concept supervision (L267), how are the concepts learned?\n* The paper brings up desiderata for high-quality concepts and inference layer earlier in the paper. Yet these aren’t evaluated in any way or even used much in the derivations, for that matter. Could you explain why they were introduced?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}