Title: Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neuro-Symbolic Integration

Trustworthy AI

Concept Learning

Learning Shortcuts

Mitigation Strategies
Abstract: Neuro-Symbolic (NeSy) predictive models hold the promise of improved compliance with given constraints, systematic generalization, and interpretability, as they allow to infer labels that are consistent with some prior knowledge by reasoning over high-level concepts extracted from sub-symbolic inputs. It was recently shown that NeSy predictors are affected by *reasoning shortcuts*: they can attain high accuracy but by leveraging concepts with \textit{unintended semantics}, thus coming short of their promised advantages. Yet, a systematic characterization of reasoning shortcuts and of potential mitigation strategies is missing. This work fills this gap by characterizing them as unintended optima of the learning objective and identifying four key conditions behind their occurrence. Based on this, we derive several natural mitigation strategies, and analyze their efficacy both theoretically and empirically. Our analysis shows reasoning shortcuts are difficult to deal with, casting doubts on the trustworthiness and interpretability of existing NeSy solutions.

Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts

Abstract

Neuro-Symbolic (NeSy) predictive models hold the promise of improved compliance with given constraints, systematic generalization, and interpretability, as they allow to infer labels that are consistent with some prior knowledge by reasoning over high-level concepts extracted from sub-symbolic inputs. It was recently shown that NeSy predictors are affected by reasoning shortcuts : they can attain high accuracy but by leveraging concepts with unintended semantics , thus coming short of their promised advantages. Yet, a systematic characterization of reasoning shortcuts and of potential mitigation strategies is missing. This work fills this gap by characterizing them as unintended optima of the learning objective and identifying four key conditions behind their occurrence. Based on this, we derive several natural mitigation strategies, and analyze their efficacy both theoretically and empirically. Our analysis shows reasoning shortcuts are difficult to deal with, casting doubts on the trustworthiness and interpretability of existing NeSy solutions.

1 Introduction

Neuro-Symbolic (NeSy) AI aims at improving the robustness and trustworthiness of neural networks by integrating them with reasoning capabilities and prior knowledge [1, 2]. We focus on NeSy predictors [3, 4], neural structured-output classifiers that infer one or more labels by reasoning over high-level concepts extracted from sub-symbolic inputs, like images or text [5, 6, 7, 8, 9, 10, 11]. They leverage reasoning techniques to encourage - or even guarantee - that their predictions comply with domain-specific regulations and inference rules. As such, they hold the promise of improved systematic generalization , modularity , and interpretability , in that learned concepts can be readily reused in different NeSy tasks, as done for verification [12], and for explaining the model's inference process to stakeholders [13, 14, 15]. On paper, this makes NeSy predictors ideal for high-stakes applications that require both transparency and fine-grained control over the model's (in- and out-ofdistribution) behavior, such as medical diagnosis [16], robotics [17] and self-driving cars [18].

Much of the promise of these models relies on learned concepts being high quality . The general consensus is that the prior knowledge constrains learned concepts to behave as expected [19] and issues with them are often tackled heuristically [20]. It was recently shown that, however, NeSy predictors can attain high accuracy by leveraging concepts with unintended semantics [21]. Following

K = (pedestrian V red ≥ stop)

K2 = (emergency ^ -pedestrian → go）^ K

red

green

Task

1

Task

red

red

y=stop

y=stop

y=stop

y=stop

y=go

Marconato et al. [21], we refer to these as reasoning shortcuts (RSs). RSs are problematic, as concepts encoding unintended semantics compromise generalization across NeSy tasks, as shown in Fig. 1, as well as interpretability and verification of NeSy systems [12]. Moreover, the only known mitigation strategies are based on heuristics [20, 11].

The issue is that RSs - and their root causes - are not well understood, making it difficult to design effective remedies. In this paper, we fill this gap. We introduce a formal definition of RSs and theoretically characterize their properties, highlighting how they are a general phenomenon affecting a variety of state-of-the-art NeSy predictors. Specifically, our results show that RSs can be shared across many NeSy architectures, and provide a way of counting them for any given learning problem. They also show that RSs depend on four key factors , namely the structure of the prior knowledge and of the data, the learning objective, and the architecture of the neural concept extractor. This enables us to identify several supervised and unsupervised mitigation strategies, which we systematically analyze both theoretically and empirically. Finally, we experimentally validate our findings by testing a number of representative NeSy predictors and mitigation strategies on four NeSy data sets.

Contributions. Summarizing, we: ( i ) Formalize RSs and identify four key root causes. ( ii ) Show that RSs are a general issue impacting a variety of NeSy predictors. ( iii ) Identify a number of mitigation strategies and analyze their effectiveness, or lack thereof. ( iv ) Empirically show that RSs arise even when the data set is large and unbiased, and evaluate the efficacy of different mitigation strategies, highlighting the limits of unsupervised remedies and the lack of a widely applicable recipe.

2 The Family of Neuro-Symbolic Predictors

Notation. Throughout, we indicate scalar constants x in lower-case, random variables X in upper case, and ordered sets of constants x and random variables X in bold typeface. Also, x i : j denotes the subset { x i , . . . , x j } , [ n ] the set { 1 , . . . , n } , and x | = K indicates that x satisfies a logical formula K .

NeSy predictors. A NeSy predictor is a model that infers n labels Y (taking values in Y ) by reasoning over a set of k discrete concepts C (taking values in C ) extracted from a sub-symbolic continous input X (taking values in X ). Reasoning can be implemented in different ways, but overall its role is to encourage the model's predictions to comply with given prior knowledge K , in the sense that predictions y that do not satisfy the knowledge are generally avoided. The prior knowledge K is assumed to be provided upfront and correct, as formalized in Section 3. Normally, only supervision on the labels Y is available for training, with the concepts C treated as latent variables.

Example 1. In MNIST-Addition [7], given a pair of MNIST images [22], say x = ( , ) , the model has to infer the concepts C = ( C 1 , C 2 ) encoding the digit classes, to predict their sum Y , in this case 8 . Reasoning drives the model towards complying with constraint K = ( Y = C 1 + C 2 ) .

The concepts are modeled by a conditional distribution p θ ( C | X ) parameterized by θ ∈ Θ , typically implemented with a neural network. The predicted concepts can be viewed as 'soft' or 'neural' predicates with a truth value ranging in [0 , 1] . As for the reasoning step, the most popular strategies

involve penalizing the model for producing concepts and/or labels inconsistent with the knowledge at training time [8, 23, 24] or introducing a reasoning layer that infers labels from the predicted concepts and also operates at inference time [7, 9, 25, 10]. In either case, end-to-end training requires to differentiate through the knowledge. Mainstream options include softening the knowledge using fuzzy logic [26, 6, 27, 11] and casting reasoning in terms of probabilistic logics [28, 7, 10].

To investigate the scope and impact of RSs, we consider three representative NeSy predictors. The first one is DeepProbLog (DPL) [7], which implements a sound probabilistic-logic reasoning layer on top of the neural predicates. DPL is a discriminative predictor of the form [21]:

where the concept distribution is fully factorized and the label distribution is uniform 1 over all label-concept combinations compatible with the knowledge, that is, u K ( y | c ) = ✶ { c | = K [ Y / y ] } /Z ( c ; K ) where the indicator ✶ { c | = K [ Y / y ] } guarantees all labels inconsistent with K have zero probability, and Z ( c ; K ) = ∑ y ✶ { c | = K [ Y / y ] } is a normalizing constant. Inference amounts to computing a most likely label argmax y p θ ( y | x ; K ) , while learning is carried out via maximum (log-)likelihood estimation, that is given a training set D = { ( x , y ) } , maximizing

In general, it is intractable to evaluate Eq. (1) and solve inference exactly. DPL leverages knowledge compilation [29, 30] to make both steps practical.

Our analysis on DPL can be carried over to other NeSy predictors implementing analogous reasoning layers [31, 32, 33, 10, 34]. Furthermore, we show that certain RSs affect also alternative NeSy approaches such as the Semantic Loss (SL) [8] and Logic Tensor Networks (LTNs) [6], two stateof-the-art penalty-based approaches. Both reward a neural network for predicting labels y consistent with the knowledge, but SL measures consistency in probabilistic terms, while LTNs use fuzzy logic to measure a fuzzy degree of knowledge satisfaction. See Appendix A for a full description.

3 Reasoning Shortcuts as Unintended Optima

It was recently shown that NeSy predictors are vulnerable to reasoning shortcuts (RSs), whereby the model attains high accuracy by leveraging concepts with unintended semantics [21].

Example 2. To build intuition, consider MNIST-Addition and assume the model is trained on examples of only two sums: + = 1 and + = 2 . Note that there exist two distinct maps from images to concepts that perfectly classify such examples: one is the intended solution ( ↦→ 0 , ↦→ 1 , ↦→ 2) , while the other is ( ↦→ 1 , ↦→ 0 , ↦→ 1) . The latter is unintended.

The ground-truth data generating process. In order to properly define what a RS is, we have to first define what non -shortcut solutions are. In line with work on causal representation learning [35, 36, 37], we do so by specifying the ground-truth data generation process p ∗ ( X , Y ; K ) . Specifically, we assume it takes the form illustrated in Fig. 2. In short, we assume there exist k unobserved ground-truth concepts G ( e.g. , in MNIST-Addition these are the digits 0 to 9 ) that determine both the observations X (the MNIST images) and the labels Y (the sum). We also allow for extra stylistic factors S , independent from G , that do influence the observed data ( e.g. , calligraphic style) but not the labels. The groundtruth concepts are discrete and range in G = [ m 1 ] ×··· × [ m k ] , whereas the style s ∈ R q is continuous. The training and test examples ( x , y ) are then obtained by first sampling g and s and then x ∼ p ∗ ( X | g , s ) and y ∼ p ∗ ( Y | g ; K ) . Later on, we will use supp ( G ) to denote the support

S

G

X

Y

C

of p ∗ ( G ) . We also assume that the ground-truth process is consistent with the prior knowledge K , in the sense that invalid examples are never generated: p ∗ ( y | g ; K ) = 0 for all ( g , y ) that violate K .

What is a reasoning shortcut? By definition, a NeSy predictor p θ ( Y | X ; K ) - shown in blue in Fig. 2 - acquires concepts with the correct semantics if it recovers the ground-truth concepts, i.e. ,

A model satisfying Eq. (3) easily generalizes to other NeSy prediction tasks that make use of the same ground-truth concepts G , as p θ ( C | X ) can be reused for solving the new task [21, 38]. It is also interpretable, in the sense that as long as stakeholders understand the factors G , they can also interpret concept-based explanations of the inference process that rely on C [39]. Naturally, there may exist many concept distributions that violate Eq. (3) and that as such do not capture the correct semantics. However, all those that achieve sub-par log-likelihoods can be ideally avoided simply by improving learning or supplying more examples. We define RSs as those concept distributions for which these strategies are not enough.

Definition 1. A reasoning shortcut is a distribution p θ ( C | X ) that achieves maximal log-likelihood on the training set but does not match the ground-truth concept distribution,

This makes RSs difficult to improve by regular means and also hard to spot based on predictions alone. Yet, since the concepts do not recover the correct semantics, RSs can compromise systematic generalization and interpretability. For instance, the shortcut concepts learned in Example 2 would fail completely at MNIST-Addition tasks involving digits other than 0 , 1 , and 2 and also at other arithmetic tasks, like multiplication, involving the same digits, as we show experimentally in Section 6. An examples of RSs in a high-stakes scenario is shown in Fig. 1.

4 Properties of Reasoning Shortcuts

Counting deterministic RSs. We begin by assessing how many RSs exist in an idealized setting in which the ground-truth generative process is simple, we have access to the true risk, and the concepts C have been specified correctly, that is, C = G , and then proceed to work out what this entails. Specifically, we work with these assumptions:

A1 . The distribution p ∗ ( X | G , S ) is induced by a map f : ( g , s ) ↦→ x , i.e. , p ∗ ( X | G , S ) = δ { X = f ( g , s ) } , where f is invertible , and smooth over s .

A2 . The distribution p ∗ ( Y | G ; K ) is induced by a map β K : g ↦→ y . This is what happens in MNIST-Addition , as there exists a unique value y that is the sum of any two digits ( g 1 , g 2 ) .

Our analysis builds on a link between the NeSy predictor p θ ( Y | X ; K ) and the concept extractor p θ ( C | X ) , which depend on X , and their analogues p θ ( Y | G ; K ) and p θ ( C | G ) that depend directly on the ground-truth concepts G . This link is formalized by the following lemma:

Lemma 1. It holds that: (i) The true risk of p θ can be upper bounded as follows:

where KL is the Kullback-Leibler divergence and H is the Shannon entropy. Moreover, under A1 and A2 , p θ ( Y | X ; K ) is an optimum of the LHS of Eq. (5) if and only if p θ ( Y | G ; K ) is an optimum of the RHS. (ii) Under A1 , there exists a bijection between the deterministic concept distributions p θ ( C | X ) that are constant over the support of p ( X | g ) , for each g ∈ supp( G ) , and the deterministic distributions of the form p θ ( C | G ) .

All proofs can be found in Appendix B. Lemma 1 implies that the deterministic concept distributions p θ ( C | X ) of NeSy predictors p θ ( Y | X ; K ) that maximize the LHS of Eq. (5), including those that are RSs, correspond one-to-one to the deterministic distributions p θ ( C | G ) yielding label distributions p θ ( Y | G ; K ) that maximize the RHS of Eq. (5). Hence, we can count the number of deterministic RSs by counting the deterministic distributions p θ ( C | G ) :

Theorem 2. Let A be the set of mappings α : g ↦→ c induced by all possible deterministic distributions p θ ( C | G ) , i.e., each p θ ( C | G ) = ✶ { C = α ( G ) } for exactly one α ∈ A . Under A1 and A2 , the number of deterministic optima p θ ( C | G ) of Eq. (5) is:

Intuitively, this sum counts the deterministic concept distributions p θ ( C | X ) - embodied here by the maps α - that output concepts predicting a correct label for each example in the training set. The

̸

̸

ground-truth distribution p ∗ ( G | X ) is one such distribution, so the count is always at least one, but there may be more, and all of these are RSs. Eq. (6) gives us their exact number. This formalizes the intuition of Marconato et al. [21] that, as long as the prior knowledge K admits the correct label y to be inferred from more than one concept vector c , there is room for RSs. So far, we have assumed Eq. (2) is computed as in DPL. However, RSs are chiefly a property of the prior knowledge , and as such also affect NeSy predictors employing different reasoning procedures or different relaxations of the knowledge. We show this formally in Appendix A. Deterministic RS are also important because in certain cases - they define a basis for all reasoning shortcuts:

Proposition 3. For probabilistic logic approaches (including DPL and SL): (i) All convex combinations of two or more deterministic optima p θ ( C | X ) of the likelihood are also (non-deterministic) optima. However, not all convex combinations can be expressed in DPL and SL. (ii) Under A1 and A2 , all optima of the likelihood can be expressed as a convex combination of deterministic optima. (iii) If A2 does not hold, there may exist non-deterministic optima that are not convex combinations of deterministic ones. These may be the only optima.

Combining Proposition 3 ( i ) with Theorem 2 gives us a lower bound for the number of nondeterministic RSs, in the sense that if there are at least two deterministic RS, then there exist infinitely many non-deterministic ones. An important consequence is that, if we can somehow control what deterministic RSs affect the model, then we may be able to implicitly lower the number of non-deterministic RSs as well. However, Proposition 3 implies that there may exist non-deterministic RSs that are unrelated to the deterministic ones and that as such cannot be controlled this way.

5 Analysis of Mitigation Strategies

The key factors underlying the occurrence of deterministic RSs appear explicitly in Eq. (6). These are: ( i ) the knowledge K , ( ii ) the structure of supp( G ) , ( iii ) the objective function L used for training (via Lemma 1), and ( iv ) the architecture of the concept extractor p θ ( C | X ) , embodied in the Theorem by p θ ( C | G ) . This gives us a starting point for identifying possible mitigation strategies and analyzing their impact on the number of deterministic RSs. Our main results are summarized in Table 1.

5.1 Knowledge-based Mitigation

The prior knowledge K is the main factor behind RSs and also a prime target for mitigation. The most direct way of eliminating unintended concepts is to edit K directly, for instance by eliciting additional constraints from a domain expert. However, depending on the application, this may not be feasible: experts may not be available, or it may be impossible to constrain K without also eliminating concepts with the intended semantics.

A more practical alternative is to employ Multi-Task Learning (MTL). The idea is to train a NeSy predictor over T tasks sharing the same ground-truth concepts G but differing prior knowledge K ( t ) , for t ∈ [ T ] . E.g. , one could learn a model to predict both the sum and product of MNIST digits, as in our experiments (Section 6). Intuitively, by constraining the concepts to work well across tasks, MTL leaves less room for unintended semantics. The following result confirms this intuition:

Proposition 4. Consider T NeSy prediction tasks with knowledge K ( t ) , for t ∈ [ T ] and data sets D ( t ) , all sharing the same p ∗ ( G ) . Under A1 and A2 , any deterministic optimum p θ ( C | G ) of the MTL loss (i.e., the average of per-task losses) is a deterministic optimum of a single task with prior

knowledge ∧ t K ( t ) . The number of deterministic optima amounts to:

This means that, essentially, MTL behaves like a logical conjunction: any concept extractor p θ ( C | G ) incompatible with the knowledge of any task t is not optimal. This strategy can be very effective, and indeed it performs very well in our experiments, but it necessitates gathering or designing a set of correlated learning tasks, which may be impractical in some situations.

5.2 Data-based Mitigation

Another key factor is the support of p ∗ ( G ) : if the support is not full, the conjunction in Eq. (6) becomes looser, and the number of α 's satisfying it increases. This is what happens in MNIST-EvenOdd (Example 2): here, RSs arise precisely because the training set only includes a subset of combinations of digits, leaving ample room for acquiring unintended concepts.

We stress, however, that RSs can also occur if the data set is exhaustive , as in the next example.

Example 3 ( XOR task) . Consider a task with three binary ground-truth concepts G = ( G 1 , G 2 , G 3 ) in which the label Y is the parity of these bits, that is K = ( Y = G 1 ⊕ G 2 ⊕ G 3 ) . Each label Y ∈ { 0 , 1 } can be inferred from four possible concept vectors g , meaning that knowing y is not sufficient to identify the g it was generated from. In this case, it is impossible to pin down the ground-truth distribution p ∗ ( C | G ; K ) even if all possible combinations of inputs x are observed.

One way of avoiding RSs is to explicitly guide the model towards satisfying the condition in Eq. (3) by supplying supervision for a subset of concepts C I ⊆ C , with I ⊆ [ k ] , and then augmenting the log-likelihood with a cross-entropy loss over the concepts of the form ∑ i ∈ I log p θ ( C i = g i | x ) . Here, training examples ( x , g I , y ) come with annotations for the concepts indexed by I . The impact of this strategy on the number of deterministic RSs is given by the following result:

Proposition 5. Assume that concept supervision is available for all g in S ⊆ supp ( G ) . Under A1 , the number of deterministic optima p θ ( C | G ) minimizing the cross-entropy over the concepts is:

This strategy is very powerful: if I = [ k ] , S ≡ supp ( G ) , and the support is complete, there exists only one map α that is consistent with the condition in Eq. (8) and it is the identity. Naturally, this comes at the cost of obtaining dense annotations for all examples, which is often impractical.

5.3 Objective-based Mitigation

A natural alternative is to augment the log-likelihood with an unsupervised penalty designed to improve concept quality. We focus on reconstruction penalties like those used in auto-encoders [40, 41, 42, 43, 44], which encourage the model to capture all information necessary to reconstruct the input x . To see why these might be useful, consider Example 2. Here, the model learns a RS mapping both and to the digit 1 : this RS hinders reconstruction of the input images, and therefore could be avoided by introducing a reconstruction penalty.

In order to implement this, we introduce additional latent variables Z that capture the style S of the input X and modify the concept extractor to output both C and Z , that is: p θ ( c , z | x ) = p θ ( c | x ) · p θ ( z | x ) . The auto-encoder reconstruction penalty is then given by:

where, p ψ ( x | c , z ) is the decoder network. We need to introduce an additional assumption A3 : the encoder and the decoder separate content from style, that is, p θ ( C , Z | G , S ) := E x ∼ p ∗ ( x | G , S ) p θ ( C , Z | x ) factorizes as p θ ( C | G ) p θ ( Z | S ) and p ψ ( G , S | C , Z ) := E x ∼ p ψ ( x | C , Z ) p ∗ ( G , Z | x ) as p ψ ( G | C ) p ψ ( S | Z ) . In this case, we have the following result:

Proposition 6. Under A1 and A3 , the number of deterministic distributions p θ ( C | G ) that minimize the reconstruction penality in Eq. (9) is:

̸

̸

In words, this shows that indeed optimizing for reconstruction facilitates disambiguating between different concepts, i.e. , different ground-truth concepts cannot be mapped to the same concept. However, minimizing the reconstruction can be non-trivial in practice, especially for complex inputs.

5.4 Architecture-based Mitigation

One last factor is the architecture of the concept extractor p θ ( C | X ) , as it implicitly controls the number of candidate deterministic maps A and therefore the sum in Theorem 2. If the architecture is unrestricted, p θ ( C | X ) can in principle map any ground-truth concept g that generated x to any concept c , thus the cardinality of A increases exponentially with k .

A powerful strategy for reducing the size of A is disentanglement . A model is disentangled if and only if p θ ( C | G ) factorizes as ∏ j ∈ [ k ] p θ ( C j | G j ) [45, 36]. In this case, the maps α also factorize into per-concept maps α j : [ m j ] → [ m j ] , dramatically reducing the cardinality of A , as shown by our first experiment. In applications where the k concepts are naturally independent from one another, e.g. , digits in MNIST-Addition , one can implement disentanglement by predicting each concept using the same neural network, although more general techniques exist [46, 47].

5.5 Other Heuristics based on Entropy Regularization

Besides these mitigation strategies, we investigate empirically the effect of the Shannon entropy loss, defined as 1 -1 k ∑ k i =1 H m i [ p θ ( c i )] , which was shown to increase concept quality in DPL [20]. Here, p θ ( C ) is the marginal distribution over the concepts, and H m i is the normalized Shannon entropy over m i possible values for the distribution. Notice that this term goes to zero only when each distribution p θ ( C i ) is uniform, which may conflict with the real objective of the NeSy prediction (especially when only few concepts are observed). Other similar heuristics that are suited for reducing over-confidence in label predictions are based on label smoothing [48], energy-based models [49], annealing [50], and many others [51, 52, 53]. In principle, when applied at the concept level, they help reduce the over-confidence in concepts that is typical in deterministic RSs. While they could also be beneficial to mildly reduce the number of RSs, we take the Shannon entropy regularization as a representative of this family for our experiments.

6 Case Studies

In this section, we evaluate the impact of RSs in synthetic and real-world NeSy prediction tasks and how the mitigation strategies discussed in Section 5 fare in practice. More details about the models and data are reported in Appendix C. The code is available at github.com/reasoning-shortcuts.

Q1: More data does not prevent RSs but disentanglement helps. We start by evaluating the robustness of DPL, SL, and LTN to RSs in two settings where the data set is exhaustive . In XOR (cf. Example 3), the goal is to predict the parity of three binary concepts g ∈ { 0 , 1 } 3 given prior knowledge K = ( y = g 1 ⊕ g 2 ⊕ g 3 ) . The predictor receives the ground-truth concepts g as input and has to learn a distribution p θ ( C | G ) . In MNIST-Addition (cf. Example 1) the goal is to correctly predict the sum of two MNIST digits, e.g. , x = ( , ) and y = 3 . In both tasks, the training set contains examples of all possible combinations of concepts, i.e. , supp ( G ) = G .

Since the tasks are relatively simple, we can afford to study models that achieve near-optimal likelihood, for which Definition 1 approximately applies. To this end, for each NeSy architecture, we train several models using different seeds and stop as soon as we obtain 30 models with likelihood ≥ 0 . 95 . Then, we measure the percentage of models that have acquired a RS. We do the same also for models modified to ensure they are disentangled (DIS in the Table), see Appendix C for details. The results, reported in Table 2, clearly show that RSs affect all methods if disentanglement is not in place. This confirms that, without implicit architectural biases, optimizing for label accuracy alone is not

MNIST-EvenOdd

DPL

SL

LTN

F

1

(

Y

)

F

1

(

C

)

F

1

(

Y

)

F

1

(

C

)

F

1

(

Y

)

F

1

(

C

)

-

85

.

1

±

4

.

6

0

.

1

±

0

.

1

99

.

3

±

0

.

2

0

.

2

±

0

.

1

98

.

1

±

0

.

2

0

.

3

±

0

.

1

R

79

.

8

±

1

.

0

0

.

1

±

0

.

0

99

.

5

±

0

.

2

0

.

1

±

0

.

0

76

.

3

±

1

.

1

0

.

0

±

0

.

0

H

98

.

1

±

0

.

1

0

.

1

±

0

.

1

99

.

4

±

0

.

1

0

.

1

±

0

.

0

81

.

9

±

0

.

5

53

.

9

±

0

.

7

C

84

.

9

±

0

.

1

0

.

1

±

0

.

1

99

.

3

±

0

.

4

21

.

5

±

6

.

2

98

.

1

±

0

.

2

0

.

2

±

0

.

1

R + H

75

.

4

±

0

.

4

0

.

2

±

0

.

1

99

.

6

±

0

.

1

0

.

1

±

0

.

0

97

.

9

±

2

.

3

38

.

1

±

16

.

7

R + C

84

.

0

±

2

.

2

1

.

9

±

4

.

4

99

.

3

±

0

.

2

61

.

5

±

7

.

8

98

.

1

±

0

.

2

0

.

2

±

0

.

1

H + C

91

.

9

±

3

.

5

88

.

0

±

6

.

3

99

.

4

±

0

.

2

41

.

5

±

8

.

2

98

.

2

±

0

.

2

98

.

6

±

0

.

1

R + H + C

95

.

4

±

0

.

4

96

.

2

±

0

.

2

99

.

5

±

0

.

2

47

.

2

±

9

.

8

98

.

1

±

0

.

3

98

.

5

±

0

.

2

MNIST-AddMul

DPL

SL

LTN

F

1

(

Y

)

F

1

(

C

)

F

1

(

Y

)

F

1

(

C

)

F

1

(

Y

)

F

1

(

C

)

ADD

68

.

1

±

6

.

7

0

.

0

±

0

.

0

99

.

5

±

0

.

2

0

.

0

±

0

.

1

67

.

4

±

0

.

1

0

.

0

±

0

.

0

MULT

100

.

0

±

0

.

0

37

.

6

±

0

.

2

100

.

0

±

0

.

0

76

.

1

±

11

.

7

98

.

1

±

0

.

5

78

.

1

±

0

.

4

MULTIOP

100

.

0

±

0

.

0

99

.

8

±

0

.

1

100

.

0

±

0

.

0

99

.

8

±

0

.

1

98

.

3

±

0

.

2

98

.

3

±

0

.

2

sufficient to rule out RSs, even when the data is exhaustive . However, when forcing disentanglement, the percentage of models affected by RSs drops to zero for all data sets and methods, indicating that mitigation strategies that go beyond the standard learning setup - and specifically, disentanglement can be extremely effective.

Q2: Disentanglement is not enough under selection bias. Next, we look at two (non-exhaustive) data sets where RS occur due to selection bias . Label and concept quality is measured using the F 1 -score on the macro average measured on the test set. From here onward, all models are disentangled by construction, and despite this are affected by RSs, as shown below.

We start by evaluating MNIST-EvenOdd , a variant of MNIST-Addition (inspired by [21]) where only 16 possible pairs of digits out of 100 are given for training: 8 comprise only even digits and 8 only odd digits. As shown in [21], this setup allows to exchange the semantics of the even and odd digits while ensuring all sums are correct (because, e.g. , + = 12 = + ). As commonly done in NeSy, hyperparameters were chosen to optimize for label prediction performance on a validation set, cf. Appendix C. The impact of reconstruction (R), concept supervision (C), and Shannon entropy loss (H) on all architectures are reported in Table 3. Roughly speaking, a concept F 1 below 95% typically indicates a RSs, as shown by the concept confusion matrices in Appendix C.5. The main take away is that no strategy alone can effectively mitigate RSs for any of the methods . Additionally, for DPL and LTN, these also tend to interfere with label supervision, yielding degraded prediction performance. The SL is not affected by this, likely because it is the only method using a separate neural layer to predict the labels. Combining multiple strategies does improve concept quality on average, depending on the method. In particular, H + C and R + H + C help LTN identify good concepts, and similarly for DPL, although concept quality is slightly less stable. For the SL only concept supervision is relatively effective, but the other strategies are not, probably due to the extra flexibility granted by the top neural layer compared to DPL and LTN.

Next, we evaluate the impact of multi-task learning on an arithmetic task, denoted MNIST-AddMul . Here, the model observes inputs x ∈ { ( , ) , ( , ) , ( , ) } and has to predict both their sum and their product, either separately (no MTL) or jointly (MTL). The results in Table 3 show that, as in Example 2, all methods are dramatically affected by RSs when MTL is not in place. DPL and LTN also yield sub-par F 1 -scores on the labels for the addition task. For SL and LTN, we also observe that, despite high concept F 1 for the multiplication task, the digit is never predicted correctly. This is clearly visible in the confusion matrices in Appendix C.5. However, solving addition and multiplication jointly via MTL ensures all methods acquire very high quality concepts.

Q3: Reasoning Shortcuts are pervasive in real-world tasks. Finally, we look at RSs occurring in BDD-OIA [54], an autonomous vehicle prediction task. The goal is to predict multiple possible actions Y = ( move_forward , stop , turn_left , turn_right ) from frames x of real driving scenes. Each scene is described by 21 binary concepts C and the knowledge K ensures the predictions are consistent with safety constraints ( e.g. , stop ⇒ ¬ move_forward ) and guarantees concepts are predicted consistently with one another ( e.g. , road_clear ⇔¬ obstacle ). Since this is a high-stakes task, we solve it with DPL, as it is the only approach out of the ones we consider that guarantees hard compliance with the knowledge [7]. See Appendix C for the full experimental setup.

Table 4 lists the results for DPL paired with different mitigation strategies. In order to get a sense of the model's ability to acquire good concepts, we also include two baselines: CBM-AUC introduced by

FORWARD

STOP

LEFT

RIGHT

1.0

DPL

0.8

0.6

DPL+C+H

0.4

0.2

0.0

Sawada and Nakamura [55] for this task, which learns jointly the concepts together with unsupervised ones, and processes both of them with a linear layer to obtain the labels; C-only, in which the concept extractor is trained with full concept supervision and frozen, and DPL is only used to perform inference. This avoids any interference between label and concept supervision. The F 1 -scores are computed over the test set and averaged over all 4 labels and all 21 concepts. On the right, we also report the concept confusion matrices (CMs) for the training set: each row corresponds to a ground-truth concept vector g , and each column to a predicted concept vector c .

Overall, the results show that, unless supplied with concept supervision, DPL optimizes for label accuracy ( F 1 (Y)) by leveraging low quality concepts ( F 1 (C)). This occurs even when it is paired with a Shannon entropy penalty H. This is especially evident in the CMs, which show that, for certain labels, all ground-truth concepts g are mapped a single c , not unlike what happens in our first experiment for entangled models. Conversely, the concept quality of DPL substantially improves when concept supervision is available, at the cost of a slight degradation in prediction accuracy. The CMs back up this observation as the learned concepts tend to align much closer to the diagonal. Only for the turn_left label concept supervision fails to prevent collapse. This occurs because annotations for the corresponding concepts are poor: for a large set of examples, the concepts necessary to predict turn_left = 1 are annotated as negatives, complicating learning. In practice, this means that all variants of DPL predict those concepts as negative, hindering concept quality.

7 Related Work

Shortcuts in ML. State-of-the-art ML predictors often achieve high performance by exploiting spurious correlations - or 'shortcuts' - in the training data [56]. Well known examples include watermarks [57], background pixels [58, 59], and textual meta-data in X-ray scans [16]. Like RSs, regular shortcuts can compromise the classifier's reliability and out-of-distribution generalization and are hard to identify based on accuracy alone. Proposed solutions include dense annotations [60], out-of-domain data [61], and interaction with annotators [62]. Shortcuts are often the result of confounding resulting from, e.g. , selective sampling. RSs may also arise due to confounding, as is the case in MNIST-Addition (cf. Example 1), however - as discussed in Section 5, data is not the only factor underlying RSs. For instance, in XOR (cf. Example 3) RSs arise despite exhaustive data.

Reasoning shortcuts. The issue of RSs has so far been mostly neglected in the NeSy literature, and few remedies have been introduced but never theoretically motivated. Stammer et al. [63] have investigated shortcuts affecting NeSy architectures, but consider only input-level shortcuts that occur even if concepts are high-quality and fix by injecting additional knowledge in the model. In contrast, we focus on RSs that impact the quality of learned concepts. Marconato et al. [21] introduce the concept of RSs in the context of NeSy but for continual learning, and proposes a combination of concept supervision and concept-level rehearsal to address them, without delving into a theoretical justification. Li et al. [11] propose a minimax objective that ensures the concepts learned by the model satisfy K . Like the entropy regularizer [20] we addressed in Section 5, this strategy ends up spreading probability across all concepts that satisfy the knowledge, including those that have unintended semantics. As such, it does not directly address RSs.

Neuro-symbolic integration. While RSs affect a number of NeSy predictors, NeSy encompasses a heterogeneous family of architectures integrating learning and reasoning [1, 2]. We conjecture that

RSs do transfer to all NeSy approaches that do not specifically address the factors we identified, but an in-depth analysis of RSs in NeSy is beyond the scope of this paper. In a concurrent work, Wang et al. [64] derive the sample complexity of NeSy predictors when no RSs are in place. It is unclear how these bounds may change when RSs are present.

Relation to disentanglement. Recovering the latent variables G has been the center of several works in representation learning [65, 35, 66]. Among these, achieving identifiability of the latent components has been studied in non-linear independent component analysis [67, 68, 69] and recently in Causal Representation Learning [46, 70, 71, 72]. In this respect, several overlaps exist with the natural mitigation strategies that we investigated: (i) multi-task learning has been shown to increase disentanglement [73] and provably leads to identifiability [74, 75], and (ii) latent variables supervision also largely increases the amount of disentanglement [76, 77, 47]. Our work is the witness that the intersection with disentanglement literature is beneficial for learning the intended concepts in NeSy and, vice-versa, that knowledge-guided learning can be a new way of acquiring identifiable representations, by avoiding RSs.

8 Conclusion

In this work, we provide the first in-depth analysis of RS affecting NeSy predictors. Our analysis highlights four key causes of RS and suggests several mitigation strategies, which we analyze both theoretically and empirically. Our experiments indicate that RSs do naturally appear in both synthetic and real-world NeSy prediction tasks, and that the effectiveness of mitigation strategies is model and task dependent, and that a general recipe for avoiding RSs is currently missing.

We foresee that reasoning shortcuts extend beyond the current scope of NeSy predictors with known prior knowledge. This includes approaches that learn jointly both the concepts and the knowledge, like ROAP [78] and DSL [79], as well as fully neural models, like CBMs [80] and their variants [39, 81], which are also likely to be affected by RSs when concept supervision is limited. We plan to investigate further this direction in the near future.

Ultimately, this work aims at jumpstarting research on the analysis and mitigation of RSs, with the hope of leading to more trustworthy and explainable NeSy architectures.

Broader impact. Our work brings the subtle but critical issue of RSs to the spotlight, and highlights benefits and limitations of a variety of mitigation strategies, but it is otherwise fundamental research that has no direct societal impact.

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their valuable feedback in improving the current manuscript. We acknowledge Emile von Krieken for proofreading our claims and Yoshihide Sawada for his help in deploying the models on BDD-OIA . We are also grateful to Pedro Zuidberg dos Martires for insightful discussions with us at the early stages of the work. We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. The research of ST and AP was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.

References

Luc De Raedt, Sebastijan Dumanˇ ci´ c, Robin Manhaeve, and Giuseppe Marra. From statistical relational to neural-symbolic artificial intelligence. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 4943-4950, 2021.

Artur d'Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Illuminoo, Hoifung Poon, and COPPE Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art , 342:1, 2022.

Eleonora Giunchiglia, Mihaela Catalina Stoian, and Thomas Lukasiewicz. Deep learning with logical constraints. arXiv preprint arXiv:2205.00523 , 2022.

Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan. A review of some techniques for inclusion of domain-knowledge into deep neural networks. Scientific Reports , 12 (1):1-15, 2022.

Michelangelo Diligenti, Marco Gori, and Claudio Sacca. Semantic-based regularization for learning and inference. Artificial Intelligence , 2017.

Ivan Donadello, Luciano Serafini, and Artur D'Avila Garcez. Logic tensor networks for semantic image interpretation. In IJCAI , 2017.

Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS , 2018.

Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for deep learning with symbolic knowledge. In ICML , 2018.

Eleonora Giunchiglia and Thomas Lukasiewicz. Coherent hierarchical multi-label classification networks. NeurIPS , 2020.

Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic Probabilistic Layers for Neuro-Symbolic Learning. In NeurIPS , 2022.

Zenan Li, Zehua Liu, Yuan Yao, Jingwei Xu, Taolue Chen, Xiaoxing Ma, L Jian, et al. Learning with logical constraints but without shortcut satisfaction. In The Eleventh International Conference on Learning Representations , 2023.

Xuan Xie, Kristian Kersting, and Daniel Neider. Neuro-symbolic verification of deep neural networks. 2022.

Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence , 1(5):206-215, 2019.

Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. NeurIPS , 2019.

Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence , 2020.

Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence , pages 1-10, 2021.

Elisa Maiettini, Giulia Pasquale, Vadim Tikhanoff, Lorenzo Rosasco, and Lorenzo Natale. A weakly supervised strategy for learning object detection on a humanoid robot. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids) , pages 194-201. IEEE, 2019.

Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al. Self-driving cars: A survey. Expert Systems with Applications , 165:113816, 2021.

Matt Fredrikson, Kaiji Lu, Saranya Vijayakumar, Somesh Jha, Vijay Ganesh, and Zifan Wang. Learning modulo theories. arXiv preprint arXiv:2301.11435 , 2023.

Robin Manhaeve, Sebastijan Dumanˇ ci´ c, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Neural probabilistic logic programming in deepproblog. Artificial Intelligence , 298: 103504, 2021.

Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea Passerini, and Stefano Teso. Neuro symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal. In ICML , 2023.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev. Dl2: Training and querying neural networks with logic. In International Conference on Machine Learning , pages 1931-1941. PMLR, 2019.

Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck. Neuro-symbolic entropy regularization. In UAI , 2022.

Nick Hoernle, Rafael Michael Karampatsis, Vaishak Belle, and Kobi Gal. Multiplexnet: Towards fully satisfied logical constraints in neural networks. In AAAI , 2022.

Michelangelo Diligenti, Marco Gori, Marco Maggini, and Leonardo Rigutini. Bridging logic and kernel machines. Machine learning , 86(1):57-88, 2012.

Connor Pryor, Charles Dickens, Eriq Augustine, Alon Albalak, William Wang, and Lise Getoor. Neupsl: Neural probabilistic soft logic. arXiv preprint arXiv:2205.14268 , 2022.

Luc De Raedt and Angelika Kimmig. Probabilistic (logic) programming concepts. Machine Learning , 2015.

Adnan Darwiche and Pierre Marquis. A knowledge compilation map. Journal of Artificial Intelligence Research , 17:229-264, 2002.

Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. Advances in Neural Information Processing Systems , 34, 2021.

Robin Manhaeve, Giuseppe Marra, and Luc De Raedt. Approximate inference for neural probabilistic logic programming. In KR , 2021.

Jiani Huang, Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, Le Song, and Xujie Si. Scallop: From probabilistic deductive databases to scalable differentiable reasoning. NeurIPS , 2021.

Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt. DeepStochLog: Neural Stochastic Logic Programming. In AAAI , 2022.

Emile van Krieken, Thiviyan Thanapalasingam, Jakub M Tomczak, Frank van Harmelen, and Annette ten Teije. A-nesi: A scalable approximate method for probabilistic neurosymbolic inference. arXiv preprint arXiv:2212.12393 , 2022.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 2021.

Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning , pages 6056-6065. PMLR, 2019.

Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems , 34: 16451-16467, 2021.

Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning . Mit Press, 2008.

Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretabile, leakproof concept-based models. NeurIPS , 2022.

Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in neural information processing systems , 6, 1993.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International conference on machine learning , 2014.

Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael J Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. In ICLR , 2020.

Eleonora Misino, Giuseppe Marra, and Emanuele Sansone. VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming. NeurIPS , 2022.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML , 2019.

Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning , pages 6348-6359. PMLR, 2020.

Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In ICLR , 2019.

Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems , 32, 2019.

Shuang Li, Yilun Du, Gido Martijn van de Ven, and Igor Mordatch. Energy-based models for continual learning. In Energy Based Models Workshop-ICLR 2021 , 2021.

Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun Zhu. Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In International Conference on Machine Learning , pages 5884-5894. PMLR, 2020.

Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning , pages 23631-23644. PMLR, 2022.

Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems , 33:15288-15299, 2020.

Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regularization. The Journal of Machine Learning Research , 23(1):14632-14662, 2022.

Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access , 10:41758-41765, 2022.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications , 10(1):1-8, 2019.

Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In ICLR , 2020.

Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence , 2(8):476-486, 2020.

Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. Right for the right reasons: training differentiable models by constraining their explanations. In Proceedings of the 26th International Joint Conference on Artificial Intelligence , pages 2662-2670, 2017.

Giambattista Parascandolo, Alexander Neitz, ANTONIO ORVIETO, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations , 2020.

Stefano Teso, Öznur Alkan, Wolfang Stammer, and Elizabeth Daly. Leveraging explanations in interactive machine learning: An overview. Frontiers in Artificial Intelligence , 2023.

Wolfgang Stammer, Marius Memmel, Patrick Schramowski, and Kristian Kersting. Interactive disentanglement: Learning concepts by interacting with their prototype representations. arXiv preprint arXiv:2112.02290 , 2021.

Kaifu Wang, Efi Tsamoura, and Dan Roth. On learning latent models with multi-instance weak supervision. arXiv preprint arXiv:2306.13796 , 2023.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8): 1798-1828, 2013.

Aapo Hyvärinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning. Patterns , 4(10), 2023.

Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In AISTATS , 2020.

Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable nonlinear independent component analysis. Advances in Neural Information Processing Systems , 35:16946-16961, 2022.

Luigi Gresele, Julius V on Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, a new concept? Advances in neural information processing systems , 34:28233-28248, 2021.

Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International conference on machine learning , pages 372-407. PMLR, 2023.

Wendong Liang, Armin Keki´ c, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. arXiv preprint arXiv:2305.17225 , 2023.

Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Keki´ c, Elias Bareinboim, David M Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. arXiv preprint arXiv:2306.00542 , 2023.

Łukasz Maziarka, Aleksandra Nowak, Maciej Wołczyk, and Andrzej Bedychaj. On the relationship between disentanglement and multi-task learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 625-641. Springer, 2022.

Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: generalization and identifiability in multi-task learning. In International Conference on Machine Learning , pages 18171-18206. PMLR, 2023.

Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. arXiv preprint arXiv:2304.07939 , 2023.

Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations , 2020.

Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit B Patel, and Anima Anandkumar. Semi-supervised stylegan for disentanglement learning. In Proceedings of the 37th International Conference on Machine Learning , pages 7360-7369, 2020.

Hao Tang and Kevin Ellis. From perception to programs: regularize, overparameterize, and amortize. In International Conference on Machine Learning , pages 33616-33631. PMLR, 2023.

Alessandro Daniele, Tommaso Campari, Sagar Malhotra, and Luciano Serafini. Deep symbolic learning: Discovering symbols and rules from perceptions. arXiv preprint arXiv:2208.11561 , 2022.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning , pages 5338-5348. PMLR, 2020.

Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209.09056 , 2022.

Francesco Giannini, Michelangelo Diligenti, Marco Gori, and Marco Maggini. On a convex logic fragment for learning and reasoning. IEEE Transactions on Fuzzy Systems , 2018.

Thomas M Cover. Elements of information theory . John Wiley & Sons, 1999.

Kareem Ahmed, Kai-Wei Chang, and Guy Van den Broeck. A pseudo-semantic loss for deep generative models with logical constraints. In Knowledge and Logical Reasoning in the Era of Data-driven Learning Workshop , July 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Tommaso Carraro. LTNtorch: PyTorch implementation of Logic Tensor Networks, mar 2022. URL https://doi.org/10.5281/zenodo.6394282 .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980 .

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015.

A Other NeSy Predictors

In this appendix, we outline the NeSy prediction approaches used in our experiments and then show that they share deterministic RSs as DPL.

The semantic loss (SL) [8] is a penalty term that encourages a neural network to place all probability mass on predictions that are consistent with prior knowledge K . In our setting, the SL is applied to a predictor p θ ( Y | C ) placed on top of a concept extractor p θ ( C | X ) and it can be written as:

SL ( p θ , ( x , y ) , K ) := -log ∑ c ✶ { ( c , y ) | = K } p θ ( c | x ) (11) Like DPL, the SL relies on knowledge compilation to efficiently implement Eq. (11). Importantly, if the distribution p θ ( c | x ) allocates mass only to concepts c that satisfy the knowledge given y , these are optimal solutions, in the sense that the SL is exactly zero . We will make use of this fact in Appendix A.1.

During training, the SL is combined with any regular supervised loss ℓ , for instance the cross-entropy, leading to the overall training loss:

with µ > 0 a hyperparameter. During inference, the SL plays no role: the predicted label is obtained by simply taking the most likely configuration through a forward pass over the network.

Logic tensor networks (LTNs) [6] are another state-of-the-art NeSy architecture that combines elements of reasoning- and penalty-based approaches. The core idea behind LTNs, and of all other NeSy predictors based of fuzzy logic [82], is to relax the prior knowledge K into a real-valued function T [ K ] quantifying how close a prediction is to satisfying K . LTNs perform this transformation using product real logic [6]. In our context, this function takes the form T [ K ] : [0 , 1] k × [0 , 1] n → [0 , 1] , and it takes as input the probabilities of the various concepts C and labels Y and outputs a degree of satisfaction.

Crucially, fuzzy logics are designed such that, if all probability mass is allocated to configurations ( c , y ) that do satisfy the logic, then the degree of satisfaction is exactly 1 , i.e. , maximal. We will leverage this fact in Appendix A.1.

During training, LTNs guide the concept extractor p θ ( C | X ) towards predicting concepts that satisfy the prior knowledge K by penalizing it proportionally to how far away their predictions are from satisfying K given the ground-truth label y , that is, 1 -T [ K ]( p ( C | x ) , ✶ { Y = y } ) . During inference, LTNs first predict the most likely concepts ˆ c = argmax c p θ ( c | x ) using a forward pass through the network, and then predict a label ˆ y that maximally satisfies the knowledge given ˆ c , again according to T [ K ] .

A.1 Deterministic Optima are Shared

The bulk of our theoretical analysis focuses on DPL because it offers a clear probabilistic framework, however in the following we show that deterministic optima do transfer to other NeSy predictors approaches. Specifically, under A2 , DPL, SL, and LTN all admit the same deterministic optima (det-opts).

To see this, let C y be the set of concepts vectors c from which the label y can be inferred, that is, C y = { c ∈ C : p ∗ ( y | c ) > 0 } . Now, take a deterministic RS p θ ( C | X ) for DPL, i.e. , a concept distribution that satisfies Definition 1 when the likelihood is computed as in Eq. (2). By determinism, this distribution can be equivalently written as a function mapping inputs x to concept vectors ˆ c ( x ) ∈ C , that is, p θ ( C | X = x ) = ✶ { C = ˆ c ( x ) } . At the same time, by optimality and A2 we have that, for every ( x , y ) ∈ D , the NeSy predictor p θ ( Y | X ; K ) = ∑ c u K ( Y | c ) · p θ ( c | X ) allocates all probability mass to the correct label y . As a consequence, ˆ c ( x ) ∈ C y , for every ( x , y ) ∈ D .

Consider a NeSy predictor that first predicts concepts using p θ ( C | X ) and then labels using a reasoning layer based on fuzzy logic, such that a prediction ˆ y ∈ Y is chosen so that it minimial distance from satisfaction w.r.t. K , and the distance from satisfaction is also used as a training objective. Since ˆ c ∈ C y for every ( x , y ) ∈ D , by definition of T-norms, the label with minimal distance from satisfaction will necessarily be y : all other labels cannot be inferred from the knowledge,

so they have larger distance from satisfaction. Hence, training loss will be minimal for this predictor as well, meaning that p θ ( C | X ) is a deterministic RS for it too.

Next, consider a NeSy penalty-based predictor where predictions are obtained by first inferring concepts using MAP over the concept extractor, that is, ˆ c = argmax c p θ ( c | X ) , and then performing a forward pass over a neural prediction layer p θ ( Y | C ) . By construction, the penalty will be minimal whenever ( y , ˆ c ( x )) | = K . We already established that this is the case for all ( x , y ) ∈ D , meaning that p θ ( C | X ) is a deterministic RS for penalty-based predictors too.

B Proofs

In the proofs, we suppress K from the notation for readability.

B.1 Proof of Lemma 1: Upper Bound of the Log-Likelihood

Proof plan. The proof of the first point of our claim is split in three parts:

We show that in expectation the log-likelihood in Eq. (2) is upper bounded by a term containing the KL divergence for p θ ( Y | G ) .

We prove that, under A1 , any optimum of Eq. (2) minimizes also the KL. In this step, we make use of Information Theory [83] to connect the two.

We show that assuming A1 and A2 the optima p θ ( Y | G ) for the KL are given only by optima p θ ( Y | X ) for Eq. (2).

We proceed to prove the second point by leveraging the fact that p θ ( C | G ) is given by marginalizing p θ ( C | X ) over the generating distribution p ∗ ( X | G , S ) .

Point ( i ). (1) We upper bound the log-likelihood in Eq. (1) as follows:

In the second line, we introduced the distribution p ∗ ( x | g ) := E s ∼ p ( S ) [ p ∗ ( x | g , s )] , in the third line we applied Jensen's inequality, and in the fifth line we added and subtracted p ∗ ( y | g ) . This proves our claim.

(2) We proceed showing that for a deterministic p ∗ ( X | G , S ) every optimum p θ ( Y | X ) for the log-likelihood leads to an optimum p θ ( Y | G ) for the RHS of Eq. (29). We can rewrite the joint distribution as:

where p ∗ ( y | x ) = E g ∼ p ∗ ( g | x ) [ p ∗ ( y | g )] and p ∗ ( G | x ) is the posterior underlying the data generative process. Hence, the log-likelihood in Eq. (1) can be equivalently written as:

In the first line, we used p ( X , Y ) = p ∗ ( Y | X ) p ( X ) , and then added and subtracted log p ∗ ( y | x ) . By comparing Eq. (18) and Eq. (27), we obtain:

Now, consider a distribution p θ ( Y | X ) that attains maximal likelihood. Then, KL [ p ∗ ( y | x ) ∥ p θ ( y | x )] = 0 , and we can rearrange the inequality in Eq. (28) to obtain:

Here, H [ Y | X ] = E x ∼ p ( X ) [ H [ p ∗ ( Y | x )] and H [ Y | G ] = E g ∼ p ( G ) H [ p ∗ ( Y | g )] are conditional entropies.

We want to show that, under A1 , the right-hand side of Eq. (29) is in fact zero. As for the other conditional entropy term, recall that [83]:

where I [ · , · ] is the mutual information. By the chain rule of the mutual information, cf. [83, Theorem 2.8.1], we have:

where I [ · , · | · ] is the conditional mutual information. The structure of our generative process in Fig. 1 implies that I [ X : Y | G ] = 0 , so Eq. (31) boils down to:

Plugging this into Eq. (32) entails that I [ X : Y ] = I [ Y : G ] , or equivalently that H [ Y : X ] = H [ Y : G ] . This means that the right-hand side of Eq. (29) is indeed zero, which entails that the KL is zero and that therefore p θ ( Y | G ) optimizes the right-hand side of Eq. (5).

(3) We proceed showing that by assuming also A2 whatever optimum p θ ( Y | G ) is identified by p θ ( Y | X ) that is also optimum. First, note that under A2 p ∗ ( Y | G ) is deterministic, so we have H [ Y | G ] = 0 . With A1 and A2 , both p ∗ ( y | g ) and p ∗ ( y | x ) are deterministic and therefore Eq. (28) can be rewritten as:

In particular, for each x the maximum of the log-likelihood is 0 and it is attained when the label probability is one. We make use of this observation in the next step.

Next, we show that the bound in Eq. (28) is in fact tight , in the sense that whenever p θ ( Y | G ) maximizes the left-hand side, p θ ( Y | X ) maximizes the right-hand side. We proceed by contradiction. Fix g and let O be the set of those s for which p θ ( y | x = f ( g , s )) < 1 and assume that it has

By A1 , we have that:

non-vanishing measure. Then, the posterior distribution is also strictly less than one:

Therefore, there cannot be any optimal solutions p θ ( Y | G ) that are given by non-optimal probabilities p θ ( Y | X ) . This proves the claim.

Point ( ii ). We consider now which distributions p θ ( C | G ) correspond to a unique distribution p θ ( C | X ) . First, we define as P the set of candidate distributions p φ ( C | X ) , with φ ∈ Θ , for which it holds:

̸

For all distributions of the form p θ ( C | G ) = ✶ { C = c } , for c ∈ C , P restricts to a single element, i.e., p φ ( C | x ) = ✶ { C = c } . We proceed by contradiction and consider a set O X of non-vanishing measure such that: p φ ( C | x ) = { C = c } , ∀ x ∈ O X (43)

✶ Let p ∗ ( X | G ) = E s ∼ p ∗ ( S ) p ∗ ( X | G , S ) . Then, it holds that:

✶ where we denoted ˜ p φ ( C | G ) the normalized probability distribution given by integrating p φ ( C | X ) on O X solely, and λ is the measure of O X . Notice that the RHS of Eq. (44) is exactly ✶ { C = c } iff λ = 0 or ˜ p φ ( C | G ) = ✶ { C = c } , which contradicts the claim. Hence, all probabilities p θ ( C | G ) = ✶ { C = c } are only given by probabilities p θ ( C | x ) = ✶ { C = c } , for all x ∈ X . This yields the claim.

B.2 Proof of Theorem 2: Counting the Deterministic Optima

We want to count the number of deterministic optima of the log-likelihood in Eq. (1). Recall that, by Lemma 1, under A1 and A2 any optimum p θ ( Y | X ) of Eq. (2) yields an optimum p θ ( Y | G ) of the upper bound in Eq. (5). Following, by point ( ii ) of Lemma 1 we have that deterministic optima are shared between p θ ( C | X ) and p θ ( C | G ) . This means that we can equivalently count the number of deterministic optima p θ ( C | G ) for the upper bound in Eq. (5). We proceed to do exactly this.

Let A be the set of all possible maps α : g ↦→ c , each inducing a candidate concept distribution p θ ( C | G ) = ✶ { C = α ( G ) } . The only α 's that achieve maximal likelihood are those that satisfy the knowledge for all g ∈ supp ( G ) for the learning problem, that is:

i.e. , that it is indeed the case that the concepts output by α ( g ) predict the ground-truth label h K ( g ) . Notice that, only one of them is correct and coincides with the identity, i.e. , α ( g ) = g .

These α are those that satisfy the knowledge on all examples, or equivalently the conjunction of the knowledge applied to all examples, that is:

This means that only a subset of A contains those maps consistent with the knowledge. The total number of these maps is then given by:

This yields the claim.

B.3 Proof of Proposition 3: Link Between Deterministic and Non-deterministic Optima

Point ( i ) . We begin by proving that, for DPL, any convex combination of optima of the likelihood Eq. (1) is itself an optimum. Fix any input x . First, p ∗ ( y | x ) is the optimal value of the log-likelihood in Eq. (2), according to Eq. (29) from point ( i ) of Lemma 1. Now, let p (1) ( C | x ) and p (2) ( C | x ) be two concept distributions that both attain optimal likelihood, i.e. , for i ∈ { 1 , 2 } it holds that ∑ c u K ( y | c ) · p ( i ) ( c | x ) = p ∗ ( y | x ) . The likelihood term of any convex combination of the two optima is given by:

Hence, the convex combination is also an optimum of the likelihood. Note that the very same reasoning applies to the Semantic Loss (Eq. (11)), again due to linearity of the expectation over C .

Remark 7. Since SL and DPL limit p θ ( C | X ) to be factorized as ∏ i p ( C i | X ) , some convex combinations that would be in principle solutions as per point (i) cannot be expressed. Essentially, this translates into an additional constraint for the solutions of SL and DPL that is, given two factorized probabilities p (1) ( C | x ) and p (2) ( C | x ) , then:

This applies only to models that integrate probabilistic logic by predicting the concepts independently, whereby relaxations of SL and DPL, like [10, 84], could naturally express arbitrary convex combinations of deterministic RSs.

Remark 8. The above result does not hold for LTNs, in general. We show this by constructing a counter-example for the default choice of T-conorms [6]. That is, a convex combination of deterministic optima that is not itself an optimum. Recall that LTN uses product real logic to define a degree of knowledge satisfaction and consider the prior knowledge K = ( C 1 ∨ C 2 ) , where C 1 and C 2 are two distinct binary concepts. In LTN, the degree of satisfaction of K is given by the T-conorm of the logical disjunction, which is defined as S ( a, b ) = a + b -ab . Fix an input x and consider two deterministic distributions p (1) ( C | x ) = ✶ { C = (1 , 0) ⊤ } and p (2) ( C | x ) = ✶ { C = (0 , 1) ⊤ } . It is clear that both distributions satisfy the prior knowledge and as such are optimal. Now take any convex combination p ( C ) = λp (1) ( C ) + (1 -λ ) p (2) ( C ) . Then, a = λp (1) ( C 1 = 1) + (1 -λ ) p (2) ( C 1 = 1) = λ and, for similar reasons, b = 1 -λ . Then, it holds:

where the equality holds iff λ ∈ { 0 , 1 } .

Point ( ii ). Under A1 and A2 , we can count all deterministic solutions p θ ( C | G ) via Theorem 2. Here, we show that these deterministic optima constitute a complete basis for all optimal solutions of Eq. (29). From ( i ), we have that any convex combination of the deterministic optima is also an optimum. We will show that these are the only optimal solutions.

Recall that the optimal solutions are all of the form p ∗ ( Y = β K ( g ) | g ) = 1 , as a consequence of A1 and A2 from Lemma 1, for all g ∈ supp ( G ) . Notice that any optimal solution p θ ( C | g ) must place

mass to those concepts that lead to the correct label. Formally, for every g and y = β K ( g ) , it holds:

where the equality holds iff u K ( y | c ) = 1 for all p θ ( c | x ) > 0 . In other words, this means that each c ∼ p θ ( C | g ) must be an optimal solution for the logic. This proves the claim.

Point ( iii ) . If A2 does not hold, there can be optima solutions that are given as convex combinations of non-optimal deterministic probabilities. First, notice that according to Lemma 1, point ( i ), the optimal solutions for p θ ( Y | X ; K ) minimize the KL term and are equivalent to p ∗ ( Y | X ) . Then, consider for a given x an optimal solution p ( C | x ) = λ ✶ { C = c 1 } +(1 -λ ) ✶ { C = c 2 } that is a convex combination of two deterministic probabilities. From the convexity of the KL it holds: KL [ p ∗ ( y | x ) || λu K ( y | c 1 ; K ) + (1 -λ ) u K ( y | c 2 ; K )] (57)

where the equality holds iff u K ( y | c 1 ; K ) = u K ( y | c 2 ; K ) = p ∗ ( y | x ) . This shows that there can exist solutions that are convex combinations of non-optimal deterministic probabilities . We proceed to show how the space of optimal solutions is defined. On the converse, if u K ( y | c 1 ; K ) ✶ { C -c 1 } is a solution, also u K ( y | c 2 ; K ) { C -c 2 } must be a solution:

We now show that any optimum given by a generic p φ ( C | x ) , with φ ∈ Θ , can be expressed as a convex combination of (1) an optimum that is a convex combination of deterministic optima and (2) an optimum that is a convex combination of deterministic, but non-optimal, probabilities:

where C y = { c ∈ C : u K = p ∗ ( y | x ) } and S c y = C / C y are two disjoint sets. In the second line, we rewrote the summation on the two terms considering the two sets, whereas in the third line we introduced: λ = ∑ c ∈C y p ( c | x ) and ˜ p ( c | x ) = p ( c | x ) /λ , 1 -λ = ∑ c ∈S ¯ y p ( c | x ) and ¯ p ( c | x ) = p ( c | x ) / (1 -λ ) . Since each ˜ p ( c | x ) lead to an optimum by construction, it must be that also ¯ p ( c | x ) is an optimum, by the previous point. In general, there can be many ¯ p ( c | x ) leading to optima, even when ˜ p ( c | x ) reduces to only the ground-truth element, i.e. , ˜ p ( c | x ) = ✶ { C -f -1 1: k ( x ) } . Notice that this must hold for all x ∼ p ∗ ( X ) . This proves the claim. B.4 Proof of Proposition 4: Multi-task Learning

When considering multiple task T , suppose that, for each x ∈ ⋂ t D t , we get T different labels y (1) , . . . , y ( T ) , each given in accordance to knowledge K (1) , . . . , K ( T ) , respectively. Similarly to Eq. (2), we consider the learning objective for all tasks with the joint log-likelihood term:

Under A1 and A2 , Lemma 1 point ( i ) holds for each task t . In the following, we denote with β ( t ) K the underlying maps for each K ( t ) , which by A2 are deterministic. Hence, the learning objective becomes:

and, similarly to point ( i ) of Lemma 1, we get the following upper-bound:

Notice that the optimal values for Eq. (69) are given by those distributions that are one for each term in the summation. Then, following from Theorem 2, we have that the deterministic maps α 's that optimize Eq. (69) must be consistent with each task t and for each g ∈ supp ( G ) satisfy:

which, equivalently to Theorem 2, points to the condition that the maps α must be consistent with solving all tasks t . This exactly amounts to solving the conjunction of all knowledge K = ∧ t K ( t ) . This yields the claim.

B.5 Proof of Proposition 5: Concept Supervision

Let p ∗ ( G | S ) be the distribution of ground-truth concepts restricted to a subset S ⊆ supp ( G ) :

and p ∗ ( X | S ) = E ( g , s ) ∼ p ∗ ( G |S ) p ∗ ( S ) p ∗ ( X | g , s ) be the corresponding restricted input distribution. Under A1 , the expectation of the log-likelihood term for concept supervision in Section 5 is:

where p ∗ ( X | S ) = E g ∼ p ∗ ( G |S ) E s ∼ p ∗ ( S ) p θ ( X | g , s ) . In the first line we made use of A1 to write x = f ( g , s ) , in the second line we used Jensen's inequality, and then we introduced the marginal distribution p θ ( C | G ) = E s ∼ p ∗ ( s ) [ p θ ( C | f ( G , s ))] . Recall that I denotes the subset of the supervised ground-truth concepts and that the log-likelihood is evaluated only w.r.t. those concepts.

Now notice that, given A1 , the maximum for the LHS of Eq. (72) is zero, i.e. , coincides with maximum log-likelihood. Consistently the RHS of Eq. (72) is zero only when p θ ( C I | g ) places all mass on g I . We proceed considering those maps α : g ↦→ c which lead to deterministic, optimal solutions for the RHS. For these, it must hold that for each g ∈ S :

When taken all together, we can estimate how many α ∈ A (cf. Appendix B.2) satisfy the above condition for all g :

This yields the claim.

B.6 Proof of Proposition 6: Reconstruction

Under A1 , the reconstruction penalty can be written as:

From A3 , we have that the both encoder and decoder distributions are factorized:

This yields:

where in the first row we separated the two logarithms and removed the expectations on p θ ( Z | s ) and p θ ( C | g ) for the terms p ψ ( g | c ) and p ψ ( s | z ) , respectively. In the third line, we discarded the term with the match of the reconstruction of the style, giving the lower bound.

When R goes to zero, the lower bound also goes to zero and the last term of Eq. (79) is minimized. This happens whenever, for each c ∼ p θ ( c | g ) , it holds that p ψ ( g | c ) = 1 . This condition also implies that if for two different g ′ and g ′′ there exist at least one c such that p θ ( c | g ′ ) · p θ ( c | g ′′ ) > 0 , then p ψ ( g ′ | c ) and p ψ ( g ′′ | c ) cannot both be optimal.

We restrict now to a deterministic map α : g ↦→ c for p θ ( C | G ) and describe the condition when optimal decoders are attained, i.e. , p ψ ( g | c ) = 1 for some c . By the previous argument, an optimal map α that leads to optimal decoders must always map ground-truth concepts g to different concepts c , that is:

̸

̸

In particular, this condition must hold for all different g ∈ supp ( G ) . The number of all solutions is then given by:

̸

̸

as claimed.

C Experimental Details and Further Results

We report here all further details concerning the experiments in Section 6.

C.1 Implementation

The code of the experiments builds on top of nesy-cl [21] and CBM-AUC [55]. All experiments are implemented with Python 3.8.16 and Pytorch [85] and run over one A100 GPU. The code for the experiments is available at github.com/reasoning-shortcuts.

The implementation of DPL is taken from [21] and follows exactly Eq. (1), where to each world a label is assigned accordingly to the prior knowledge K . We implemented SL following the original paper [8], with the only difference that the prediction of the labels Y is done on top of the logits of p θ ( C | x ) . This relaxes the conditional independence between labels and concepts while being more in line with the generative process we assumed. The implementation of LTN is adapted from LTN-pytorch [86] and is based on the satisfaction loss introduced in Appendix A.

C.2 Data sets & Count of the Reasoning Shortcuts

Here, we illustrate how to count explicitly the number of deterministic RSs using the equations in Table 1, restricting ourselves to the case where no disentanglement is in place, for simplicity. Similarly to Section 4, we assume A1 and A2 and that supp ( G ) = C = G , and count the total number of deterministic optima (or det-opt s for short) under different mitigation strategies. In the following, C y ⊆ C = G refers to the set of c ∈ C or g ∈ G that are mapped by K to label y ∈ Y . Note that ∑ y |C y | = |G| .

Explicit count for the likelihood. When the concept extractor p θ ( C | X ) is sufficiently expressive, the deterministic mappings α are essentially arbitrary. Specifically, the set A of these α 's includes all functions from G to C . These functions can be explicitly enumerated by counting how many ways there are to map each input vector g to an arbitrary vector c .

Here, we are interested in counting the number of α 's that attain optimal likelihood. Each such α has to ensure that each g ∈ G is mapped to a c ∈ C β K ( g ) that yields the correct label, or in short, ∀ g . α ( g ) ∈ C β K ( g ) . This condition is satisfied if and only if, for every y in the data set, every g ∈ C y is mapped to a c that is also in C y , and there are exactly |C y | |C y | ways to do this. This immediately shows that the overall number of det-opt s α is:

This associates an explicit number to Theorem 2.

Explicit count for the reconstruction. The effect of adding a reconstruction penalty is that now, in order to achieve optimal loss, α has to map different g 's to distinct concepts c 's. Unless this is the case, it becomes impossible to reconstruct the ground-truth concepts from the learned ones, and therefore the input x generated from them.

The resulting computation is the same as above, except that now we have to associate the different g 's to different c 's without replacement . This yields the following count:

̸

̸

Explicit count for concept supervision. For the combination of logic and concept supervision, we suppose that for each equivalence class C y , there are ν y ground-truth concepts provided with supervision and that all concept dimensions C i receive this supervision. Let S ⊂ G be the set of supervised concepts, such that |S| = ∑ y ∈Y ν y . From above, this means that we are specifying a total of |S| ground-truth concepts. From Table 1, we have that the number of det-opt s amounts to:

✶ ✶ Finally, combining together the terms of prediction, reconstruction, and concept supervision, we obtain:

̸

̸

The disentangled case. If the network is disentangled , the enumeration procedure becomes substantially more complicated and cannot be written compactly in closed form. While the number of deterministic optima α can still be computed exactly using model counting [29, 30], doing so is not necessary for the scope of our paper and therefore left to future work.

C.2.1 Dataset: XOR

This dataset, introduced in Example 3, is a toy data set containing 3 bits g = ( g 1 , g 2 , g 3 ) , for a total of 8 possible combinations. The task consists in predicting the XOR operation among them, namely y = ( g 1 ⊕ g 2 ⊕ g 3 ) . The dataset is exhaustive and has no validation and test set. The model performances are evaluated on the training set.

Reasoning shortcut. For this dataset, we have |C 0 | = |C 1 | = 4 . RSs arise depending on the structure of the underlying network. When the ground-truth concepts are processed all together without any mitigation strategy, we obtain that the number of det-opt s amounts to:

The confusion matrices for all methods are reported in Appendix C.5. On the other hand, we show that in the disentangled case, only two combinations suffice to identify the correct solution, e.g. ,

and here the only viable solution for the two is α (0) = 0 and α (1) = 1 . This condition is met in all our experiments in Table 2.

C.2.2 Dataset: MNIST-Addition

We consider the version introduced in [7], which consists of couples of digits, each ranging from 0 to 9 , and the target consists in predicting the correct sum, i.e. , y = g 1 + g 2 . This data set contains all possible combinations, for a total of 100 . The training set contains 42 k data, the validation set 12 k, and the test set 6 k.

Reasoning shortcuts. RSs arise only as a result of the joint prediction of both digits. Notice that the number of elements |C y | for each sum y can be evaluated as:

Therefore, the total number of det-opt s amounts to:

When providing disentanglement , the number of RSs reduce to 0 , as it sufficient to have the sums:

to uniquely identify the value of the digit c .

C.2.3 Dataset: MNIST-EvenOdd

This data set, proposed by [21], is a biased version of MNIST-Addition , where only some combinations of the digits appear. We consider here a more challenging scenario w.r.t. the proposed version, consisting of the sums:

Overall, the training set contains 6720 data, the validation set 1920 , and the test set 960 .

Reasoning shortcuts. We describe the RSs that arise even when the architecture incorporates disentanglement . We evaluate the possible RSs empirically noticing that the system of observed sums can be written as a linear system, as done by Marconato et al. [21]:

Now, notice that we can find independent reasoning shortcuts for each of the two sides since they do not share any digits. For the LHS, we consider the sum α (2) + α (8) = 10 and notice that we can find at most 10 different attributions for having a correct sum. Notice that, some of them are not allowed as α (8) = 0 , 1 leads to inconsistent values for the fourth sum, and α (8) = 3 leads to an inconsistent sum for the first equation. So in total, for the LHS, we obtain 7 possible solutions and, by symmetry, the same number also for the LHS. In total, the number of det-opt s is equal to 7 · 7 .

Experimentally, we consider limited concept supervision on I = { 4 , 9 } which should be sufficient, in principle, to disambiguate between even and odds digits. This happens because specifying α (4) = 4 and α (9) = 9 admits only the ground-truth concepts as the optimal solution.

C.2.4 Dataset: MNIST-AddMul

In this dataset, we consider fewer combinations of digits, explicitly:

and similarly for multiplication. This data set contains 1680 training examples, 480 for the validation, and 240 for the test set.

Reasoning shortcuts. For the case of the addition task, we can have 2 possible solutions, which are:

α (0) = 0 , α (1) = 1 , α (2) = 2 , and α (3) = 3 ;

α (0) = 1 , α (1) = 0 , α (2) = 1 , and α (3) = 4 .

For multiplication, we have that since α (1) · α (3) = 3 it can be either α (1) = 1 or α (1) = 3 . In both cases, it holds that α (0) = 0 and α (2) can be arbitrary. Hence, there are in total 2 · 4 possible det-opt s. In MTL, the reasoning shortcut for the addition does not hold since it leads to a sub-optimal solution for multiplication.

C.2.5 Dataset: BDD-OIA

This data set contains frames of driving scene videos for autonomous predictions [54]. Each frame is annotated with 4 binary labels, indicating the possible actions, Y = ( move_forward , stop , turn_left , turn_right ) . Each scene is also annotated with 21 binary concepts C , underlying the reasons for the possible actions, see Table 5. The training set contains 16 k frames, with full label and concept supervision; the validation and the test set contain 2 k and 4 . 5 k annotated data, respectively.

For designing the prior knowledge w.r.t. to the concepts in Table 5, we make use of the following rules for move_forward / stop predictions :

For turn_left , and similarly for turn_right , we use:

Notice that, since the concepts are predicted together, as explained in Appendix C.4, we can count the number of RSs as follows:

For move_forward and stop , the labels are predicted with the constraints such that ( move_forward , stop ) = (1 , 1) has no support. Hence, we consider only the predictions (0 , 0) , (0 , 1) , (1 , 0) . Next, we identify:

-|C 0 , 0 | = 1 , since it corresponds just to the case where no concepts are predicted.

-|C (1 , 0) | = 2 3 -1 , which is the number of different concepts attribution for forward yielding a positive label;

-|C (0 , 1) | = 280 are the combination of the remaining concepts that yield the stop action, in agreement with the constraints. These were counted explicitly from the logic implementation.

Overall, the number of det-opt s amount to 1 · 7 7 · 280 280 .

For turn_left and turn_right , we count the cardinality of positive and negative predictions of the two classes |C 0 | and |C 1 | :

-|C 1 | = 2 3 -1 , that are the only concepts attributions for the positive label;

-|C 0 | = 2 6 -|C 1 | , are all the remaining concept combinations.

For left and right, separately, we obtain that possible optimal solutions amount to 7 7 · 57 57 .

Altogether, the count of det-opt s for BDD-OIA goes as follows:

C.3 Optimizer and Hyper-parameter Selection

The Adam optimizer [87] was employed for all experiments, with exponential decay of the learning rate amounting to γ = 0 . 95 , exception made for BDD-OIA , where we added a weight-decay with parameter ω = 4 · 10 -5 and γ was set to 0 . 1 , to avoid over-fitting.

The learning rate for all experiments was tuned by searching over the range 10 -4 ÷ 10 -2 , with a total of 5 log steps. We found the SL penalty of 2 and 10 for XOR and MNIST-AddMul , respectively, to work well in our experiments. The strength of the single mitigation strategies ( η ) for each method was chosen accordingly to a grid-search over η ∈ { 0 . 1 , 0 . 5 , 1 , 2 , 5 , 10 } , varying the learning rate. The best hyper-parameters were selected, in the first step, based on the highest performances in F 1 -score for label accuracy on the validation set and, in the second step, on the lowest mitigation loss, accordingly to the value in the validation set. In particular, for SL we chose those runs that yielded the best trade-off between SL minimization and label prediction. For LTN, we found that concept supervision did interfere with the original training objective, for which the best weight for the mitigation strength was found at γ = 10 -2 . It was adopted for both LTN+C and LTN+R +C in Q2 of Section 6. The hyper-parameters for the combined mitigation strategies were selected according to the aforementioned criterion, by only searching the best-combined mitigation strength, while keeping the individual best strengths fixed from the previous grid search.

For BDD-OIA , we selected the learning rate ranging in the interval 10 -4 ÷ 10 -2 upon selecting those runs with best F 1 -scores on labels in the validation set. The strength of the concept supervision and entropy regularization were varied in between { 0 . 1 , 1 , 5 } .

All best hyper-parameters for our tests are reported in the code in the Supplementary Material.

C.4 Architectures

XOR : For this data set, we adopted two MLPs, one for the encoder p θ ( C | G ) and one for the decoder p ψ ( G | C ) , both with a hidden size of 3 and ReLU activations. This architecture is used to empirically validate RSs without forcing disentanglement. For the disentangled case, we considered a linear layer with weight ω and bias b .

For SL only, we added an additional MLP, with a hidden size of 3 and tanh activations, implementing the map from the logits of C to Y .

MNIST-AddMul : We report here the architectures that has been used for MNIST-Addition , MNIST-Multiplication , and MNIST-EvenOdd . For the joint prediction, i.e. , without disentanglement , we used the encoder in Table 6. When considering disentanglement , we processed each digit with the encoder in Table 7 and then stacked the two concepts together. For the reconstruction, we used the decoder in Table 8. For SL only, we added an MLP with a hidden size of 50 , taking as input the logits of both concepts and processing them to the label.

BDD-OIA : Images of BDD-OIA are preprocessed following [55] with a Faster-RCNN [88] pre-trained on MS-COCO and fine-tuned on BDD-100k [54]. Successively, we adopted the pre-trained convolutional layer on [55] to extract linear features, with dimension 2048 . These are the inputs for the NeSy model, which is implemented with a fully-connected NN, see Table 9.

C.5 Confusion Matrices

Following, we report the label and concept-level confusion matrices (CMs) for XOR , MNIST-EvenOdd , and MNIST-Multiplication . For all of them, we report those obtained in runs with maximal F 1 -score over the labels.

C.6 XOR

NONE

DIS

(0,0,0)

(0,0,1)

(0,1,0)

(0,1,1)

(1,0,0)

(1,0,1)

(1,1,0)

(1,1,1)

(0,0,0)

(0,0,1)

(0,1,0)

(0,1,1)

(1,0,0)

(1,0,1)

(1,1,0)

(1,1,1)

(0,0,0)

(0,0,0)

(0,0,1)

(0,0,1)

(0,1,0)

(0,1,0)

(0,1,1)

(0,1,1)

(1,0,0)

(1,0,0)

(1,0,1)

(1,0,1)

(1,1,0)

(1,1,0)

(1,1,1)

(1,1,1)

DPL

SL

LTN

DPL

SL

LTN

C.7 MNIST-EvenOdd

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

0123456789

0123456789

Ground-truth

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

0123456789

0123456789

Ground-truth

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

0123456789

0123456789

Ground-truth

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

0123456789

0123456789

Ground-truth

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

0123456789

01２3456789

Ground-truth

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

0123456789

01２3456７8９

Ground-truth

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

0１２３４56７８９

0123456789

Ground-truth

Predicted

Predicted

Predicted

0123456789

0123456789

0123456789

0123456789

01２3456７8９

01２3456７89

Ground-truth

C.8 MNIST-AddMul

Predicted

Predicted

Predicted

0

1

2

3

1

2

m

1

2

3

Ground-truth

Predicted

Predicted

Predicted

0

1

2

3

1

2

3

1

2

3

Ground-truth

Predicted

Predicted

Predicted

1

2

3

1

2

3

1

2

3

Ground-truth

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reasoning shortcuts (RS) in neuro-symbolic learning. RS refers to that the neural network does not learn a generalizable concept. This work provides a systematic characterization of RS, which not only gives a formal definition of RS, but also identifies key conditions of its occurrence. Several mitigation strategies are empirically or theoretically analyzed.\n\nSTRENGTHS:\n- The paper is well-motivated and well-written. I particularly appreciate the illustrative example in Figure 1. \n- The definition of RS is derived from causal representation learning, and it is quite reasonable.\n- Some promising theoretical results are presented, with proofs provided in the appendix.\n\nWEAKNESSES:\n- The paper focuses solely on probabilistic logic approaches like DPL, which limits its scope to a relatively narrow scope of neuro-symbolic learning. \n- This paper introduces too many concepts that may be unnecessary, such as NeSy predictors, unintended semantics, concept extractors, concept distributions, deterministic distributions, etc. Therefore, it may not be easily understood by readers who are unfamiliar with this subfield. \n- Some more experiments on other baseline methods are needed.\n\nQUESTIONS:\n- The discussed neuro-symbolic learning is essentially in a weakly supervised learning setting (correct me if I'm wrong) [1]. Therefore, can the RS and its analysis be extended to general supervised learning problems?\n- From my understanding, using Shannon entropy loss is critical to avoid shortcuts. As mentioned in the last paragraph of section 5.4, entropy loss conflicts with NeSy prediction, but [1] proposes using an annealing strategy to gradually remove this entropy regularization. Furthermore, there are quite a few classic methods using maximal entropy-like regularization, e.g., mixup [2], label smoothing [3], focal loss [4], energy-based models [5], and Logit normalization [6]. I strongly recommend separating this part as an additional subsection to reveal this clue for alleviating RS.\n\n[1] Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, Song-Chun Zhu. Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning.\n\n[1] Zenan Li, Yuan Yao, Taolue Chen, Jingwei Xu, Chun Cao, Xiaoxing Ma, Jian Lu. Softened Symbol Grounding for Neuro-symbolic Systems.\n\n[2] Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, Jean-Philippe Vert. On Mixup Regularization.\n\n[3] Rafael Müller, Simon Kornblith, Geoffrey Hinton. When Does Label Smoothing Help?\n\n[4] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, Puneet Dokania. Calibrating Deep Neural Networks using Focal Loss. \n\n[5] Connor Pryor, Charles Dickens, Eriq Augustine, Alon Albalak, William Wang, Lise Getoor. NeuPSL: Neural Probabilistic Soft Logic.\n\n[6] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, Yixuan Li. Mitigating Neural Network Overconfidence with Logit Normalization.\n\nLIMITATIONS:\nNo, the limitations of the method were not discussed. The work does not have negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes mitigation and evaluation strategies for \"reasoning shortcuts\" in neuro-symbolic reasoning models, roughly corresponding to when the concepts (i.e., latent variables) extracted do not match the \"true\" ground-truth factors. The paper then examines reasoning shortcuts on several proposed, simple datasets, including XOR, MNIST-Addition, MNIST-EvenOdd, MNIST-AddMul, and BDD-OIA.\n\nSTRENGTHS:\nThe topic is important and underexplored, while the approach taken in this paper is novel. In general, the presentation is also excellent. The experiments, while fairly simple, are appropriate given the topic. The work is highly related to and extends prior work on disentanglement in machine learning, which is referenced (though not especially explicitly discussed). It is also decently well-motivated and some of the tasks like the BDD dataset highlighted interesting failure cases.\n\nWEAKNESSES:\nFundamentally, this paper is subject to many of the same challenges as prior work on disentanglement. That is, the task of identifying the \"correct\" latent variables is the task of disentanglement. While Locatello et al. (2019)'s \"Challenging common assumptions\" is cited for the definition of disentanglement (as [44]), the key lesson is never explicitly referenced (at least, not attributed): without supervision or inductive biases on the model and data, disentanglement is impossible. However, the authors highlight a similar point empirically, that \"Disentanglement is not enough under selection bias.\" Of course, this is not a particularly high bar, and indeed, the strategies taken to mitigate reasoning shortcuts in this paper do so with varying degrees of explicitness. For example, the paper's \"data-based mitigation\" explicitly provides supervision using a subset of the latent variables. However, in their objective-based approach, which draws inspiration from autoencoders, they could have plausibly also built on disentanglement-focused literature like InfoGAN (Chen et al. 2016) or $\\beta$-TCVAE (Chen et al. 2018). However, with that being said, it approaches many of these questions from a refreshing new perspective.\n\nThis leads to the second concern - to some extent, many of the proposed mitigation strategies are standard in the disentanglement literature. For example, \"On the relationship between disentanglement and multi-task learning\" (Maziarka et al. 2021) highlights the role of training on multiple tasks for learning disentangled representations -- in this paper, that is seemingly equivalent to knowledge-based mitigation. On the other hand, the data-based mitigation seemingly corresponds to the approach taken in \"Semi-Supervised StyleGAN for Disentanglement Learning\" (Nie et al. 2020). The objective-based mitigation (i.e., reconstruction loss) is, of course, central to many representation learning approaches. Then in the discussion of \"architecture-based mitigation,\" the connection to disentanglement work is made explicit. However, approaching these questions from the perspective of reasoning is fairly novel and is an important connection.\n\nUltimately, while the problem itself is well-motivated and novel, and the paper itself is excellent, and I believe it should likely be accepted based on the analysis alone, my primary reservation stems from the question of how reasoning, in particular, has shaped the mitigation strategies taken in this paper, beyond the datasets considered.\n\n---- Post-rebuttal update ----\nMost of my concerns have been addressed - I've raised my score to a 7.\n\nQUESTIONS:\nWhat differences would you highlight between the reasoning-focused tasks in this paper and the more traditional tasks in the disentanglement literature? How are the approaches discussed in this paper particularly well-suited for those tasks?\n\nLIMITATIONS:\nI think some more explicit discussion of the theoretical limitations demonstrated by related works (e.g., Locatello et al. 2019) would have been useful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n_Background_: The authors are interested in studying the properties of sequential, two-stage neuro-symbolic pipelines. Stage 1 is a neural network that reduces a high dimensional input (eg: an MNIST image) to a low-dimensional relaxation of a symbolic space (eg: 10 dimensional onehot vector) and Stage 2 is a probabilistic function (neural network or otherwise) that predicts the labels given the symbols utilizing a provided knowledge base. However, when trained end-to-end, the symbolic representation often collapses and the symbols learned are not equivalent to the ground truth symbols. This is called a reasoning shortcut (RS).  \nThis work focuses on investigating the causes of such reasoning shortcuts and presenting suitable remedies for the same. The authors formally define reasoning shortcuts for NeSy architectures as symbolic mappings which allow the model to achieve a locally optimal loss, but whose semantics do not match the ground truth semantics.  \nThe authors identify four factors to mitigate the occurrence of reasoning shortcuts: 1) additional prior knowledge,  2) additional data, 3) additional reconstruction targets in the objective function, and 4) additional inductive bias in model architecture design. They study the impact of these reasoning shortcuts mitigations on five NeSy prediction tasks with three NeSy algorithms.  \nThe authors find that different combinations of mitigation strategies helps improve performance in different cases.\n\nSTRENGTHS:\n**Originality:** \n-  While the concept of reasoning shortcuts was previously known, this paper is the first to offer a formal definition and analysis of reasoning shortcuts as a limitation of instantiations of neurosymbolic models.  \n- The mitigation strategies proposed are widely used in the neurosymbolic community, but they haven’t been connected before under the umbrella definition of reasoning shortcut mitigations. Hence, this paper brings a fresh perspective to why such strategies work well for NeSy systems.\n\n**Quality and Clarity**\n- The setup, definition, and analysis of reasoning shortcuts is well motivated and well explained.\n  \n**Significance**\n- The paper identifies a significant problem with previous work and provides a formal definition of the problem and mitigation strategies to address the problem. \n- I’m intrigued by the analysis of counting reasoning shortcuts because it presents a method to quantify how well a knowledge base describes a dataset / data generation process.\n\nWEAKNESSES:\n**Clarity:**\n- Table 1: DIS is mentioned in the caption, but no row for it exists in the table. \n- Table 2: It took me a long time to figure out that each number is the frequency of RSs for a baseline on a dataset. Please explain what the rows, columns, and values mean for each table. \n-  Line 276: “when forcing disentanglement…” Disentanglement is a property of the architecture. How is the architecture modified to enforce disentanglement? The appendix doesn't mention this. \n- “Roughly speaking, a concept F1 below 95% typically indicates an RS” I’ll defer to the author's judgement here because I’m not familiar with the data, but to me, a low concept F1 *on its own* signifies that the symbol extraction hasn’t been successful. Did the authors perhaps mean a low concept F1 but a high prediction F1 hints at the existence of RS's? \n- Table 4 results: I'm not sure that I understand the results here. I defer to the authors, but I think this table and task demands extra scrutiny. \n  - The prediction accuracy and the concept accuracy both seem low (< 80% as a ballpark figure). Is this close to the state-of-the-art performance for this dataset? \n  - \"concepts tend to align much closer to the diagonal...\" I don't quite understand how being closer to the diagonal implies better concept generalization. The concepts aren't ordered, so an incorrect prediction, regardless of perceived distance from the correct concept, should be incorrect. Maybe I'm misunderstanding this sentence. \n\n**Significance:**\n- The method only compares against ablations of the same model. I'm concerned that this paints an incomplete picture about the relative performance of NeSy predictors compared to other models on these tasks. This lowers the significance of this architecture for the neurosymbolic community. I'd advise the authors to look into other fully neural and neurosymbolic methods that might have similar problem statements, and/or provide reasons for why the baselines are not suitable for the datasets. I've provided a couple of works I'm familiar with that are related (but not exactly equivalent) to the base algorithm studied in this paper.\n  * ROAP[1]: ROAP presents an end to end algorithm for symbol grounding and symbol manipulation when external knowledge is not available, but we have diversity of tasks to instantiate a multi-task learning objective. I am reminded of ROAP because one of their tasks is similar to the MNIST digit grounding tasks presented here. \n  * Concept Embedding Bottleneck models[2]: Concept embedding bottleneck models instantiate an extra layer that creates a concept embedding that's used for downstream prediction. I am reminded of CEM models because their algorithm is geared towards leveraging explainable symbolic information in real world tasks without loosing accuracy.    \n  * I'd also be interested in whether the authors think these models use the mitigation strategies presented for reasoning shortcuts.\n\nOverall, I'm giving this paper a __weak accept__. The authors formalize an interesting problem which is definitely interesting to the broader neurosymbolic community. However, the related work / evaluation focuses on a very specific definition of neurosymbolic models.\n\n[1] https://arxiv.org/abs/2206.05922\n[2] https://arxiv.org/abs/2209.09056\n\nQUESTIONS:\nSee weaknesses section\n\nLIMITATIONS:\nThe authors have adequately addressed potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a in depth analysis of reasoning shortcuts and the impact they have of the learning process of neuro-symbolic learners. \nOn the ground of the performed analysis, they propose 4 different mitigation strategies to alleviate the problem: \n\n1. Knowledge-based mitigation\n\n2. Data-based mitigation\n\n3. Objective-based mitigation \n\n4. Architecture based mitigation\n\nSTRENGTHS:\nThe paper studies an important problem and it is very well presented. It is easy to follow and the formalisation of the concept of reasoning shortcuts was needed. \n\nThe experimental analysis supports the claims.\n\nWEAKNESSES:\nI have only couple of perplexities regarding the first two mitigation strategies: \n\n1. in the knowledge-based mitigation, the authors claim that depending on the application, collecting new knowledge might not be feasible. They then proceed in proposing Multi-task learning as a practical alternative. Here it is were I get confused: \n     - how can this be more practical? In addition to collecting more knowledge you now need to collect additional labels. \n     - if I understand correctly, in proposition 4 in the end the conclusion is anyway that the deterministic optimum $p_{\\theta}(\\mathbf{C} \\mid \\mathbf{G})$ of the MTL loss is a deterministic optimum of a single task whose prior knowledge is the conjunction of all the prior knowledge used in the different MTL tasks. How is it then better defining different MTL tasks than collecting more knowledge? \n\n2. In section 5.2 the authors say that RSs can occur also when the dataset is exhaustive. This to me seems more of a problem related to not choosing the right concepts, for the task, i.e., there is not a bijective correspondence between the concepts in $\\mathbf{C}$ and $\\mathbf{G}$. For example, suppose we have the (very simple) task of distinguishing bird images from non-bird images, and suppose we have one concept encoding the classes \"pigeon, flamingo, duck\" (and we have the simple background knowledge that any of these classes implies \"bird\"). In this case, one mitigation strategy could be to add more supervisions on the concept (as proposed), but wouldn't it be better to actually take concepts that are \"meaningful\" for the task? \n\nOn the line of the last point above, I was very surprised from the amount of reasoning shortcuts learnt in the MNIST-addition task in table 2. Can you provide some examples for such RSs? Also, these results are taken on *optimal* runs. What would be the results on 30 *standard runs*?  Also, can you provide some results in terms of accuracy on both the concepts and the final outcome when using disentanglement and when not?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}