SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {\alpha \in \mathcal {A}} \mathbb {1} \left\{\bigwedge_ {\mathbf {g} \in \operatorname {s u p p} (\mathbf {G})} \bigwedge_ {t = 1} ^ {T} \left(\left(\beta_ {\mathrm {K} ^ {(t)}} \circ \alpha\right) (\mathbf {g}) = \beta_ {\mathrm {K} ^ {(t)}} (\mathbf {g})\right) \right\} \tag {7}
$$
This means that, essentially, MTL behaves like a logical conjunction: any concept extractor  $p_{\theta}(\mathbf{C} \mid \mathbf{G})$  incompatible with the knowledge of any task  $t$  is not optimal. This strategy can be very effective, and indeed it performs very well in our experiments, but it necessitates gathering or designing a set of correlated learning tasks, which may be impractical in some situations.
5.2 Data-based Mitigation
Another key factor is the support of  $p^*(\mathbf{G})$ : if the support is not full, the conjunction in Eq. (6) becomes looser, and the number of  $\alpha$ 's satisfying it increases. This is what happens in MNIST-EvenOdd (Example 2): here, RSs arise precisely because the training set only includes a subset of combinations of digits, leaving ample room for acquiring unintended concepts.
We stress, however, that RSs can also occur if the data set is exhaustive, as in the next example.
Example 3 (XOR task). Consider a task with three binary ground-truth concepts  $\mathbf{G} = (G_1, G_2, G_3)$  in which the label  $Y$  is the parity of these bits, that is  $\mathsf{K} = (Y = G_1 \oplus G_2 \oplus G_3)$ . Each label  $Y \in \{0, 1\}$  can be inferred from four possible concept vectors  $\mathbf{g}$ , meaning that knowing  $\mathbf{y}$  is not sufficient to identify the  $\mathbf{g}$  it was generated from. In this case, it is impossible to pin down the ground-truth distribution  $p^*(\mathbf{C} \mid \mathbf{G}; \mathsf{K})$  even if all possible combinations of inputs  $\mathbf{x}$  are observed.
One way of avoiding RSs is to explicitly guide the model towards satisfying the condition in Eq. (3) by supplying supervision for a subset of concepts  $\mathbf{C}_I \subseteq \mathbf{C}$ , with  $I \subseteq [k]$ , and then augmenting the log-likelihood with a cross-entropy loss over the concepts of the form  $\sum_{i \in I} \log p_{\theta}(C_i = g_i \mid \mathbf{x})$ . Here, training examples  $(\mathbf{x}, \mathbf{g}_I, \mathbf{y})$  come with annotations for the concepts indexed by  $I$ . The impact of this strategy on the number of deterministic RSs is given by the following result:
Proposition 5. Assume that concept supervision is available for all  $\mathbf{g}$  in  $S\subseteq \mathrm{supp}(\mathbf{G})$ . Under  $\mathbf{A}\mathbf{I}$ , the number of deterministic optima  $p_{\theta}(\mathbf{C}\mid \mathbf{G})$  minimizing the cross-entropy over the concepts is:
$$
\sum_ {\alpha \in \mathcal {A}} \mathbb {1} \left\{\bigwedge_ {\mathbf {g} \in \mathcal {S}} \bigwedge_ {i \in I} \alpha_ {i} (\mathbf {g}) = g _ {i} \right\} \tag {8}
$$
This strategy is very powerful: if  $I = [k]$ ,  $S \equiv \operatorname{supp}(\mathbf{G})$ , and the support is complete, there exists only one map  $\alpha$  that is consistent with the condition in Eq. (8) and it is the identity. Naturally, this comes at the cost of obtaining dense annotations for all examples, which is often impractical.
5.3 Objective-based Mitigation
A natural alternative is to augment the log-likelihood with an unsupervised penalty designed to improve concept quality. We focus on reconstruction penalties like those used in auto-encoders [40, 41, 42, 43, 44], which encourage the model to capture all information necessary to reconstruct the input  $\mathbf{x}$ . To see why these might be useful, consider Example 2. Here, the model learns a RS mapping both  $\mathbf{0}$  and  $\mathbf{2}$  to the digit 1: this RS hinders reconstruction of the input images, and therefore could be avoided by introducing a reconstruction penalty.
In order to implement this, we introduce additional latent variables  $\mathbf{Z}$  that capture the style  $\mathbf{S}$  of the input  $\mathbf{X}$  and modify the concept extractor to output both  $\mathbf{C}$  and  $\mathbf{Z}$ , that is:  $p_{\theta}(\mathbf{c},\mathbf{z}|\mathbf{x}) = p_{\theta}(\mathbf{c}|\mathbf{x})\cdot p_{\theta}(\mathbf{z}|\mathbf{x})$ . The auto-encoder reconstruction penalty is then given by:
$$
\mathcal {R} (\mathbf {x}) = - \mathbb {E} _ {(\mathbf {c}, \mathbf {z}) \sim p _ {\theta} (\mathbf {c}, \mathbf {z} | \mathbf {x})} \left[ \log p _ {\psi} (\mathbf {x} \mid \mathbf {c}, \mathbf {z}) \right] \tag {9}
$$
where,  $p_{\psi}(\mathbf{x}\mid \mathbf{c},\mathbf{z})$  is the decoder network. We need to introduce an additional assumption A3: the encoder and the decoder separate content from style, that is,  $p_{\theta}(\mathbf{C},\mathbf{Z}\mid \mathbf{G},\mathbf{S}):= \mathbb{E}_{\mathbf{x}\sim p^{*}(\mathbf{x}|\mathbf{G},\mathbf{S})}p_{\theta}(\mathbf{C},\mathbf{Z}\mid$ $\mathbf{x})$  factorizes as  $p_{\theta}(\mathbf{C}\mid \mathbf{G})p_{\theta}(\mathbf{Z}\mid \mathbf{S})$  and  $p_{\psi}(\mathbf{G},\mathbf{S}\mid \mathbf{C},\mathbf{Z}):= \mathbb{E}_{\mathbf{x}\sim p_{\psi}(\mathbf{x}|\mathbf{C},\mathbf{Z})}p^{*}(\mathbf{G},\mathbf{Z}\mid \mathbf{x})$  as  $p_{\psi}(\mathbf{G}\mid \mathbf{C})p_{\psi}(\mathbf{S}\mid \mathbf{Z})$ . In this case, we have the following result:
Proposition 6. Under A1 and A3, the number of deterministic distributions  $p_{\theta}(\mathbf{C} \mid \mathbf{G})$  that minimize the reconstruction penalty in Eq. (9) is:
$$
\sum_ {\alpha \in \mathcal {A}} \mathbb {1} \left\{\bigwedge_ {\mathbf {g}, \mathbf {g} ^ {\prime} \in \operatorname {s u p p} (\mathbf {G}): \mathbf {g} \neq \mathbf {g} ^ {\prime}} \alpha (\mathbf {g}) \neq \alpha \left(\mathbf {g} ^ {\prime}\right) \right\} \tag {10}
$$
XOR	MNIST-Addition
DPL	SL	LTN	DPL	SL	LTN
-	100%	100%	100%	96.7%	82.9%	100%
DIS	0%	0%	0%	0%	0%	0%
Table 2: Q1: Disentanglement (DIS) can be very powerful to lower the frequency of RSs on XOR and MNIST-Addition data sets (the lower the better). Results are averaged over 30 optimal runs.
In words, this shows that indeed optimizing for reconstruction facilitates disambiguating between different concepts, i.e., different ground-truth concepts cannot be mapped to the same concept. However, minimizing the reconstruction can be non-trivial in practice, especially for complex inputs.
5.4 Architecture-based Mitigation
One last factor is the architecture of the concept extractor  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$ , as it implicitly controls the number of candidate deterministic maps  $\mathcal{A}$  and therefore the sum in Theorem 2. If the architecture is unrestricted,  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  can in principle map any ground-truth concept  $\mathbf{g}$  that generated  $\mathbf{x}$  to any concept  $\mathbf{c}$ , thus the cardinality of  $\mathcal{A}$  increases exponentially with  $k$ .
A powerful strategy for reducing the size of  $\mathcal{A}$  is disentanglement. A model is disentangled if and only if  $p_{\theta}(\mathbf{C} \mid \mathbf{G})$  factorizes as  $\prod_{j \in [k]} p_{\theta}(C_j \mid G_j)$  [45, 36]. In this case, the maps  $\alpha$  also factorize into per-concept maps  $\alpha_j : [m_j] \to [m_j]$ , dramatically reducing the cardinality of  $\mathcal{A}$ , as shown by our first experiment. In applications where the  $k$  concepts are naturally independent from one another, e.g., digits in MNIST-Addition, one can implement disentanglement by predicting each concept using the same neural network, although more general techniques exist [46, 47].
5.5 Other Heuristics based on Entropy Regularization
Besides these mitigation strategies, we investigate empirically the effect of the Shannon entropy loss, defined as  $1 - \frac{1}{k}\sum_{i = 1}^{k}\mathsf{H}_{m_i}[p_\theta (c_i)]$ , which was shown to increase concept quality in DPL [20]. Here,  $p_{\theta}(\mathbf{C})$  is the marginal distribution over the concepts, and  $\mathsf{H}_{m_i}$  is the normalized Shannon entropy over  $m_{i}$  possible values for the distribution. Notice that this term goes to zero only when each distribution  $p_{\theta}(C_i)$  is uniform, which may conflict with the real objective of the NeSy prediction (especially when only few concepts are observed). Other similar heuristics that are suited for reducing over-confidence in label predictions are based on label smoothing [48], energy-based models [49], annealing [50], and many others [51, 52, 53]. In principle, when applied at the concept level, they help reduce the over-confidence in concepts that is typical in deterministic RSs. While they could also be beneficial to mildly reduce the number of RSs, we take the Shannon entropy regularization as a representative of this family for our experiments.
6 Case Studies
In this section, we evaluate the impact of RSs in synthetic and real-world NeSy prediction tasks and how the mitigation strategies discussed in Section 5 fare in practice. More details about the models and data are reported in Appendix C. The code is available at github.com/reasoning-shortcuts.
Q1: More data does not prevent RSs but disentanglement helps. We start by evaluating the robustness of DPL, SL, and LTN to RSs in two settings where the data set is exhaustive. In XOR (cf. Example 3), the goal is to predict the parity of three binary concepts  $\mathbf{g} \in \{0,1\}^3$  given prior knowledge  $K = (y = g_1 \oplus g_2 \oplus g_3)$ . The predictor receives the ground-truth concepts  $\mathbf{g}$  as input and has to learn a distribution  $p_{\theta}(\mathbf{C} \mid \mathbf{G})$ . In MNIST-Addition (cf. Example 1) the goal is to correctly predict the sum of two MNIST digits, e.g.,  $\mathbf{x} = (\mathbf{1}, \mathbf{2})$  and  $y = 3$ . In both tasks, the training set contains examples of all possible combinations of concepts, i.e.,  $\mathrm{supp}(\mathbf{G}) = \mathcal{G}$ .
Since the tasks are relatively simple, we can afford to study models that achieve near-optimal likelihood, for which Definition 1 approximately applies. To this end, for each NeSy architecture, we train several models using different seeds and stop as soon as we obtain 30 models with likelihood  $\geq 0.95$ . Then, we measure the percentage of models that have acquired a RS. We do the same also for models modified to ensure they are disentangled (DIS in the Table), see Appendix C for details. The results, reported in Table 2, clearly show that RSs affect all methods if disentanglement is not in place. This confirms that, without implicit architectural biases, optimizing for label accuracy alone is not
Table 3: Q2: Impact of mitigation strategies. We report the  $F_{1}$  -score on the labels (Y) and concepts (C). All tested methods incorporate DIS and are averaged over 10 runs. Top: NeSy methods combined with R, C, H on MNIST-EvenOdd.
MNIST-EvenOdd	DPL	SL	LTN
F1(Y)	F1(C)	F1(Y)	F1(C)	F1(Y)	F1(C)
-	85.1 ± 4.6	0.1 ± 0.1	99.3 ± 0.2	0.2 ± 0.1	98.1 ± 0.2	0.3 ± 0.1
R	79.8 ± 1.0	0.1 ± 0.0	99.5 ± 0.2	0.1 ± 0.0	76.3 ± 1.1	0.0 ± 0.0
H	98.1 ± 0.1	0.1 ± 0.1	99.4 ± 0.1	0.1 ± 0.0	81.9 ± 0.5	53.9 ± 0.7
C	84.9 ± 0.1	0.1 ± 0.1	99.3 ± 0.4	21.5 ± 6.2	98.1 ± 0.2	0.2 ± 0.1
R + H	75.4 ± 0.4	0.2 ± 0.1	99.6 ± 0.1	0.1 ± 0.0	97.9 ± 2.3	38.1 ± 16.7
R + C	84.0 ± 2.2	1.9 ± 4.4	99.3 ± 0.2	61.5 ± 7.8	98.1 ± 0.2	0.2 ± 0.1
H + C	91.9 ± 3.5	88.0 ± 6.3	99.4 ± 0.2	41.5 ± 8.2	98.2 ± 0.2	98.6 ± 0.1
R + H + C	95.4 ± 0.4	96.2 ± 0.2	99.5 ± 0.2	47.2 ± 9.8	98.1 ± 0.3	98.5 ± 0.2
Bottom: evaluation on single tasks vs MTL on MNIST-AddMul.
MNIST-AddMul	DPL	SL	LTN
F1(Y)	F1(C)	F1(Y)	F1(C)	F1(Y)	F1(C)
ADD	68.1 ± 6.7	0.0 ± 0.0	99.5 ± 0.2	0.0 ± 0.1	67.4 ± 0.1	0.0 ± 0.0
MULT	100.0 ± 0.0	37.6 ± 0.2	100.0 ± 0.0	76.1 ± 11.7	98.1 ± 0.5	78.1 ± 0.4
MULTIOP	100.0 ± 0.0	99.8 ± 0.1	100.0 ± 0.0	99.8 ± 0.1	98.3 ± 0.2	98.3 ± 0.2
sufficient to rule out RSs, even when the data is exhaustive. However, when forcing disentanglement, the percentage of models affected by RSs drops to zero for all data sets and methods, indicating that mitigation strategies that go beyond the standard learning setup – and specifically, disentanglement – can be extremely effective.
Q2: Disentanglement is not enough under selection bias. Next, we look at two (non-exhaustive) data sets where RS occur due to selection bias. Label and concept quality is measured using the  $F_{1}$ -score on the macro average measured on the test set. From here onward, all models are disentangled by construction, and despite this are affected by RSs, as shown below.
We start by evaluating MNIST-EvenOdd, a variant of MNIST-Addition (inspired by [21]) where only 16 possible pairs of digits out of 100 are given for training: 8 comprise only even digits and 8 only odd digits. As shown in [21], this setup allows to exchange the semantics of the even and odd digits while ensuring all sums are correct (because, e.g.,  $\mathbf{4} + \mathbf{2} = 12 = \mathbf{9} + \mathbf{3}$ ). As commonly done in NeSy, hyperparameters were chosen to optimize for label prediction performance on a validation set, cf. Appendix C. The impact of reconstruction (R), concept supervision (C), and Shannon entropy loss (H) on all architectures are reported in Table 3. Roughly speaking, a concept  $F_{1}$  below  $95\%$  typically indicates a RSs, as shown by the concept confusion matrices in Appendix C.5. The main take away is that no strategy alone can effectively mitigate RSs for any of the methods. Additionally, for DPL and LTN, these also tend to interfere with label supervision, yielding degraded prediction performance. The SL is not affected by this, likely because it is the only method using a separate neural layer to predict the labels. Combining multiple strategies does improve concept quality on average, depending on the method. In particular,  $\mathrm{H} + \mathrm{C}$  and  $\mathrm{R} + \mathrm{H} + \mathrm{C}$  help LTN identify good concepts, and similarly for DPL, although concept quality is slightly less stable. For the SL only concept supervision is relatively effective, but the other strategies are not, probably due to the extra flexibility granted by the top neural layer compared to DPL and LTN.
Next, we evaluate the impact of multi-task learning on an arithmetic task, denoted MNIST-AddMul. Here, the model observes inputs  $\mathbf{x} \in \{(\mathbf{0},\mathbf{1}), (\mathbf{0},\mathbf{2}), (\mathbf{1},\mathbf{3})\}$  and has to predict both their sum and their product, either separately (no MTL) or jointly (MTL). The results in Table 3 show that, as in Example 2, all methods are dramatically affected by RSs when MTL is not in place. DPL and LTN also yield sub-par  $F_{1}$ -scores on the labels for the addition task. For SL and LTN, we also observe that, despite high concept  $F_{1}$  for the multiplication task, the digit  $\mathbf{z}$  is never predicted correctly. This is clearly visible in the confusion matrices in Appendix C.5. However, solving addition and multiplication jointly via MTL ensures all methods acquire very high quality concepts.
Q3: Reasoning Shortcuts are pervasive in real-world tasks. Finally, we look at RSs occurring in BDD-0IA [54], an autonomous vehicle prediction task. The goal is to predict multiple possible actions  $\mathbf{Y} = (\text{move_forward}, \text{stop}, \text{turn_left}, \text{turn_right})$  from frames  $\mathbf{x}$  of real driving scenes. Each scene is described by 21 binary concepts  $\mathbf{C}$  and the knowledge  $\mathsf{K}$  ensures the predictions are consistent with safety constraints (e.g., stop  $\Rightarrow$  move_forward) and guarantees concepts are predicted consistently with one another (e.g., road_clear  $\Leftrightarrow$  obstacle). Since this is a high-stakes task, we solve it with DPL, as it is the only approach out of the ones we consider that guarantees hard compliance with the knowledge [7]. See Appendix C for the full experimental setup.
Table 4 lists the results for DPL paired with different mitigation strategies. In order to get a sense of the model's ability to acquire good concepts, we also include two baselines: CBM-AUC introduced by
Table 4: Q3. Left: Means and std. deviations over 10 runs for DPL paired with different mitigations. Right: Confusion matrices on the training set for DPL alone (top) and paired with mitigation strategies (bottom) for {move_forward, stop, turn_left, turn_right} concept vectors.
BDD-OIA
F1-mean (Y)	F1-mean(C)
CBM-AUC	70.8 ± 0.1	62.1 ± 0.1
C-only	64.8 ± 0.2	60.3 ± 0.1
DPL	71.4 ± 0.1	39.4 ± 6.2
DPL+H	72.1 ± 0.1	48.1 ± 0.3
DPL+C	68.2 ± 0.2	60.5 ± 0.1
DPL+C +H	68.3 ± 0.3	61.7 ± 0.1
Sawada and Nakamura [55] for this task, which learns jointly the concepts together with unsupervised ones, and processes both of them with a linear layer to obtain the labels; C-only, in which the concept extractor is trained with full concept supervision and frozen, and DPL is only used to perform inference. This avoids any interference between label and concept supervision. The  $F_{1}$ -scores are computed over the test set and averaged over all 4 labels and all 21 concepts. On the right, we also report the concept confusion matrices (CMs) for the training set: each row corresponds to a ground-truth concept vector  $\mathbf{g}$ , and each column to a predicted concept vector  $\mathbf{c}$ .
Overall, the results show that, unless supplied with concept supervision, DPL optimizes for label accuracy  $(F_{1}(\mathrm{Y}))$  by leveraging low quality concepts  $(F_{1}(\mathrm{C}))$ . This occurs even when it is paired with a Shannon entropy penalty  $\mathrm{H}$ . This is especially evident in the CMs, which show that, for certain labels, all ground-truth concepts  $\mathbf{g}$  are mapped a single  $\mathbf{c}$ , not unlike what happens in our first experiment for entangled models. Conversely, the concept quality of DPL substantially improves when concept supervision is available, at the cost of a slight degradation in prediction accuracy. The CMs back up this observation as the learned concepts tend to align much closer to the diagonal. Only for the turn_left label concept supervision fails to prevent collapse. This occurs because annotations for the corresponding concepts are poor: for a large set of examples, the concepts necessary to predict turn_left = 1 are annotated as negatives, complicating learning. In practice, this means that all variants of DPL predict those concepts as negative, hindering concept quality.
Shortcuts in ML. State-of-the-art ML predictors often achieve high performance by exploiting spurious correlations – or “shortcuts” – in the training data [56]. Well known examples include watermarks [57], background pixels [58, 59], and textual meta-data in X-ray scans [16]. Like RSs, regular shortcuts can compromise the classifier's reliability and out-of-distribution generalization and are hard to identify based on accuracy alone. Proposed solutions include dense annotations [60], out-of-domain data [61], and interaction with annotators [62]. Shortcuts are often the result of confounding resulting from, e.g., selective sampling. RSs may also arise due to confounding, as is the case in MNIST-Addition (cf. Example 1), however – as discussed in Section 5, data is not the only factor underlying RSs. For instance, in XOR (cf. Example 3) RSs arise despite exhaustive data.
Reasoning shortcuts. The issue of RSs has so far been mostly neglected in the NeSy literature, and few remedies have been introduced but never theoretically motivated. Stammer et al. [63] have investigated shortcuts affecting NeSy architectures, but consider only input-level shortcuts that occur even if concepts are high-quality and fix by injecting additional knowledge in the model. In contrast, we focus on RSs that impact the quality of learned concepts. Marconato et al. [21] introduce the concept of RSs in the context of NeSy but for continual learning, and proposes a combination of concept supervision and concept-level rehearsal to address them, without delving into a theoretical justification. Li et al. [11] propose a minimax objective that ensures the concepts learned by the model satisfy K. Like the entropy regularizer [20] we addressed in Section 5, this strategy ends up spreading probability across all concepts that satisfy the knowledge, including those that have unintended semantics. As such, it does not directly address RSs.
Neuro-symbolic integration. While RSs affect a number of NeSy predictors, NeSy encompasses a heterogeneous family of architectures integrating learning and reasoning [1, 2]. We conjecture that
RSs do transfer to all NeSy approaches that do not specifically address the factors we identified, but an in-depth analysis of RSs in NeSy is beyond the scope of this paper. In a concurrent work, Wang et al. [64] derive the sample complexity of NeSy predictors when no RSs are in place. It is unclear how these bounds may change when RSs are present.
Relation to disentanglement. Recovering the latent variables  $\mathbf{G}$  has been the center of several works in representation learning [65, 35, 66]. Among these, achieving identifiability of the latent components has been studied in non-linear independent component analysis [67, 68, 69] and recently in Causal Representation Learning [46, 70, 71, 72]. In this respect, several overlaps exist with the natural mitigation strategies that we investigated: (i) multi-task learning has been shown to increase disentanglement [73] and provably leads to identifiability [74, 75], and (ii) latent variables supervision also largely increases the amount of disentanglement [76, 77, 47]. Our work is the witness that the intersection with disentanglement literature is beneficial for learning the intended concepts in NeSy and, vice-versa, that knowledge-guided learning can be a new way of acquiring identifiable representations, by avoiding RSs.
In this work, we provide the first in-depth analysis of RS affecting NeSy predictors. Our analysis highlights four key causes of RS and suggests several mitigation strategies, which we analyze both theoretically and empirically. Our experiments indicate that RSs do naturally appear in both synthetic and real-world NeSy prediction tasks, and that the effectiveness of mitigation strategies is model and task dependent, and that a general recipe for avoiding RSs is currently missing.
We foresee that reasoning shortcuts extend beyond the current scope of NeSy predictors with known prior knowledge. This includes approaches that learn jointly both the concepts and the knowledge, like ROAP [78] and DSL [79], as well as fully neural models, like CBMs [80] and their variants [39, 81], which are also likely to be affected by RSs when concept supervision is limited. We plan to investigate further this direction in the near future.
Ultimately, this work aims at jumpstarting research on the analysis and mitigation of RSs, with the hope of leading to more trustworthy and explainable NeSy architectures.
Broader impact. Our work brings the subtle but critical issue of RSs to the spotlight, and highlights benefits and limitations of a variety of mitigation strategies, but it is otherwise fundamental research that has no direct societal impact.
We thank the anonymous reviewers for their valuable feedback in improving the current manuscript. We acknowledge Emile von Krieken for proofreading our claims and Yoshihide Sawada for his help in deploying the models on BDD-01A. We are also grateful to Pedro Zuidberg dos Martires for insightful discussions with us at the early stages of the work. We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. The research of ST and AP was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.