SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
I ^ {(m)} (i, j) = \mathbb {E} _ {S \subseteq N \backslash \{i, j \}, | S | = m} [ \Delta v (i, j, S) ], \tag {25}
$$
where  $\Delta v(i,j,S) = v(\pmb{x}_{S\cup \{i,j\}}) - v(\pmb{x}_{S\cup \{i\}}) - v(\pmb{x}_{S\cup \{j\}}) + v(\pmb{x}_S)$ .
Let  $\Delta v_{T}(S) = \sum_{L\subseteq T}(-1)^{|T| - |L|}v(\pmb{x}_{L\cup S})$  denote the marginal benefit of variables in  $T\subseteq N\setminus S$ , given the environment  $S$ . In this way,  $\Delta v_{T}(S)$  can be represented as the sum of interaction effects inside  $T$  and sub-environments of  $S$ , i.e.  $\Delta v_{T}(S) = \sum_{S^{\prime}\subseteq S}I(T\cup S^{\prime})$  [45].
Thus, the  $I^{(m)}(i,j)$  can be represented as follows,
$$
\begin{array}{l} I ^ {(m)} (i, j) = \mathbb {E} _ {S \subseteq N \backslash \{i, j \}, | S | = m} [ \Delta v (i, j, S) ] \\ = \mathbb {E} _ {S \subseteq N \backslash \{i, j \}, | S | = m} \left[ \sum_ {L \subseteq S} I (L \cup \{i, j \}) \right] \\ = \frac{1}{\binom{n - 2}{m}}\sum_{\substack{S\subseteq N\setminus \{i,j\} \\ |S| = m}}[\sum_{L\subseteq S}I(L\cup \{i,j\})] \\ = \sum_{\substack{L\subseteq N\setminus \{i,j\} \\ |L|\leq m}}I(L\cup \{i,j\})\sum_{\substack{L\subseteq S\subseteq N\setminus \{i,j\} \\ |S| = m}}\frac{1}{\binom{n - 2}{m}} \\ = \sum_{\substack{L\subseteq N\setminus \{i,j\} \\ |L|\leq m}}I(L\cup \{i,j\})\frac{\binom{n - 2}{m - l}}{\binom{n - 2}{m}} \\ = \sum_{l = 0}^{m}\sum_{\substack{L\subseteq N\setminus \{i,j\} \\ |L| = m}}\frac{\binom{n - 2}{m - l}}{\binom{n - 2}{m}} I(L\cup \{i,j\}). \\ \end{array}
$$
Therefore, we prove that  $I^{(m)}(i,j) = \sum_{l=0}^{m}\sum_{\substack{L\subseteq N\setminus \{i,j\} \\ |L| = m}}\frac{\binom{n-2}{m-l}}{\binom{n-2}{m}}I(L\cup \{i,j\})$ .
G.5 Theoretical connection between variance and learning difficulty
There is a close connection between the variance of interactive concepts and the learning difficulty. In fact, a similar connection has been discovered by [46]. The theoretical connection is proved as follows. Specifically, we simplify the learning of a DNN into a linear regression problem. In this very simple setting, we assume each  $i$ -th concept encoded by the DNN to be a specific feature, and use  $f_{i}$  to represent the triggering (presence) state of the  $i$ -th concept in a training sample. Because according to Eq. (9) in the main paper, the network output  $v(\cdot)$  is the sum of a small number of salient interactive concepts, we roughly simplify and rewrite the inference output of the DNN as the following linear function
$$
y = \boldsymbol {w} ^ {\top} \boldsymbol {f} = w _ {1} f _ {1} + \dots + w _ {d} f _ {d},
$$
where  $\pmb{f} = [f_{1},\dots ,f_{d}]^{\top}$ . Then,  $w_{i}$  can be viewed as the strength of the DNN encoding the  $i$ -th interactive concept. In this way, if  $w_{i}\approx 0$ , the DNN does not learn the  $i$ -th interactive concept.
We suppose that training samples are sampled from  $f \sim P(f) = N(\mu, \Sigma^2)$  and  $y^*$  denotes the ground truth label. Specifically,  $\pmb{\mu} = [\mu_1, \mu_2, \dots, \mu_d]^\top$  and  $\pmb{\Sigma} = \mathrm{diag}(\sigma_1^2, \sigma_2^2, \dots, \sigma_d^2)$ .
The toy regression problem can be formulated as follows,
$$
L = E _ {P (f)} \left[ \frac {1}{2} \left(\boldsymbol {w} ^ {\top} \boldsymbol {f} - y ^ {*}\right) ^ {2} \right] \tag {26}
$$
We derive the optimal weights to the above regression task in three steps.
Step 1. The optimal weight  $w_{i}$  for the regression problem in Eq. (26) is computed by
$$
\forall 1 \leq i \leq d, \quad w _ {i} = \frac {1}{\det K} \det \left(\boldsymbol {K} _ {1}, \dots , \boldsymbol {K} _ {i - 1}, \boldsymbol {\zeta}, \boldsymbol {K} _ {i + 1}, \dots , \boldsymbol {K} _ {d}\right) \tag {27}
$$
where  $\pmb{K} = \pmb{\mu}\pmb{\mu}^{\top} + \pmb{\Sigma}^{2},\zeta = y^{*}\pmb{\mu}$ , and  $\pmb{K}_{j}$  denotes the  $j$ -th column of the matrix  $\pmb{K}$ .
Step 2. We further prove that for the optimal  $w$ , we have
$$
\forall 1 \leq i, j \leq d, \quad \frac {\left| w _ {i} \right|}{\left| w _ {j} \right|} = \frac {\left| \mu_ {i} / \sigma_ {i} ^ {2} \right|}{\left| \mu_ {j} / \sigma_ {j} ^ {2} \right|} \tag {28}
$$
Step 3. Based on Step 2, we can derive  $|w_i| \propto |1 / \sigma_i^2|$ , where  $w_i$  denotes the strength of the DNN encoding the  $i$ -th interactive concept.  $|w_i| \propto |1 / \sigma_i^2|$  indicates that the learning effect of the  $i$ -th interactive concept is inversely proportional to the variance of the  $i$ -th interactive concept. Therefore, interactive concepts with high variances (high-order interactive concepts) are more difficult to learn.
H Experimental Settings
Training details. We trained AlexNet [23], VGG-11 [55], ResNet-18/20 [19] on the CIFAR-10 dataset [22] and the Tiny ImageNet dataset [25], respectively. We also trained a five-layer MLP on the UCI census dataset (namely census dataset) and the UCI TV news dataset (namely TV news dataset) [6], respectively. Each layer of the MLP contained 100 neurons. We trained each neural networks for 200 epochs with the SGD optimizer.
Sampling details. Since, the computational cost of  $I(S|x)$  was intolerable in real implementation, we applied the sampling-based approximation method in [67] to calculate  $I(S|x)$ . Due to the high dimension of image data (e.g.  $224 \times 224$  for ImageNet), we uniformly split the input image into  $8 \times 8$  patches. Furthermore, we random sampled 12 patches and considered these patches as input variables for each image. The remaining 52 patches are set to the baseline value.
Implementations details. Here, we introduce how to measure  $\beta(S)$  and  $\kappa(S)$  in Section 2.3. On the Tiny ImageNet dataset, we randomly sampled 100 training images. These training images were randomly sampled from different 10 classes. On the CIFAR-10 dataset, we randomly sampled 10 training images from each class. For tabular datasets, we randomly sampled 50 training samples from each class. For image datasets, we set  $\tau = 2$  when the input data is normalized by its standard deviation. For tabular datasets, we set  $\tau = 1$  when the input data is normalized by its standard deviation. In this way, we set the baseline  $b_i = \max(x_i - \tau, \mu)$ , if  $x_i > \mu_i$ , and we set the baseline  $b_i = \min(x_i + \tau, \mu)$ , if  $x_i < \mu_i$ . For Gaussian perturbation  $\epsilon$ , we set  $\sigma = 0.02$ . Besides, for each training sample, we randomly sampled five Gaussian perturbation with five different seeds, respectively.
Adversarial attack. Here, we introduce how to measure  $A^{(s)}$  in Section 3.1. For tabular datasets, we randomly sampled 50 training samples from each class from the training set. On the Tiny-ImageNet dataset, we randomly sampled 100 training images. These training images were randomly sampled from different 10 classes. On the CIFAR-10 dataset, we randomly sampled 10 training images from each class. We used the  $l_{\infty}$  untargeted PGD attack by following [33], in which the constraint  $\epsilon = 16 / 255$ , and the attack was conducted with 5 steps with the step size  $\epsilon = 3 / 255$ .