SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(1) Efficiency axiom (proved by [18]). The reward of a neural network can be decomposed into interaction effects of different contexts, i.e.  $v(N) = \sum_{S \subset N} I(S)$ .
(2) Linearity axiom. If we merge rewards of two neural networks  $w$  and  $v$  as the reward of model  $u$ , i.e.  $\forall S \subseteq N$ ,  $u(S) = w(S) + v(S)$ , then their interaction effects  $I_v(S)$  and  $I_w(S)$  can be represented as  $\forall S \subseteq N, I_u(S) = I_v(S) + I_w(S)$ .
(3) Dummy axiom. If a variable  $i \in N$  is a dummy variable, i.e.  $\forall S \subseteq N \setminus \{i\}$ ,  $v(S \cup \{i\}) = v(S) + v(\{i\})$ , then it does not interact with other variables,  $\forall \emptyset \neq S \subset N \setminus \{i\}$ ,  $I(S \cup \{i\}) = 0$ .
Figure 7: The average histograms of absolute effects of interactive concepts encoded by five-layer MLPs.
(4) Symmetry axiom. Given two input variables  $i,j\in N$  , if  $\forall S\subseteq N\backslash \{i,j\}$ $v(S\cup \{i\}) = v(S\cup \{j\})$  then  $\forall S\subseteq N\backslash \{i,j\}$ $I(S\cup \{i\}) = I(S\cup \{j\})$
(5) Anonymity axiom. For any permutations  $\pi$  on  $N$ , we have  $\forall S \subseteq N, I_v(S) = I_{\pi v}(\pi S)$ , where  $\pi S\{\pi(i) | i \in S\}$ , and the new model  $\pi v$  is defined by  $(\pi v)(\pi S) = v(S)$ . This indicates that interaction effects are not changed by permutation.
(6) Recursive axiom. The interaction effects can be calculated recursively. Given  $i \in N$  and  $S \subseteq N \setminus \{i\}$ , the interaction effect of the pattern  $S \cup \{i\}$  can be represented as the interaction effect of  $S$  with  $i$  minus the interaction effect of  $S$  without  $i$ , i.e.  $\forall S \subseteq N \setminus \{i\}$ ,  $I(S \cup \{i\}) = I(S | i$  is always present) -  $I(S)$ .  $I(S | i$  is always present) denotes the interaction effect when the variable  $i$  is always present as a constant context, i.e.  $I(S | i$  is always present) =  $\sum_{L \subseteq S} (-1)^{|S| - |L|} \cdot v(L \cup \{i\})$ .
(7) Interaction distribution axiom. This axiom shows how interactions are distributed for "interaction functions" [57]. The definition of the interaction function  $v_{T}$  parameterized by a subset of variables  $T$  is given as follows.  $\forall S \subseteq N$ , if  $T \subseteq S$ ,  $v_{T}(S) = c$ ; otherwise,  $v_{T}(S) = 0$ . The function  $v_{T}$  models pure interaction among the variables in  $T$ , because only if all variables in  $T$  are present, the output value will be increased by  $c$ . The interactions encoded in the function  $v_{T}$  satisfies  $I(T) = c$ , and  $\forall S \neq T$ ,  $I(S) = 0$ .
E Sparsity of interactive concepts
In this section, we conducted experiments to verify the sparsity of interactive concepts. To this end, we computed effects  $U_{S}$  of all  $2^{n}$  interactive concepts encoded by a DNN following [45]. Specifically, we trained a five-layer MLP on the census dataset and the TV news dataset), respectively. For better visualization, we re-scaled effects of interactive concepts  $U_{S}$  by  $|U_{S}| / \max_{S^{\prime}\subseteq N}|U_{S^{\prime}}|$ . Moreover, the strength of effects are average over different samples in each dataset. Fig. 7 shows histograms of absolute effects of interactive concepts.
F Complexity of interactive concepts
Many previous studies [11, 67, 59, 45, 43] used multi-order interactions to analyze DNNs. Specifically, Given a pre-trained DNN  $v$  and a masked sample  $\boldsymbol{x}_S$ , the multi-order interaction  $I^{(m)}(i,j)$  used in [59, 11, 67] is given as follows:
$$
I ^ {(m)} (i, j) = \mathbb {E} _ {S \subseteq N \backslash \{i, j \}, | S | = m} [ \Delta v (i, j, S) ], \tag {1}
$$
where  $\Delta v(i,j,S) = v(\pmb{x}_{S\cup \{i,j\}}) - v(\pmb{x}_{S\cup \{i\}}) - v(\pmb{x}_{S\cup \{j\}}) + v(\pmb{x}_S)$ . They consider that  $I^{(m)}(i,j)$  reflects the collaboration with  $m$  contextual variables ( $m$  pixels). In this way, the order  $m$  measures the number of variables in  $S$ . Therefore, a low-order interaction denotes a relatively simple collaboration between input variables with a small context  $S$ . In contrast, a high-order interaction represents a complex collaboration between input variables with a large context  $S$ .
In this paper, we consider that the number of variables in an interactive concept  $S$  can measure the complexity of an interactive concept encoded by a DNN, namely the order of the interactive concept, complexity  $(S) = \text{order}(S) = |S|$ . Thus, low-order concepts usually represent simple AND relationships among a few input variables. In comparison, high-order concepts often refer to as relatively complex AND relationships among a large number of input variables.
G Proof of Theorems
G.1 Proof of Theorem 2 in the main paper
Theorem 2. Given a neural network  $v$  and an arbitrary input sample  $\pmb{x}^{\prime} \in \mathbb{R}^{n}$ , the network output can be decomposed by using the Taylor expansion [10], i.e.,  $v(\pmb{x}^{\prime}) = \sum_{S \subseteq N} \sum_{\pmb{\pi} \in Q_S} U_{S,\pmb{\pi}} \cdot J(S, \pmb{\pi} | \pmb{x}^{\prime})$ . In this way, according to Eq. (1), the interaction effect  $I(S | \pmb{x}^{\prime})$  on the sample  $\pmb{x}^{\prime}$  can be reformulated as follows.
$$
I (S | \boldsymbol {x} ^ {\prime}) = \sum_ {\boldsymbol {\pi} \in Q _ {S}} U _ {S, \boldsymbol {\pi}} \cdot J (S, \boldsymbol {\pi} | \boldsymbol {x} ^ {\prime}). \tag {2}
$$
Here,  $J(S,\pi |x^{\prime}) = \prod_{i\in S}\left(\mathrm{sign}(x_{i}^{\prime} - b_{i})\cdot \frac{x_{i}^{\prime} - b_{i}}{\tau}\right)^{\pi_{i}}$  denotes a Taylor expansion term of the degree  $\pi$  where the degree  $\pi \in Q_S = \{[\pi_1,\dots ,\pi_n]|\forall i\in S,\pi_i\in \mathbb{N}^+;\forall i\notin S,\pi_i = 0\}$  and  $b_{i}$  is the baseline value to mask the input variable  $x_{i}$ . In addition,  $U_{S,\pi} = \frac{\tau^{m}}{\prod_{i = 1}^{n}\pi_{i}!}\frac{\partial^{m}v(\pmb{x}_{\emptyset}^{\prime})}{\partial x_{1}^{\pi_{1}}\cdots\partial x_{n}^{\pi_{n}}}\cdot \prod_{i\in S}[\mathrm{sign}(x_{i}^{\prime} - b_{i})]^{\pi_{i}}$  where  $\pmb{x}_{\emptyset}^{\prime}$  denotes the sample whose input variables are all masked.  $m = \sum_{i = 1}^{n}\pi_{i}$ .
Proof. Let us denote the function on the right of Eq. (2) by  $\tilde{I}(S|x')$ , i.e.,
$$
\tilde {I} (S | \boldsymbol {x} ^ {\prime}) = \sum_ {\pi \in Q _ {S}} U _ {S, \pi} J (S, \pi | \boldsymbol {x} ^ {\prime}) \tag {3}
$$
We need to prove that for any arbitrary input sample  $\forall \pmb{x}^{\prime}\in \mathbb{R}^{n}$ $\tilde{I} (S|\pmb{x}^{\prime}) = I(S|\pmb{x}^{\prime})$
Actually, it has been proven in [16] and [45] that the Harsanyi dividend  $I(S|x')$  is the unique metric satisfying the faithfulness requirement mentioned in the main paper, i.e., satisfying
$$
\forall T \subseteq N, v \left(\boldsymbol {x} _ {T} ^ {\prime}\right) = \sum_ {S \in \Omega , S \subseteq T} I (S \mid \boldsymbol {x} ^ {\prime}). \tag {4}
$$
Thus, as long as we can prove that  $\tilde{I}(S|\pmb{x}^{\prime})$  also satisfies the above faithfulness requirement, we can obtain  $\tilde{I}(S|\pmb{x}^{\prime}) = I(S|\pmb{x}^{\prime})$ .
To this end, we only need to prove  $\tilde{I}(S|\pmb{x}^{\prime})$  also satisfies the faithfulness requirement in Eq. (4). Specifically, given an input sample  $\forall \pmb{x}^{\prime} \in \mathbb{R}^{n}$ , let us consider the Taylor expansion of the network output  $v(\pmb{x}_T)$  of an arbitrarily masked sample  $\pmb{x}_{\pmb{T}}^{\prime}(T \subseteq N)$ , which is expanded at  $\pmb{x}_{\emptyset}^{\prime} = [b_{1}, \dots, b_{n}]$ . Then, we have
$$
\forall T \subseteq N, \quad v \left(\boldsymbol {x} _ {T} ^ {\prime}\right) = \sum_ {\pi_ {1} = 0} ^ {\infty} \sum_ {\pi_ {2} = 0} ^ {\infty} \dots \sum_ {\pi_ {n} = 0} ^ {\infty} \frac {1}{\prod_ {i = 1} ^ {n} \pi_ {i} !} \frac {\partial^ {m} v \left(\boldsymbol {x} _ {\emptyset} ^ {\prime}\right)}{\partial x _ {1} ^ {\pi_ {1}} \cdots \partial x _ {n} ^ {\pi_ {n}}} \cdot \prod_ {i = 1} ^ {n} \left[ \left(\boldsymbol {x} _ {T} ^ {\prime}\right) _ {i} - b _ {i} \right] ^ {\pi_ {i}}, \tag {5}
$$
where  $\pi \in \{[\pi_1,\dots ,\pi_n]|\forall i\in N,\pi_i\in \mathbb{N}\}$  denotes the degree vector of Taylor expansion terms, and  $m = \sum_{i = 1}^{n}\pi_{i}$ . In addition,  $b_{i}$  denotes the reference value to mask the input variable  $x_{i}$ .
According to the definition of the masked sample  $\pmb{x}_T'$ , we have that all variables in  $T$  keep unchanged and other variables are masked to the reference value. That is,  $\forall i \in T, (\pmb{x}_T')_i = x_i; \forall i \notin T, (\pmb{x}_T')_i = b_i$ . Hence, we obtain  $\forall i \notin T, [(\pmb{x}_T')_i - b_i]^{n_i} = 0$ . Then, among all Taylor expansion terms, only terms corresponding to degrees  $\pi$  in the set  $P = \{ [\pi_1, \dots, \pi_n] | \forall i \in T, \pi_i \in \mathbb{N}; \forall i \notin T, \pi_i = 0 \}$  may not be zero. Therefore, Eq. (5) can be rewritten as
$$
\forall T \subseteq N, \quad v \left(\boldsymbol {x} _ {T} ^ {\prime}\right) = \sum_ {\boldsymbol {\pi} \in P} \frac {1}{\prod_ {i = 1} ^ {n} \pi_ {i} !} \frac {\partial^ {m} v \left(\boldsymbol {x} _ {\emptyset} ^ {\prime}\right)}{\partial x _ {1} ^ {\pi_ {1}} \cdots \partial x _ {n} ^ {\pi_ {n}}} \cdot \prod_ {i \in T} \left(x _ {i} ^ {\prime} - b _ {i}\right) ^ {\pi_ {i}}. \tag {6}
$$
We find that the set  $P$  can be divided into multiple disjoint sets as follows,  $P = \cup_{S\subseteq T}Q_S$ , where  $Q_{S} = \{[\pi_{1},\dots ,\pi_{n}]|\forall i\in S,\pi_{i}\in \mathbb{N}^{+};\forall i\notin S,\pi_{i} = 0\}$ . Then, we can derive that
$$
\begin{array}{l} \forall T \subseteq N, \quad v \left(\boldsymbol {x} _ {T} ^ {\prime}\right) = \sum_ {S \subseteq T} \sum_ {\boldsymbol {\pi} \in Q _ {S}} \frac {1}{\prod_ {i = 1} ^ {n} \pi_ {i} !} \frac {\partial^ {m} v \left(\boldsymbol {x} _ {\emptyset} ^ {\prime}\right)}{\partial x _ {1} ^ {\pi_ {1}} \cdots \partial x _ {n} ^ {\pi_ {n}}} \cdot \prod_ {i \in S} \left(x _ {i} ^ {\prime} - b _ {i}\right) ^ {\pi_ {i}} \\ = \sum_ {S \subseteq T} \sum_ {\boldsymbol {\pi} \in Q _ {S}} \underbrace {\frac {\tau^ {m}}{\prod_ {i = 1} ^ {n} \pi_ {i} !} \frac {\partial^ {m} v \left(\boldsymbol {x} _ {\emptyset} ^ {\prime}\right)}{\partial x _ {1} ^ {\pi_ {1}} \cdots \partial x _ {n} ^ {\pi_ {n}}} \prod_ {i \in S} (\delta_ {i}) ^ {\pi_ {i}}} _ {\text {t e r m e d} U _ {S, \pi}} \cdot \underbrace {\prod_ {i \in S} \left(\delta_ {i} \frac {x _ {i} ^ {\prime} - b _ {i}}{\tau}\right) ^ {\pi_ {i}}} _ {\text {t e r m e d} J (S, \pi | \boldsymbol {x} ^ {\prime})}, \tag {7} \\ \end{array}
$$
where  $\tau \in \mathbb{R}$  is a pre-defined constant and  $\delta_{i} = \mathrm{sign}(x_{i} - r_{i})\in \{-1,1\}$  is a sign function satisfying  $\delta_i^{2m} = 1$  ( $m = \sum_{i = 1}^{n}\pi_{i}$ ). Then, Eq. (7) can be re-written as
$$
\forall T \subseteq N, v \left(\boldsymbol {x} _ {T} ^ {\prime}\right) = \sum_ {S \subseteq T} \sum_ {\boldsymbol {\pi} \in Q _ {S}} U _ {S, \boldsymbol {\pi}} \cdot J (S, \boldsymbol {\pi} | \boldsymbol {x} ^ {\prime}) = \sum_ {S \subseteq T} \tilde {I} (S | \boldsymbol {x} ^ {\prime}). \tag {8}
$$
Thus,  $\tilde{I}(S|\pmb{x}^{\prime})$  satisfies the faithfulness requirement in Eq. (4) when  $\Omega = 2^{N}$ .
Therefore, Theorem 2 holds.
G.2 Proof of Theorem 3 in the main paper
Theorem 3. Let us add a Gaussian perturbation  $\epsilon \sim \mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})$  to the input sample  $\pmb{x}$ . Let us first consider the case with the lowest degree  $\hat{\pi} = [\hat{\pi}_1,\dots ,\hat{\pi}_n]\in Q_S$ , satisfying that  $\forall i\in S,\hat{\pi}_i = 1;\forall i\notin S,\hat{\pi}_i = 0$ . The mean and variance of  $J(S,\hat{\pi} |\pmb {x} + \pmb {\epsilon})$  over the Gaussian perturbation  $\epsilon$  are given as
$$
\mathbb {E} _ {\boldsymbol {\epsilon}} [ I (S | \boldsymbol {x} + \boldsymbol {\epsilon}) ] = U _ {S, \hat {\boldsymbol {\pi}}}, \quad \mathrm {V a r} _ {\boldsymbol {\epsilon}} [ I (S | \boldsymbol {x} + \boldsymbol {\epsilon}) ] = U _ {S, \hat {\boldsymbol {\pi}}} ^ {2} \bigg [ \left(1 + (\sigma / \tau) ^ {2}\right) ^ {s} - 1 \bigg ].
$$
$$
\mathbb {E} _ {\epsilon} [ J (S, \pmb {\pi} | \pmb {x} + \pmb {\epsilon}) ] = \mathbb {E} _ {\epsilon} [ \prod_ {i \in S} (1 + \epsilon_ {i} / \tau) ^ {\pi_ {i}} ], \quad \mathrm {V a r} _ {\epsilon} [ J (S, \pmb {\pi} | \pmb {x} + \pmb {\epsilon}) ] = \mathrm {V a r} _ {\epsilon} [ \prod_ {i \in S} (1 + \epsilon_ {i} / \tau) ^ {\pi_ {i}} ].
$$
Proof. If we only consider Taylor expansion term of the lowest degree, then  $I(S|\pmb{x}^{\prime}) \approx U_{S,\hat{\pi}} \cdot J(S,\hat{\pi}|\pmb{x}^{\prime})$ , where  $J(S,\hat{\pi}|\pmb{x}^{\prime}) = \prod_{i\in S}\mathrm{sign}(x_i' - b_i) \cdot \frac{x_i' - b_i}{\tau}$ .
Let us add a Gaussian perturbation  $\epsilon \sim \mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})$  to the input sample  $\pmb{x}$ . Then, we have
$$
I (S | \boldsymbol {x} + \boldsymbol {\epsilon}) \approx U _ {S, \hat {\pi}} \cdot J (S, \hat {\pi} | \boldsymbol {x} + \boldsymbol {\epsilon})
$$
$$
\begin{array}{l} J (S, \hat {\pi} | \boldsymbol {x} + \epsilon) = \prod_ {i \in S} \operatorname {s i g n} \left(x _ {i} + \epsilon_ {i} - b _ {i}\right) \cdot \frac {x _ {i} + \epsilon_ {i} - b _ {i}}{\tau} \tag {9} \\ = \prod_ {i \in S} \left(\mathrm {s i g n} (x _ {i} + \epsilon_ {i} - b _ {i}) \cdot \frac {x _ {i} - b _ {i}}{\tau} + \mathrm {s i g n} (x _ {i} + \epsilon_ {i} - b _ {i}) \cdot \frac {\epsilon_ {i}}{\tau}\right) \\ \end{array}
$$
According to the setting of the baseline value, we have  $\forall i\in S,x_i - b_i\in \{-\tau ,\tau \}$ . In Section 2.2, we have assumed that the perturbation is small, i.e.,  $\forall i\in S,|\epsilon_i|\leq \tau$ . In this way, we have  $\mathrm{sign}(x_i + \epsilon_i - b_i) = \mathrm{sign}(x_i - b_i)$ , and we can obtain
$$
\begin{array}{l} J (S, \hat {\pi} | \boldsymbol {x} + \boldsymbol {\epsilon}) = \prod_ {i \in S} \left(\operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {x _ {i} - b _ {i}}{\tau} + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) (10) \\ = \prod_ {i \in S} \left(1 + \mathrm {s i g n} (x _ {i} - b _ {i}) \cdot \frac {\epsilon_ {i}}{\tau}\right) \\ \Rightarrow \mathbb {E} _ {\boldsymbol {\epsilon}} [ J (S, \hat {\boldsymbol {\pi}} | \boldsymbol {x} + \boldsymbol {\epsilon}) ] = \mathbb {E} _ {\boldsymbol {\epsilon}} \left[ \prod_ {i \in S} \left(1 + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) \right] (11) \\ \mathrm {V a r} _ {\pmb {\epsilon}} [ J (S, \hat {\pmb {\pi}} | \pmb {x} + \pmb {\epsilon}) ] = \mathrm {V a r} _ {\pmb {\epsilon}} \left[ \prod_ {i \in S} \left(1 + \mathrm {s i g n} (x _ {i} - b _ {i}) \cdot \frac {\epsilon_ {i}}{\tau}\right) \right] \\ \end{array}
$$
Since  $\mathrm{sign}(x_i - b_i)\in \{-1,1\}$ , we have  $1 + \mathrm{sign}(x_{i} - b_{i})\cdot \frac{\epsilon_{i}}{\tau}\sim \mathcal{N}(1,(\sigma /\tau)^{2}),\forall i\in S$
Proposition 1. If random variables  $X_{1}, X_{2}, \dots, X_{k}$  are independent of each other, then  $\mathbb{E}[X_1X_2\cdots X_k] = \prod_{i=1}^{k} \mathbb{E}[X_i]$ , and  $\operatorname{Var}[X_1X_2\cdots X_k] = \prod_{i=1}^{k} (\mathbb{E}[X_i]^2 + \operatorname{Var}[X_i]^2) - \prod_{i=1}^{k} \mathbb{E}[X_i]^2$ .
According to the above proposition, we have
$$
\begin{array}{l} \mathbb {E} _ {\epsilon} \left[ J (S, \hat {\pi} | \boldsymbol {x} + \epsilon) \right] = \prod_ {i \in S} 1 = 1 \\ \operatorname {V a r} _ {\epsilon} [ J (S, \hat {\pi} | \boldsymbol {x} + \epsilon) ] = \prod_ {i \in S} \left(1 ^ {2} + (\sigma / \tau) ^ {2}\right) - \prod_ {i \in S} 1 ^ {2} \tag {12} \\ = \left(1 + (\sigma / \tau) ^ {2}\right) ^ {| S |} - 1 \\ \end{array}
$$
$$
\mathbb {E} _ {\epsilon} \left[ I (S | \boldsymbol {x} + \epsilon) \right] = \mathbb {E} _ {\epsilon} \left[ U _ {S, \hat {\pi}} \cdot J (S, \hat {\pi} | \boldsymbol {x} + \epsilon) \right] = U _ {S, \hat {\pi}}
$$
$$
\operatorname {V a r} _ {\epsilon} [ I (S | \boldsymbol {x} + \epsilon) ] = \operatorname {V a r} _ {\epsilon} \left[ U _ {S, \hat {\pi}} \cdot J (S, \hat {\pi} | \boldsymbol {x} + \epsilon) \right] = U _ {S, \hat {\pi}} ^ {2} \left(\left(1 + (\sigma / \tau) ^ {2}\right) ^ {| S |} - 1\right) \tag {13}
$$
According to Theorem 2, given an arbitrary input sample  $x'$ , we have
$$
J (S, \boldsymbol {\pi} | \boldsymbol {x} ^ {\prime}) = \prod_ {i \in S} \left(\operatorname {s i g n} \left(x _ {i} ^ {\prime} - b _ {i}\right) \cdot \frac {x _ {i} ^ {\prime} - b _ {i}}{\tau}\right) ^ {\pi_ {i}} \tag {14}
$$
Let us add a Gaussian perturbation  $\epsilon \sim \mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})$  to the input sample  $\pmb{x}$ . In this way, we have
$$
\begin{array}{l} J (S, \boldsymbol {\pi} | \boldsymbol {x} + \boldsymbol {\epsilon}) = \prod_ {i \in S} \left(\operatorname {s i g n} \left(x _ {i} + \epsilon_ {i} - b _ {i}\right) \cdot \frac {x _ {i} + \epsilon_ {i} - b _ {i}}{\tau}\right) ^ {\pi_ {i}} \tag {15} \\ = \prod_ {i \in S} \left(\operatorname {s i g n} \left(x _ {i} + \epsilon_ {i} - b _ {i}\right) \cdot \frac {x _ {i} - b _ {i}}{\tau} + \operatorname {s i g n} \left(x _ {i} + \epsilon_ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \\ \end{array}
$$
According to the setting of the baseline value,  $\forall i\in S,x_{i} - b_{i}\in \{-\tau ,\tau \}$ . We assume that the perturbation is small, i.e.,  $\forall i\in S,|\epsilon_i|\ll \tau$ . In this way,  $\mathrm{sign}(x_i + \epsilon_i - b_i) = \mathrm{sign}(x_i - b_i)$ , and we can obtain
$$
\begin{array}{l} J (S, \boldsymbol {\pi} | \boldsymbol {x} + \boldsymbol {\epsilon}) = \prod_ {i \in S} \left(\operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {x _ {i} - b _ {i}}{\tau} + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} (16) \\ = \prod_ {i \in S} \left(1 + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \\ \Rightarrow \mathbb {E} _ {\epsilon} [ J (S, \pi | \boldsymbol {x} + \boldsymbol {\epsilon}) ] = \mathbb {E} _ {\epsilon} \left[ \prod_ {i \in S} \left(1 + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right] (17) \\ \operatorname {V a r} _ {\boldsymbol {\epsilon}} [ J (S, \boldsymbol {\pi} | \boldsymbol {x} + \boldsymbol {\epsilon}) ] = \operatorname {V a r} _ {\boldsymbol {\epsilon}} \left[ \prod_ {i \in S} \left(1 + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right] \\ \end{array}
$$
Since  $\forall i\in S,\epsilon_{i}$  is independent of each other, according to Proposition 1 and Eq. (17), we have
$$
\mathbb {E} _ {\epsilon} \left[ J (S, \pi | \boldsymbol {x} + \boldsymbol {\epsilon}) \right] = \prod_ {i \in S} \mathbb {E} _ {\epsilon_ {i}} \left[ \left(1 + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right]
$$
$$
\operatorname {V a r} _ {\epsilon} [ J (S, \pi | \boldsymbol {x} + \boldsymbol {\epsilon}) ] = \prod_ {i \in S} \mathbb {E} _ {\epsilon_ {i}} \left[ \left(1 + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {2 \pi_ {i}} \right] - \prod_ {i \in S} \left(\mathbb {E} _ {\epsilon_ {i}} \left[ \left(1 + \operatorname {s i g n} \left(x _ {i} - b _ {i}\right) \cdot \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right]\right) ^ {2} \tag {18}
$$
Since  $\operatorname{sign}(x_i - b_i) \in \{-1, 1\}$ , we have  $\mathbb{E}_{\epsilon_i} \left[ \left( 1 + \operatorname{sign}(x_i - b_i) \cdot \frac{\epsilon_i}{\tau} \right)^k \right] = \mathbb{E}_{\epsilon_i} \left[ \left( 1 + \frac{\epsilon_i}{\tau} \right)^k \right], \forall k \in \mathbb{N}^+$ . Therefore, we obtain
$$
\begin{array}{l} \mathbb {E} _ {\epsilon} \left[ J (S, \pi | \boldsymbol {x} + \boldsymbol {\epsilon}) \right] = \prod_ {i \in S} \mathbb {E} _ {\epsilon_ {i}} \left[ \left(1 + \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right] \\ = \mathbb {E} _ {\boldsymbol {\epsilon}} \left[ \prod_ {i \in S} \left(1 + \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right] \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {V a r} _ {\epsilon} [ J (S, \pi | \boldsymbol {x} + \boldsymbol {\epsilon}) ] = \prod_ {i \in S} \mathbb {E} _ {\epsilon_ {i}} \left[ \left(1 + \frac {\epsilon_ {i}}{\tau}\right) ^ {2 \pi_ {i}} \right] - \prod_ {i \in S} \left(\mathbb {E} _ {\epsilon_ {i}} \left[ \left(1 + \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right]\right) ^ {2} \\ = \operatorname {V a r} _ {\epsilon} \left[ \prod_ {i \in S} \left(1 + \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right]. \\ \end{array}
$$
G.3 Proof of Theorem 4 in the main paper
Theorem 4. Given an arbitrarily masked sample  $\mathbf{x}_T(\forall T \subseteq N)$ , the function  $C_S(\mathbf{x}_T)$  defined above can well fit the binary activation state of the concept  $S$  in the sample  $\mathbf{x}_T$ .
$$
\forall T \subseteq N, C (S | \boldsymbol {x} _ {T}) = \prod_ {i \in S} A _ {i} (\boldsymbol {x} _ {T}) = \mathbb {1} (S \subseteq T), \tag {19}
$$
Proof. We know that the function of the activation state on an arbitrary sample  $x'$  is given by
$$
C _ {S} \left(\boldsymbol {x} ^ {\prime}\right) = \sum_ {\boldsymbol {\pi} \in Q _ {S}} U _ {S, \boldsymbol {\pi}} J (S, \boldsymbol {\pi} | \boldsymbol {x} ^ {\prime}) / U _ {S},
$$
$$
\text {w h e r e} J (S, \pi | \boldsymbol {x} ^ {\prime}) = \prod_ {i \in S} \left(\operatorname {s i g n} \left(x _ {i} ^ {\prime} - b _ {i}\right) \cdot \frac {x _ {i} ^ {\prime} - b _ {i}}{\tau}\right) ^ {\pi_ {i}} \tag {20}
$$
Specifically, now we consider a masked sample  $\pmb{x}_T$ , and we will prove that  $C_S(\pmb{x}_T) = \mathbb{1}(S \subseteq T)$ . We consider the following two cases.
Case 1:  $S \subsetneq T$ . Then there exists some  $j \in S \setminus T$ . Since  $j \notin T$ , according to the masking rule of the sample  $\mathbf{x}_T$ , we have  $(\mathbf{x}_T)_j - b_j = 0$ . Since  $j \in S$ , we have
$$
\forall \pi \in Q _ {S}, \quad J (S, \pi | \boldsymbol {x} _ {T}) = \prod_ {i \in S} \left(\operatorname {s i g n} \left((\boldsymbol {x} _ {T}) _ {i} - r _ {i}\right) \cdot \frac {(\boldsymbol {x} _ {T}) _ {i} - b _ {i}}{\tau}\right) ^ {\pi_ {i}} = 0 \tag {21}
$$
In this way, we have  $C_S(\pmb{x}_T) = \sum_{\pmb{\pi} \in Q_S} U_{S,\pmb{\pi}} J(S, \pmb{\pi} | \pmb{x}_T) / U_S = 0$ .
Case 2:  $S \subseteq T$ . In this case,  $\forall i \in S$ , we have  $i \in T$ . According to the setting of the reference value in Section 2.3 of the main paper, we have  $(\boldsymbol{x}_T)_i - b_i \in \{\tau, -\tau\}$ . Then we have  $\mathrm{sign}((\boldsymbol{x}_T)_i - b_i) \cdot \frac{(\boldsymbol{x}_T)_i - b_i}{\tau} = 1$ ,  $\forall i \in S$ . This further implies that
$$
\forall \pi \in Q _ {S}, \quad J (S, \pi | \boldsymbol {x} _ {T}) = \prod_ {i \in S} \left(\operatorname {s i g n} \left((\boldsymbol {x} _ {T}) _ {i} - b _ {i}\right) \cdot \frac {(\boldsymbol {x} _ {T}) _ {i} - b _ {i}}{\tau}\right) ^ {\pi_ {i}} = 1. \tag {22}
$$
$$
C _ {S} \left(\boldsymbol {x} _ {T}\right) = \sum_ {\boldsymbol {\pi} \in Q _ {S}} U _ {S, \boldsymbol {\pi}} / U _ {S}. \tag {23}
$$
Recall that  $U_S = I(S|\pmb{x}) = \sum_{\pmb{\pi} \in Q_S} U_{S,\pmb{\pi}} J(S, \pmb{\pi}|\pmb{x})$ , in which  $J(S, \pmb{\pi}|\pmb{x}) = \prod_{i \in S} (\mathrm{sign}(x_i - r_i) \cdot \frac{x_i - b_i}{\tau})^{\pi_i} = 1$ , because  $x_i - b_i \in \{\tau, -\tau\}$ . So actually, we have
$$
U _ {S} = \sum_ {\boldsymbol {\pi} \in Q _ {S}} U _ {S, \boldsymbol {\pi}}. \tag {24}
$$
Therefore,  $C_S(\pmb{x}_T) = \sum_{\pmb{\pi} \in Q_S} U_{S,\pmb{\pi}} / U_S = 1$ .
Combining the two cases, we can conclude that  $C_S(\pmb{x}_T) = \mathbb{1}(S \subseteq T) = \prod_{i \in S} A_i(\pmb{x}_T)$ .
G.4 Relation between interactive concepts and multi-order interactions
In this section, we derive that high-order interactive concepts (computed via the Harsanyi dividend [18]) can be considered as elementary components for high-order interactions used in [43].
Given a pre-trained DNN  $v$  and a masked sample  $\pmb{x}_S$ , the multi-order interaction  $I^{(m)}(i,j)$  used in [43] is given as follows: