SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongrui Liu $^{a*}$  Huiqi Deng $^{a*}$  Xu Cheng $^{a}$  Qihan Ren $^{a}$  Kangrui Wang $^{b}$  Quanshi Zhang $^{a\dagger}$ $^{a}$ Shanghai Jiao Tong University  $^{b}$ University of Chicago
This paper theoretically explains the intuition that simple concepts are more likely to be learned by deep neural networks (DNNs) than complex concepts. In fact, recent studies have observed [45, 27] and proved [47] the emergence of interactive concepts in a DNN, i.e., it is proven that a DNN usually only encodes a small number of interactive concepts, and can be considered to use their interaction effects to compute the inference score. Each interactive concept is encoded by the DNN to represent the collaboration between a set of input variables. Therefore, in this study, we aim to theoretically explain that interactive concepts involving more input variables (i.e., more complex concepts) are more difficult to learn. Our finding clarifies the exact conceptual complexity that boosts the learning difficulty.
Deep neural networks (DNNs) have exhibited superior performance in various tasks, but the reason for their superior performance remains an open problem. To this end, many attempts have been made to explain the representation capacity of DNNs from different perspectives. For example, Montufar et al. [35] used the number of linear response regions in a DNN to evaluate the expressive power of the DNN. Dinh et al. [13] and Petzka et al. [39] used the flatness of loss functions at minima to explain the generalization power.
In this paper, we explore a fundamental yet not well-formulated problem in terms of the representation capacity of DNNs, i.e., what types of concepts are easier to be learned by DNNs. However, the core challenge of this problem is that researchers have not reached a consensus on how to define a concept encoded by DNNs. Therefore, previous findings [5, 28, 34] that DNNs easily learn simple concepts remain intuition or empirical observations, without a clear theoretical formulation and explanation.
Fortunately, Ren et al. [45], Li and Zhang [27] have observed and Ren et al. [47] have, for the first time, mathematically proven that the emergence of interactive concepts is a common phenomenon shared by different DNNs. I.e., it is proved that under some common conditions, a well-trained DNN will encode just a small number of interactive concepts for inference. Specifically, the interactive concept encoded by a DNN is defined as a Harsanyi interaction [18] in game theory, which represents an AND relationship between input variables in a specific set  $S$ . For example, as Figure 1(a) shows, the DNN encodes the co-appearance relationship between input variables (image patches)  $x_{1}, x_{2}$ , and  $x_{3}$  to form the mouth concept  $S = \{x_{1}, x_{2}, x_{3}\}$ . Only when all three patches are all present, the mouth concept is triggered, and adds a numerical effect  $I(S)$  to the inference score  $y$ . The masking of any patches ( $e.g., x_{1}$ ) will break this AND/co-appearance relationship of the mouth
†This research is done under the supervision of Dr. Quanshi Zhang. He is with the Department of Computer Science and Engineering, the John Hopcroft Center at the Shanghai Jiao Tong University, China. Correspondence to: Quanshi Zhang <zqs1022@sjtu.edu.cn>.
Figure 1: (a) Interactive concepts encoded by a DNN. Each interactive concept  $S$  represents an AND relationship between input variables (image patches) in  $S$ . Masking any input variables in  $S$  will deactivate the concept  $S$ . (b) Extensive experiments illustrate the common concept-emergence phenomenon. Various DNNs all encode very sparse interactive concepts for inference. Only a small number of interactions have salient effects  $I(S)$  and are termed interactive concepts. Other interactions w.r.t. other  $S \subseteq N$  all have almost zero effect  $I(S) \approx 0$ , thus being termed noisy patterns. For clarity, we sort the strength of interaction effects in a descending order.
concept, and removes the numerical effect  $I(S)$ , i.e., making  $I(S) = 0$ . More crucially, it proved in [47] that the network output  $y$  on each input sample can be represented as the sum of numerical effects of such a few interactive concepts, i.e.,  $y = \sum_{S \in \text{ConceptSet}} I(S)$ .
The mathematical proof in [47] makes the interactive concept the first concept whose emergence in DNNs is certificated. Furthermore, Li and Zhang [27] have discovered that the interactive concepts have considerable transferability over different samples and strong discrimination power for classification. In this way, we can roughly consider such interactive concepts a rigorous definition of concepts encoded by DNNs.
Explaining the difficulty of learning complex interactive concepts. Therefore, in this study, given the above interactive concepts, we aim to further explore the learning difficulty of these concepts, and explain that a DNN is more likely to encode simple interactive concepts. Here, we use the number of variables in an interactive concept  $S$  to represent the complexity of the concept, which is also termed the order of the concept. Thus, low-order interactive concepts usually represent simple AND interactions among a few input variables.
Specifically, we find that low-order interactive concepts encoded by DNNs are generally more stable to inevitable noises in data. In this way, low-order interactive concepts usually exhibit consistently positive (or negative) effects on the inference score of different samples in the same category. In comparison, a high-order interactive concept is more likely to have significantly diverse effects on inference scores of different samples. This explains the reason why DNNs easily learn simple interactive concepts.
In addition, we discover that our research may provide a new perspective to explain previous findings/understandings of the conceptual complexity [5, 34, 63, 28]. Besides, the conceptual complexity can directly explain the adversarial robustness of a DNN. Thus, our study is of considerable value in explaining the representation capacity of a DNN.
Interactive concepts vs. cognitive concepts. Although the Harsanyi interactive concepts visualized by [27] seem partially aligned with cognitive concepts to some extent, we do not think such interactive concepts fit humans' cognition [24, 53]. For example, the recognition of a big ball concept and that of a small ball concept have almost the same cognitive complexity for humans. In comparison, it is more difficult for the DNN to recognize a large ball, because it has to use a high-order interaction that contains a lot of pixels to examine whether all pixels within the ball have the same color. Thus, this study clarifies the exact formulation for the conceptual complexity on interactive concepts that boosts the learning difficulty, which is different from the cognitive difficulty.
2 Explainable AI (XAI) theories based on game-theoretic interactions
Explaining DNNs based on a system of game-theoretic interactions becomes an emerging direction in recent years. Specifically, such a system aims to tackle the following challenges in XAI.
- Defining, extracting, and quantifying interactive concepts encoded by DNNs. Quantifying the interactions between different input variables [57, 58] is a new perspective to formulate the knowledge encoded by a DNN. Based on game theory, Zhang et al. [66, 68, 67] introduced multi-variate interaction and multi-order interaction to quantify the interactions encoded by the DNN. Ren et al. [45] empirically observed and Ren et al. [47] further proved that a DNN usually just encoded a small number of salient interactions. Ren et al. [44] defined the optimal baseline value for computing the Shapley value based on these interactions. Li and Zhang [27] found that salient interactions usually have high transferability over different samples and DNNs. These findings showed that these salient interactions could be viewed as interactive concepts encoded by a DNN.
- Using interactive concepts to explain/measure the representation power of DNNs. The representation power of DNNs has been explained by game-theoretic multi-order interaction, including generalization power [67, 70], adversarial robustness [45, 70], and adversarial transferability [60]. Besides, Cheng et al. [8] used interactions to investigate how shapes and textures were encoded in DNNs. Cheng et al. [9] found that salient interactions often represented prototypical patterns encoded by DNNs. Deng et al. [11] derived that the DNN was less likely to encode mid-order interactions. In comparison, Ren et al. [46] proved that the Bayesian neural network was less likely to encode high-order interactions, thereby obtaining good adversarial robustness.
- Unifying the common underlying mechanism shared by previous empirical findings. Interactions have superior representation power, and we find that many previous studies can be re-explained from the perspective of interactions. Deng et al. [12] proved that fourteen attribution methods could be reformulated and considered as a certain allocation of interaction effects. Besides, Zhang et al. [69] proved that reducing the interactions was the common underlying mechanism shared by twelve previous studies for improving adversarial transferability.
3 Explaining simple interactive concepts are easy to learn
3.1 Preliminaries: sparse interactive concepts emerge in DNNs
It is generally believed that the learning of DNNs is a fitting problem between the ground-truth label and the model output, instead of explicitly learning specific concepts like graphical models. However, Ren et al. [45], Li and Zhang [27] have empirically observed and Ren et al. [47] have mathematically proven the counter-intuitive emergence of concepts in DNNs, i.e., it is proven that under a set of common conditions, a DNN encodes just a small number of interactive concepts for inference.
Specifically, given a pre-trained DNN  $v$  and an input sample  $\pmb{x} = [x_1,\dots,x_n]$  with  $n$  variables indexed by  $N = \{1,\ldots ,n\}$ ,  $v(\pmb {x})\in \mathbb{R}$  denotes a scalar output of the DNN<sup>3</sup>. Each interaction in [45, 27, 47] is defined as Harsanyi dividend (or Harsanyi interaction) [18] in game theory, which represents a collaboration (AND relationship) between input variables in a specific set  $S$  ( $S\subseteq N$ ). For example, as Figure 1(a) shows, the co-appearance of image patches forms a mouth interaction  $S = \{x_{1},x_{2},x_{3}\}$ . Only when these three patches are all present, the mouth interaction will be triggered, and make a certain interaction effect  $I(S)$  on the network output. The absence (masking) of any patches of  $x_{1},x_{2}$ , and  $x_{3}$  will deactivate the mouth interaction and remove the interaction effect, i.e.,  $I(S) = 0$ . The interaction effect  $I(S|\pmb {x})$  on the input sample  $\pmb{x}$  is computed as follows.
$$
I (S | \boldsymbol {x}) = \sum_ {T \subseteq S} (- 1) ^ {| S | - | T |} \cdot v (\boldsymbol {x} _ {T}), \tag {1}
$$
where  $x_{T}$  denotes the masked input sample, when we mask input variables in  $N \setminus T$  and keep variables in  $T$  unchanged. Please see more properties of the Harsanyi dividend in the supplementary material.
Theoretically, there are  $2^{n}$  potential subsets  $S$  of input variables  $(S\in 2^{N} = \{S|S\subseteq N\})$ . The proven concept-emergence phenomenon refers to that only a small number of subsets  $S\in \Omega_{\mathrm{salient}}\subseteq 2^{N}$  make salient interaction effects  $I(S|\pmb {x})$  on the network output, and can be considered as interactive concepts. Interaction effects of all other subsets are close to zero  $(I(S|x)\approx 0)$ , which can be considered as ignorable noisy patterns. Therefore, the network output  $v(x)$  can be well approximated by interaction effects of a small number of interactive concepts, i.e.,
$$
v (\boldsymbol {x}) = \sum_ {S \subseteq N} I (S | \boldsymbol {x}) \approx \sum_ {S \in \Omega_ {\text {s a l i e n t}}} I (S | \boldsymbol {x}) \tag {2}
$$
3Here,  $v(\pmb {x})\in \mathbb{R}$  can be implemented as either a scalar output of the DNN or a dimension of an output vector (e.g., the confidence score of classifying the sample  $\pmb{x}$  to the ground-truth category  $v(\pmb {x}) = \log \frac{p(y = y_{\mathrm{truth}}|\pmb{x})}{1 - p(y = y_{\mathrm{truth}}|\pmb{x})}$ ).
We also conduct experiments to illustrate the concept-emergence phenomenon on various DNNs, including multi-layer perceptrons (MLPs), long short-term memory (LSTM), AlexNet [23], ResNet [19], convolutional neural networks (CNNs), and PointNet [41] trained on different types of data, including tabular data (UCI dataset [6]), natural language data (CoLA [61] and SST-2 [56]), image data (MNIST [26] and CelebA [31]), and point cloud data (ShapeNet [65]). We follow experimental settings in [45, 27], and Figure 1(b) shows that interactive concepts encoded by a DNN are usually indeed sparse.
Theorem 1. Given an input sample  $\mathbf{x} \in \mathbb{R}^n$  with  $n$  variables, there are  $2^n$  different masked samples  $\mathbf{x}_T$  w.r.t. all potential subsets  $T \subseteq N$ . Then, [45] has proven that
$$
\forall T \subseteq N, v (\boldsymbol {x} _ {T}) = \sum_ {S \subseteq T} I (S | \boldsymbol {x}) \approx \sum_ {S \in \Omega_ {\text {s a l i e n t}} \& S \subseteq T} I (S | \boldsymbol {x}). \tag {3}
$$
Theorem 1 means that we can use a small number of interactive concepts in  $\Omega_{\mathrm{salient}}$  to well approximate network outputs on anyone  $\pmb{x}_T$  of the  $2^{n}$  masked samples.
Trustworthiness of interactive concepts. The mathematical proof in [47] makes  $I(S)$  become the first concepts whose emergence in DNNs is certificated. Equations (2) and (3) further guarantee that interactive concepts can faithfully explain the output of DNNs. In addition, Li and Zhang [27] have discovered that (i) interactive concepts are transferable across different samples; (ii) interactive concepts are discriminative, i.e., if a concept  $S$  has salient interaction effects  $I(S)$  on a set of samples, then the concept tends to push these samples to be classified towards the same category. In this way, we can roughly consider such interactive concepts a relatively rigorous definition of concepts.
Complexity (order) of interactive concepts. The complexity of an interactive concept  $S$  is defined as the number of input variables involved in  $S$ , which is also termed the order of the interactive concept, i.e., complexity  $(S) = \text{order}(S) = |S|$ . Thus, low-order interactive concepts usually represent simple AND relationships among a few input variables. In comparison, high-order interactive concepts often refer to as relatively complex AND relationships among a large number of input variables.
3.2 Simple interactive concepts in data are more stable and easier to learn
Unlike other interaction metrics [16, 32, 57], sparse interactive concepts are certified to emerge in DNN, and can faithfully represent the inference score of DNNs. Therefore, in this study, we aim to further explore the learning difficulty of these certificated concepts. Specifically, we theoretically explain that a DNN is more likely to encode simple (i.e., low-order) interactive concepts.
In this subsection, we show that low-order interactive concepts encoded by DNNs are generally more stable to inevitable noises in data, compared to high-order interactive concepts. Specifically, we derive an approximate analytical solution to the variance (instability) of interactive concepts' effects w.r.t. data noise, and show that the variance (instability) increases along the order of concepts in an exponential manner.
Theorem 2 (proven in the supplementary material). Given a neural network  $v$  and an arbitrary input sample  $\pmb{x}' \in \mathbb{R}^n$ , the network output can be decomposed by using the Taylor expansion i.e.,  $v(\pmb{x}') = \sum_{S \subseteq N} \sum_{\pmb{\pi} \in Q_S} U_{S,\pmb{\pi}} \cdot J(S, \pmb{\pi} | \pmb{x}')$ . In this way, according to Equation (1), the Harsanyi interaction effect  $I(S | \pmb{x}')$  on the sample  $\pmb{x}'$  can be reformulated as follows.
$$
I (S | \boldsymbol {x} ^ {\prime}) = \sum_ {\boldsymbol {\pi} \in Q _ {S}} U _ {S, \boldsymbol {\pi}} \cdot J (S, \boldsymbol {\pi} | \boldsymbol {x} ^ {\prime}). \tag {4}
$$
Here,  $J(S,\pi |x^{\prime}) = \prod_{i\in S}\left(\mathrm{sign}(x_{i}^{\prime} - b_{i})\cdot \frac{x_{i}^{\prime} - b_{i}}{\tau}\right)^{\pi_{i}}$  denotes a Taylor expansion term of the degree  $\pmb{\pi}$  where the degree  $\pmb {\pi}\in Q_S = \{[\pi_1,\dots ,\pi_n]|\forall i\in S,\pi_i\in \mathbb{N}^+;\forall i\notin S,\pi_i = 0\}$  and  $b_{i}$  is the baseline value to mask the input variable  $x_{i}$ . In addition,  $U_{S,\pi} = \frac{\tau^{m}}{\prod_{i = 1}^{n}\pi_{i}!}\frac{\partial^{m}v(\pmb{x}_{\emptyset}^{\prime})}{\partial x_{1}^{\pi_{1}}\cdots\partial x_{n}^{\pi_{n}}}\cdot \prod_{i\in S}[\mathrm{sign}(x_i' - b_i)]^{\pi_i}$ , where  $\pmb{x}_{\emptyset}'$  denotes the sample whose input variables are all masked.  $m = \sum_{i = 1}^{n}\pi_{i}$ .
Theorem 2 rewrites the Harsanyi interaction effect  $I(S|x)$  of each concept from a new perspective, to facilitate the analysis of the instability of interactive concepts. The derivation of Theorem 2 is inspired by [12, 70, 46], in which the Harsanyi interaction effect is re-written as the sum of the Taylor interaction effects. Therefore, in this study, we define a new baseline value  $b_{i}$  to further extend the finding in [12] and obtain Theorem 2, which significantly simplifies the further proof. Specifically, the new baseline value  $b_{i}$  is set as follows to represent the masked state of the  $i$ -th input variable. i.e.,
$x_{i}\gets b_{i}$ . Previous studies usually set  $b_{i} = \mu_{i} = \mathbb{E}_{\boldsymbol{x}}[x_{i}]$  as the average value over different samples to represent the masked state [4, 12]. However, we consider that pushing the input variable  $x_{i}$  to move a large distance  $\tau \in \mathbb{R}$  towards  $\mu_{i}$  has been significant enough to remove its information. To this end, we set  $b_{i} = x_{i} - \tau^{4}$ , if  $x_{i} > \mu_{i}$ ; else,  $b_{i} = x_{i} + \tau^{4}$ . The above new setting ensures comparable perturbation magnitudes over different dimensions.
Next, we analyze the variance (instability) of the interaction effect  $I(S|\pmb{x}' = \pmb{x} + \pmb{\epsilon})$  when we add a Gaussian perturbation  $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$  to the input sample  $\pmb{x}$ . Here, the Gaussian perturbation  $\epsilon \in \mathbb{R}^n$  is considered as a rough representation of inevitable variations in data, e.g., shape deformation and object rotation variations in image classification. However, such variations are quite difficult to formulate, so we use a Gaussian perturbation as a rough representation. Fortunately, we find that our reformulation of interactions in Theorem 2 enables us to directly apply the finding in [46] to prove the variance (instability) of the interaction effect  $I(S|\pmb{x}' = \pmb{x} + \pmb{\epsilon})$ .
Theorem 3. Let us add a Gaussian perturbation  $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$  to the input sample  $\pmb{x}$ . Let us first consider the case with the lowest degree  $\hat{\pi} = [\hat{\pi}_1, \dots, \hat{\pi}_n] \in Q_S$ , satisfying that  $\forall i \in S, \hat{\pi}_i = 1; \forall i \notin S, \hat{\pi}_i = 0$ . The mean and variance of  $J(S, \hat{\pi} | \pmb{x} + \pmb{\epsilon})$  over the Gaussian perturbation  $\epsilon$  are given as
$$
\mathbb {E} _ {\epsilon} [ J (S, \hat {\pi} | \boldsymbol {x} + \epsilon) ] = 1, \quad \operatorname {V a r} _ {\epsilon} [ J (S, \hat {\pi} | \boldsymbol {x} + \epsilon) ] = \left(1 + \left(\frac {\sigma}{\tau}\right) ^ {2}\right) ^ {| S |} - 1. \tag {5}
$$
Furthermore, for the more general case with an arbitrary degree  $\pi \in Q_S = \{[\pi_1,\dots ,\pi_n]|\forall i\in S,\pi_i\in$ $\mathbb{N}^+;\forall i\notin S,\pi_i = 0\}$ , the mean and variance of  $J(S,\pi |\pmb {x} + \pmb {\epsilon})$  are computed as
$$
\mathbb {E} _ {\epsilon} [ J (S, \pi | \boldsymbol {x} + \epsilon) ] = \mathbb {E} _ {\epsilon} \left[ \prod_ {i \in S} \left(1 + \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right], \operatorname {V a r} _ {\epsilon} [ J (S, \pi | \boldsymbol {x} + \epsilon) ] = \operatorname {V a r} _ {\epsilon} \left[ \prod_ {i \in S} \left(1 + \frac {\epsilon_ {i}}{\tau}\right) ^ {\pi_ {i}} \right]. \tag {6}
$$
Theorem 3 shows the (variance) instability of interactive concepts of different complexities (orders) w.r.t. data variations. It indicates that the variance of  $J(S, \pi | x + \epsilon)$  roughly increases along with the order  $(S) = |S|$  in an exponential manner. Similar conclusions are also introduced in [70]. Furthermore, according to Equation (4), the interaction effect  $I(S | x + \epsilon)$  of the concept  $S$  can be represented as the weighted sum of  $J(S, \pi | x + \epsilon)$ , and coefficients  $U_{S, \pi}$  w.r.t. different orders  $s$  and degrees  $\pi$  are usually chaotic. Hence, we can roughly consider that the variance (instability) of the interaction effect  $I(S | x + \epsilon)$  increases along with the order  $|S|$  exponentially, as well. In other words, compared to high-order concepts, low-order concepts are much more stable to slight input perturbations.
Experimental verification. Here, we conduct experiments to verify whether the variance (instability) of interaction effects indeed increased along with the order in an approximately exponential manner. Specifically, to mimic variations in the data, we add Gaussian perturbations  $\epsilon \sim \mathcal{N}(0, 0.02^2 I)$  to each training sample. Then, we compute the average mean  $E^{(s)} = \mathbb{E}_{\boldsymbol{x} \in \boldsymbol{X}}[\mathbb{E}_{S \subseteq N, |S| = s}[\mathbb{E}_{\boldsymbol{\epsilon}}[I(S|\boldsymbol{x} + \boldsymbol{\epsilon})]]]$  and the average variance  $V^{(s)} = \mathbb{E}_{\boldsymbol{x} \in \boldsymbol{X}}[\mathbb{E}_{S \subseteq N, |S| = s}[\mathrm{Var}_{\boldsymbol{\epsilon}}[I(S|\boldsymbol{x} + \boldsymbol{\epsilon})]]]$  of interaction effects of the  $s$ -th order concepts w.r.t. Gaussian perturbations. For verification, we calculate such metrics on DNNs trained for image classification and DNNs trained on tabular data. We train AlexNet [23], VGG-11 [55], ResNet-18/20 [19] on the CIFAR-10 dataset [22] and the Tiny-ImageNet dataset [25], respectively. We also train a five-layer MLP [45] on the UCI census dataset (namely census dataset) and the UCI TV news dataset (namely TV news dataset) [6], respectively. In addition, considering the computational cost of  $I(S|\boldsymbol{x} + \boldsymbol{\epsilon})$  in Equation (1) is intolerable if each pixel is considered as an input variable, we divide images into  $8 \times 8$  patches [45] to calculate  $I(S|\boldsymbol{x} + \boldsymbol{\epsilon})$ . Please see the supplementary material for more implementation details.
Figure 2 shows that the variance (instability)  $V^{(s)}$  of interaction effects increases in an exponential manner along with the order  $s$  of the interactive concept. In addition, the relative stability  $E^{(s)} / \sqrt{V^{(s)}}$  of interaction effects decreases significantly along with the order  $s$ . Such phenomenon successfully verifies findings in Theorem 3.
3.2.1 Formulating the conceptual learning as a linear regression problem
Note that, Li and Zhang [27] have discovered that interactive concepts have high transferability over samples in the same category, i.e., most interactive concepts extracted from different samples may
4We need to avoid the case of over-perturbation, by setting  $b_{i}\gets \max (b_{i},\mu_{i})$  if  $x_{i} > \mu_{i};b_{i}\gets \min (b_{i},\mu_{i})$  otherwise. Fortunately, such cases are uncommon in real applications, so we ignore such settings in the following theoretical analysis.
Figure 2: The logarithm of the variance of interaction effects  $\log [V^{(s)}]$  and the stability of interaction effects (measured by  $E^{(s)} / \sqrt{V^{(s)}}$ ). The variance  $V^{(s)}$  of interaction effects increases along with the order of interactive concepts exponentially. The stability of effects decreases along with the order.