SlowGuess/NeurIPS_2023_OCR huggingface dataset:
all belong to the same subset  $\Omega_c \subseteq \Omega_{\text{salient}}$ . Therefore, we can consider the interactive concepts as common knowledge learned by the DNN, and we aim to analyze the difficulty of learning each interactive concept with different complexity.
To facilitate the analysis, we first simplify the conceptual learning as a linear regression problem. Specifically, we first rewrite the interaction effect of an interactive concept  $S$ . Given an input sample  $\mathbf{x}$ , according to Equation (4), the interaction effect of the concept  $S$  on the sample  $\mathbf{x}'$  (obtained by applying some transformations on  $\mathbf{x}$ ),  $I(S|\mathbf{x}')$ , can be rewritten as
$$
I (S \mid \boldsymbol {x} ^ {\prime}) = U _ {S} \cdot C _ {S} (\boldsymbol {x} ^ {\prime}), \tag {7}
$$
where the constant  $U_{S} = I(S|\pmb{x})$  denotes the interaction effect on the sample  $\pmb{x}$ , and the function for the triggering state on the transformed sample  $\pmb{x}'$  is given as  $C_{S}(\pmb{x}') = \sum_{\pi \in Q_{S}} U_{S,\pi} J(S, \pi |\pmb{x}') / U_{S}$ .
Theorem 4. Given an arbitrarily masked sample  $\pmb{x}_T(T \subseteq N)$ , the function  $C_S(\pmb{x}_T)$  defined above can be computed as the binary triggering state of the concept  $S$  in the sample  $\pmb{x}_T$ .
$$
\forall T \subseteq N, C _ {S} (\boldsymbol {x} _ {T}) = \prod_ {i \in S} A _ {i} (\boldsymbol {x} _ {T}) = \mathbb {1} (S \subseteq T), \tag {8}
$$
where  $A_{i}(\pmb{x}_{T}) \in \{0,1\}$  denotes whether the variable  $x_{i}$  is present  $A_{i}(\pmb{x}_{T}) = 1$  or being masked  $A_{i}(\pmb{x}_{T}) = 0$  in the sample  $\pmb{x}_{T}$ .
The function  $C_S(\pmb{x}_T)$  represents the triggering state of the concept  $S$  under an arbitrarily masking condition  $\forall T \subseteq N$ . Only when all variables in  $S$  are present under the masking condition  $T$ , the concept  $S$  is triggered  $C_S(\pmb{x}_T) = 1$ . If any of variables in  $S$  is masked, then the concept  $S$  will not be triggered  $C_S(\pmb{x}_T) = 0$ , yielding zero interaction effect  $I(S|\pmb{x}_T) = 0$ . Theorem 4 shows that given a masked sample  $\pmb{x}_T$ , the concept  $S$  is only triggered when the sample  $\pmb{x}_T$  contains all variables in  $S$ , i.e.,  $T \supseteq S$ .
Thus, inspired by [46], we can extend the conclusion to a continuous version that explains the output of the DNN as a linear regression of very few salient interactive concepts based on Equation (3).
$$
v \left(\boldsymbol {x} ^ {\prime}\right) = \sum_ {S \subseteq N} U _ {S} \cdot C _ {S} \left(\boldsymbol {x} ^ {\prime}\right) \approx \sum_ {S \in \Omega_ {\text {s a l i e n t}}} U _ {S} \cdot C _ {S} \left(\boldsymbol {x} ^ {\prime}\right), \tag {9}
$$
where the triggering state  $C_S(\pmb{x}')$  of each interactive concept  $S$  can be considered as an input dimension of the linear function, which reflects whether the input sample  $\pmb{x}'$  contains the concept  $S$ .
Therefore, the absolute value of the weight coefficient  $U_{S}$  in Equation (9) can be viewed as the strength of the DNN encoding the interactive concept  $S$ . Considering the sparsity of interactive concepts discussed in Section 3.1, most interactive concepts have ignorable coefficients  $U_{S} \approx 0$ , and not so many concepts  $S$  have large absolute value of  $|U_{S}|$ . Thus, we can consider that the DNN only learns a small number of salient interactive concepts.
3.2.2 Explaining the learning difficulty of concepts
Equation (9) in the previous subsection enables us to understand a DNN for the classification task as a pseudo-linear function. Then, if an input feature dimension has a stable value (i.e., the triggering state  $C_S$  of an interactive concept  $S$  stably being present/absent) across all samples in the same category, then we consider this feature dimension (i.e., the concept) is easy to learn. In comparison, when we extract the same concept from different samples in the same category, if this concept exhibits inconsistent interaction effects (e.g., such a concept does not consistently present or absent over different samples), then this feature dimension (i.e., the concept) is hard to learn.
Figure 3: Consistency  $\beta(S)$  of  $s$ -order interactive concepts. The curve shows the mean consistency  $\mathbb{E}_{S \subseteq N, |S| = s}[\beta(S)]$  over different interactive concepts of the  $s$ -th order, and the shade indicates the standard deviation  $\mathrm{Std}_{S \subseteq N, |S| = s}[\beta(S)]$ . Effects of low-order concepts are more consistent than those of high-order concepts.
Figure 4: Instability  $\kappa(S)$  of  $s$ -order interactive concepts to data variations. The curve shows the average instability  $\mathbb{E}_{S \subseteq N, |S| = s}[\kappa(S)]$  over different interactive concepts of the  $s$ -th order, and the shade represents the standard deviation  $\mathrm{Std}_{S \subseteq N, |S| = s}[\kappa(S)]$ . Effects of high-order concepts are more sensitive to data variations than those of low-order concepts.
To analyze the consistency/stability of the concept w.r.t data variations, we use small perturbations  $\epsilon$  to represent various inevitable variations in the data, such as shape deformation and object rotation variations. Theorem 3 shows that high-order interactive concepts are much more unstable to inevitable variations in the data than low-order interactive concepts. This makes high-order concepts more likely to be influenced by data variations and less likely to be consistently present or absent in samples of the same category, which boosts the learning difficulty.
Experimental verification. We conduct experiments to verify the claim that high-order interactive concepts are less stably extracted under data variations than low-order interaction. We use the following two metrics to evaluate the each interactive concept  $S$  (i.e., each single input dimension of the above linear function) of a certain order. Specifically, we use the first metric  $\beta(S) = \mathbb{E}_c\left[\left|\mathbb{E}_{\boldsymbol{x} \in \boldsymbol{X}_c}[I(S|\boldsymbol{x})]\right| / \operatorname{Std}_{\boldsymbol{x} \in \boldsymbol{X}_c}[I(S|\boldsymbol{x})]\right]$  to measure the relative consistency of the interactive concept appearing over different input samples in a certain category  $c$ . Here,  $\boldsymbol{X}_c$  denotes a set of training samples belonging to the category  $c$ , and  $\operatorname{Std}_{\boldsymbol{x} \in \boldsymbol{X}_c}[I(S|\boldsymbol{x})]$  indicates the standard deviation of  $I(S|\boldsymbol{x})$  over different input samples. A large  $\beta(S)$  value means that the interactive concept  $S$  in all samples of the category  $c$  has similar/consistent effects  $I(S|\boldsymbol{x})$ . It is easier for a DNN to learn such a consistent concept  $S$ .
In addition, we use another metric  $\kappa(S)$  to verify whether the interaction effect of the high-order interactive concept is usually less stably extracted than those of the low-order interactive concept. To this end, the metric  $\kappa(S) = \mathbb{E}_{\boldsymbol{x} \in \boldsymbol{X}}[\mathbb{E}_{\boldsymbol{\epsilon}}[|I(S|\boldsymbol{x} + \boldsymbol{\epsilon}) - I(S|\boldsymbol{x})|]] / \mathbb{E}_{\boldsymbol{x} \in \boldsymbol{X}}[|I(S|\boldsymbol{x})|] = \mathbb{E}_{\boldsymbol{x} \in \boldsymbol{X}}[\mathbb{E}_{\boldsymbol{\epsilon}}[|C_S(\boldsymbol{x} + \boldsymbol{\epsilon}) - C_S(\boldsymbol{x})|]] / \mathbb{E}_{\boldsymbol{x} \in \boldsymbol{X}}[|C_S(\boldsymbol{x})|]$  measures the relative instability of the interactive concept  $S$  to the inevitable slight variations  $\boldsymbol{\epsilon}$  in the data. Here,  $C_S(\boldsymbol{x} + \boldsymbol{\epsilon}) = I(S|\boldsymbol{x} + \boldsymbol{\epsilon}) / U_S$  computed in Equation (7) denotes the trigger state of the interactive concept  $S$  on the sample  $\boldsymbol{x}' = \boldsymbol{x} + \boldsymbol{\epsilon}$ .
To this end, we use DNNs and experimental settings in the experimental verification paragraph of Section 3.2 for evaluation. Figure 3 and Figure 4 show the change of the average consistency  $\beta (S)$  and the average instability  $\kappa (S)$  of  $s$ -order interactive concepts through the learning process, respectively. At each training epoch, low-order concepts usually obtain higher consistency  $\beta (S)$  and
Figure 5: The weighted Jaccard similarity  $\mathrm{sim}(I_t^{(s)}, I_{\mathrm{final}}^{(s)})$  between  $s$ -order interactive concepts learned after the intermediate epoch  $I_t^{(s)}$  and those learned after the final epoch  $I_{\mathrm{final}}^{(s)}$ . Low-order concepts usually have higher Jaccard similarity during the learning process, which indicates that DNNs first learn low-order concepts and then gradually learn more about high-order concepts.
lower instability  $\kappa(S)$  than high-order concepts. It means that low-order concepts usually are more consistent and more stable. Thus, this experiment explains the reason why low-order interactive concepts are easier to learn.
3.2.3 Fast learning of low-order concepts
In this subsection, we illustrate the phenomenon that low-order interactive concepts are usually learned faster than high-order concepts, as a support for the claim that low-order interactive concepts are easier to be learned. We have theoretically proven in Section 3.2 and experimentally verified that low-order interactive concepts are more consistently present or consistently absent in different samples of the same category, which makes low-order interactive concepts easier to be learned. Then, this experiment is conducted to check whether low-order concepts are really learned faster than high-order concepts.
Specifically, we examine whether interactive concepts encoded by the finally-learned DNN  $v_{\mathrm{final}}(\pmb{x})$  have already been encoded by the DNN that is  $v_{t}(\pmb{x})$  trained after  $t$  epochs. If so, we consider such interactive concepts are learned fast. Specifically, let  $I_{t}^{(s)} = [I_{t}(S_{1}|\pmb{x}),\dots ,I_{t}(S_{d}|\pmb{x})]^{\top}\in \mathbb{R}^{d}$  denote a vector of interaction effects for all  $d = \binom{n}{s}$  interactive concepts of  $s$ -order. Then, we compute the Jaccard similarity between  $s$ -order interactive concepts encoded by the DNN  $v_{\mathrm{final}}(x)$  and those encoded by the DNN  $v_{t}(x)$ , i.e.,  $\mathrm{sim}(I_t^{(s)},I_{\mathrm{final}}^{(s)}) = \| \min (\widetilde{I}_t^{(s)},\widetilde{I}_{\mathrm{final}}^{(s)})\| _1 / \| \max (\widetilde{I}_t^{(s)},\widetilde{I}_{\mathrm{final}}^{(s)})\| _1$  and  $\| \cdot \| _1$  denotes L1-norm. We extend the  $d$ -dimensional vector  $I_{t}^{(s)}$  into a 2d-dimensional vector with non-negative elements  $\widetilde{I}_t^{(s)} = [(I_t^{(s),+})^\top ,(I_t^{(s),-})^\top ]^\top = [\max (I_t^{(s)},0)^\top , - \min (I_t^{(s)},0)^\top ]^\top \in \mathbb{R}^{2d}$ . Similarly, we compute  $\widetilde{I}_{\mathrm{final}}^{(s)}$  with non-negative elements based on  $I_{\mathrm{final}}^{(s)}$ . In this way, a large similarity  $\mathrm{sim}(I_t^{(s)},I_{\mathrm{final}}^{(s)})$  at an earlier epoch  $t$  indicates that interactive concepts are easier to be learned.
To this end, let us use DNNs and experimental settings introduced in the experimental verification paragraph of Section 3.2 for evaluation. Figure 5 shows that DNNs first learn low-order interactive concepts, and then learn high-order interactive concepts. Such a phenomenon verifies the conclusion that a DNN easily learns low-order (simple) interactive concepts.
4 Explaining findings in previous studies
4.1 Explaining adversarial robustness
Previous study [43] has discovered that low-order interactions are more robust to adversarial attacks than high-order interactions. Notice that their high-order (low-order) interactions are different from, but highly related to our high-order (low-order) interactive concepts. Specifically, our interactive concepts can be explained as elementary components for such multi-order interactions in [43]. Please see the supplementary material for the proof. Therefore, from this perspective, our conclusion that "low-order interactive concepts are easy to learn" can also explain how a DNN encodes concepts of different adversarial robustness.
Figure 6: Average adversarial sensitivity  $A^{(s)}$  of  $s$ -order interactive concepts to adversarial perturbations. Low-order interactive concepts are usually much less sensitive to adversarial attacks than high-order interactive concepts.
Can we use interactive concepts in this paper to verify the heuristic findings of adversarial robustness in [43]? According to Equation (2), we can represent the inference logic of a DNN by a set of interactive concepts, i.e.,  $v(\pmb{x}) \approx \sum_{S \in \Omega_{\text{salient}}} I(S|\pmb{x})$ . In this way, we conduct experiments to evaluate the sensitivity of each interactive concept  $S$  to adversarial perturbations. The symbolic conceptual representation of interactive concepts allows us to measure the adversarial robustness in a more direct way. That is, if the effect of an interactive concept does not significantly change under adversarial attacks, we consider this interactive concept is robust to adversarial attacks. In contrast, if the effect of an interactive concept changes significantly under adversarial attacks, we consider this interactive concept is sensitive to adversarial attacks.
To this end, we use the metric  $\alpha(S) = \mathbb{E}_{\boldsymbol{x} \in \mathbf{X}}[|I(S|\boldsymbol{x} + \delta) - I(S|\boldsymbol{x})|] / \mathbb{E}_{\boldsymbol{x} \in \mathbf{X}}[|I(S|\boldsymbol{x})|]$  to evaluate the sensitivity of the interactive concept  $S$  to adversarial perturbations, where  $\delta$  denotes the adversarial perturbation generated by the  $\ell_{\infty}$  attack [33]. In this way, a small  $\alpha(S)$  value indicates that the interactive concept  $S$  is robust to the adversarial attack.
We follow experimental settings in the experimental verification paragraph of Section 3.2 to evaluate different DNNs. Figure 6 shows that compared to high-order interactive concepts, low-order concepts usually obtain smaller  $A^{(s)} = \mathbb{E}_{S\subseteq N,|S| = s}[\alpha (S)]$  values. Such phenomena demonstrate that low-order interactive concepts are more robust to adversarial attacks.
4.2 Connections to existing findings on what are learned first by a DNN
In this subsection, we discuss some related studies on which kind of knowledge is usually first learned by a DNN. Most previous studies conducted experiments to explore the knowledge that was easier to be learned by a DNN, without providing much theoretical support. However, we find that our theorems can partially explain mechanisms behind some previous findings.
- Arpit et al. [5] trained DNNs to classify both normal samples and white-noise samples to different object categories. In this way, they considered that the DNN encoded simple concepts to classify normal samples, but the DNN had to learn complex concepts to classify white noises to randomly-assigned labels. They observed that the DNN usually learned normal samples first, because the classification accuracy of normal samples increased before that of white noises. To this end, our research provides more insights into such an observation. Specifically, Cheng et al. [8] have proven that the classification of noisy data usually depends on high-order concepts, i.e., the classification of noisy data forces the DNN to memorize each specific white-noise sample as a specific high-order interactive concept. Let us combine this conclusion with our finding that high-order interactive concepts are hard to learn. Then, we can easily owe the slow learning of white-noise samples observed in [5] to the difficulty of learning high-order concepts.
- Mangalam and Prabhu [34] considered/defined easy samples as training samples that could be correctly classified by shallow machine learning models, such as support vector machine (SVM) and random forests (RF). They discovered that DNNs first learned easy samples, and then gradually learned more about hard samples. To this end, our research verifies such observation. Specifically, we claim that easy samples mainly contain low-order interactive concepts. Thus, hard samples mainly contain high-order interactive concepts corresponding to complex interactions between numerous
input variables, and thus are difficult to be classified by shallow models (e.g., the SVM and the RF). In this way, the fast learning of low-order concepts is another understanding of the finding in [34].
- Xu et al. [63] discovered that during the training process, DNNs usually first learned samples of low frequencies (e.g., robust to noises), and then encoded samples of high frequencies (e.g., sensitive to noises). However, the original design of DNNs is not towards learning specific spectrums, and techniques of deep learning are not developed by assuming a periodic loss landscape of training samples. Therefore, we believe there should be a more direct explanation for the spectrum-learning phenomenon discovered by [63]. To this end, our research explains this phenomenon as the difficulty of learning high-order concepts. According to Section 4.1, low-order concepts usually are less sensitive to input perturbations, thereby corresponding to low-frequency components defined in the loss landscape in [63]. Accordingly, high-order concepts correspond to high-frequency components.
- Liu et al. [28] discovered that during adversarial training, the training loss of the DNN trained on easy samples decreased faster than that of the DNN trained on hard samples. To this end, we consider easy samples in adversarial training mentioned by [28] may mainly contain low-order interactive concepts. It is because, as discussed in Section 4.1, [43] discovered that low-order interactions were robust to adversarial perturbations. Thus, the fast learning of easy samples in adversarial training can be roughly owing to the learning of low-order concepts.
In this paper, we theoretically explain the trend of DNNs learning simple concepts. Since the emergence of interactive concepts of DNNs has been observed [45, 27] and proved [47], we aim to theoretically explain the explicit theoretical connection between the conceptual complexity and the difficulty of learning concepts. In this way, we prove that low-order interactive concepts in the data are much more stable than high-order interactive concepts, which makes low-order interactive concepts more likely to be encoded. Besides, our research can also provide new insights into several previous empirical understandings [5, 34, 63, 28] w.r.t. the conceptual representation of DNNs.
There are very few ways to define and examine what is a "concept." In this paper, we only use the following three properties to support the faithfulness of using sparse salient interactions as concepts encoded by the DNN.
- Ren et al. [47] have proved that a well-trained DNN can encode just a small number of salient interactions for inference under some common conditions.
- Ren et al. [45] have proved that, such a small number of salient interactions extracted from a sample can well mimic DNN's outputs on numerous masked samples.
- Li and Zhang [27] have discovered that salient interactions have considerable transferability and strong discrimination power.
However, the above properties cannot guarantee a clear correspondence between an interactive concept and a concept in human cognition. Up to now, we cannot mathematically formulate what is a concept in cognitive science. Thus, there is still a long way to unify the learning difficulty of a DNN and the cognitive difficulty of human beings.
This work is partially supported by the National Nature Science Foundation of China (62276165), National Key R&D Program of China (2021ZD0111602), Shanghai Natural Science Foundation (21JC1403800,21ZR1434600), National Nature Science Foundation of China (U19B2043). This paper is also partially supported by Huawei Technologies Inc.
[1] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40(12):2897-2905, 2018.
[2] Sravanti Addepalli, Anshul Nasery, Venkatesh Babu Radhakrishnan, Praneeth Netrapalli, and Prateek Jain. Feature reconstruction from outputs can mitigate simplicity bias in neural networks. 2022.
A Literature on the representation capacity of DNNs
Formulating and evaluating the representation ability of DNNs is an emerging perspective to explain DNNs. Pascanu et al. [38] and Montufar et al. [35] evaluated the representation capacity of DNNs based on the number of linear response regions. Kornblith et al. [21], Raghu et al. [42], and Morcos et al. [36] analyzed representations similarity between DNNs by using canonical correlation analysis. Shen et al. [52], Chen et al. [7], and Liu et al. [30] measured the quality of knowledge representations encoded in DNNs for point cloud processing. The information-bottleneck theory [54] quantified information encoded in DNNs, and was extended to improve the representation capacity of DNNs [1, 3]. Xu [64] and Xu et al. [63] explained the generalization of DNNs from the perspective of Fourier analysis. Furthermore, several metrics were proposed to evaluate the robustness or generalization capacity of DNNs, including the flatness of loss functions at minima [13], the stability of optimization [17], the CLEVER score [62], the stiffness [14], and the sensitivity metrics [37].
In contrast to previous empirical studies, we mathematically formulate concepts encoded by DNNs and theoretically prove that DNNs mainly learn simple concepts.
B Literature on interactions
Many studies investigated interactions between input variables of DNNs in recent years. Grabisch and Roubens [16] proposed the Shapley interaction index to measure the interaction between players in a cooperative game, based on the Shapley value [51]. Lundberg et al. [32] used the Shapley interaction index to analyze tree ensembles. Sundararajan et al. [57] proposed the Shapley-Taylor interaction index, and Tsai et al. [58] defined Faith-Shap, which was another interaction index.
In this paper, we use interactions between input variables of a DNN to represent concepts encoded by the DNN. In this way, We theoretically explain and empirically verify that DNN is easier to learn simple interactive concepts.
C Literature on the shortcut learning and simplicity bias of DNNs
Many studies focused on the underlying principles and limitations of DNNs. Geirhos et al. [15], Robinson et al. [48], and Scimeca et al. [49] investigated the shortcut learning in DNNs. I.e., DNNs may find decision rules that perform well on standard datasets, but fail to generalize to real-world scenarios. For example, DNNs successfully detected pneumonia by identifying hospital-specific metal tokens on the scan, but they failed to learn much about pneumonia and achieved low performance for scans from novel hospitals. Meanwhile, many studies proposed simplicity bias, i.e., standard training procedures for DNNs have a bias towards learning simple models [50, 20, 40]. Specifically, DNNs tend to learn low-rank embeddings [20] or simple features [50, 29, 40, 2].
From the perspective of the simplicity bias, our study considers the definition of interactive concepts in [47], and analyzes the bias towards learning simple concepts. This work clarifies an exact form of the complexity of concepts that a DNN is difficult to learn.
D Axioms and theorems for the Harsanyi dividend interaction
In this section, we introduce that the Harsanyi dividend interaction  $I(S)$  satisfies several desirable axioms and theorems.
The Harsanyi dividend interactions  $I(S)$  satisfies the efficiency, linearity, dummy, symmetry, anonymity, recursive and interaction distribution axioms, as follows.