Title: Towards the Difficulty for a Deep Neural Network to Learn Concepts of Different Complexities
Venue: NeurIPS 2023
Accepted: yes
Keywords: representation complexity

deep learning
Abstract: This paper theoretically explains the intuition that simple concepts are more likely to be learned by deep neural networks (DNNs) than complex concepts. In fact, recent studies have observed [24, 15] and proved [26] the emergence of interactive concepts in a DNN, i.e., it is proven that a DNN usually only encodes a small number of interactive concepts, and can be considered to use their interaction effects to compute inference scores. Each interactive concept is encoded by the DNN to represent the collaboration between a set of input variables. Therefore, in this study, we aim to theoretically explain that interactive concepts involving more input variables (i.e., more complex concepts) are more difficult to learn. Our finding clarifies the exact conceptual complexity that boosts the learning difficulty.

Towards the Difficulty for a Deep Neural Network to Learn Concepts of Different Complexities

Abstract

This paper theoretically explains the intuition that simple concepts are more likely to be learned by deep neural networks (DNNs) than complex concepts. In fact, recent studies have observed [45, 27] and proved [47] the emergence of interactive concepts in a DNN, i.e. , it is proven that a DNN usually only encodes a small number of interactive concepts, and can be considered to use their interaction effects to compute the inference score. Each interactive concept is encoded by the DNN to represent the collaboration between a set of input variables. Therefore, in this study, we aim to theoretically explain that interactive concepts involving more input variables ( i.e. , more complex concepts) are more difficult to learn. Our finding clarifies the exact conceptual complexity that boosts the learning difficulty.

1 Introduction

Deep neural networks (DNNs) have exhibited superior performance in various tasks, but the reason for their superior performance remains an open problem. To this end, many attempts have been made to explain the representation capacity of DNNs from different perspectives. For example, Montufar et al. [35] used the number of linear response regions in a DNN to evaluate the expressive power of the DNN. Dinh et al. [13] and Petzka et al. [39] used the flatness of loss functions at minima to explain the generalization power.

In this paper, we explore a fundamental yet not well-formulated problem in terms of the representation capacity of DNNs, i.e. , what types of concepts are easier to be learned by DNNs. However, the core challenge of this problem is that researchers have not reached a consensus on how to define a concept encoded by DNNs. Therefore, previous findings [5, 28, 34] that DNNs easily learn simple concepts remain intuition or empirical observations, without a clear theoretical formulation and explanation.

Fortunately, Ren et al. [45], Li and Zhang [27] have observed and Ren et al. [47] have, for the first time, mathematically proven that the emergence of interactive concepts is a common phenomenon shared by different DNNs. I.e. , it is proved that under some common conditions, a well-trained DNN will encode just a small number of interactive concepts for inference. Specifically, the interactive concept encoded by a DNN is defined as a Harsanyi interaction [18] in game theory, which represents an AND relationship between input variables in a specific set S . For example, as Figure 1(a) shows, the DNN encodes the co-appearance relationship between input variables (image patches) x 1 , x 2 , and x 3 to form the mouth concept S = { x 1 , x 2 , x 3 } . Only when all three patches are all present, the mouth concept is triggered, and adds a numerical effect I ( S ) to the inference score y . The masking of any patches ( e.g. , x 1 ) will break this AND/co-appearance relationship of the mouth

𝒙𝒙

𝟏𝟏

𝒙𝒙

𝟖𝟖

𝒙𝒙

𝟕𝟕

𝒙𝒙

𝟐𝟐

𝒙𝒙

𝟑𝟑

𝒙𝒙

𝟏𝟏𝟑𝟑

Training a PointNet on

5

0

0

ShapeNet dataset

TV news dataset

CoLAdataset

(a)

(b)

salient

noise

2

n

|

𝐼𝐼

𝑆𝑆

|

40

0

|

𝐼𝐼

𝑆𝑆

|

Training an MLP on

salient

noise

2

n

0

Training a LSTM on

16

0

2

n

salient

noise

0

0

2

n

Training a ResNet-32 on

MNIST dataset

salient

noise

7

0

0

2

n

Training a CNN on

SST-2 dataset

salient

noise

25

0

2

n

0

masked image

(

𝑥𝑥1

) = 0

Trigger(S

2

) = 1

7-order

concept

3-order

concept

(

𝑥𝑥8

) = 1

(

𝑥𝑥7

) = 0

3

=

�

𝑖𝑖=7

13

𝑥𝑥

𝑖𝑖

= 0

Different interactions

Different interactions

Different interactions

Different interactions

2

n

0

CelebA dataset

Training a ResNet-18 on

salient

noise

8

0

CelebA dataset

0

2

n

5

0

salient

noise

Training a AlexNet on

50

0

Training an MLP on

tic-tac-toe dataset

noise

salient

𝑥𝑥

𝑇𝑇

concept, and removes the numerical effect I ( S ) , i.e. , making I ( S ) = 0 . More crucially, it proved in [47] that the network output y on each input sample can be represented as the sum of numerical effects of such a few interactive concepts , i.e. , y = ∑ S ∈ ConceptSet I ( S ) .

The mathematical proof in [47] makes the interactive concept the first concept whose emergence in DNNs is certificated . Furthermore, Li and Zhang [27] have discovered that the interactive concepts have considerable transferability over different samples and strong discrimination power for classification. In this way, we can roughly consider such interactive concepts a rigorous definition of concepts encoded by DNNs.

Explaining the difficulty of learning complex interactive concepts. Therefore, in this study, given the above interactive concepts, we aim to further explore the learning difficulty of these concepts, and explain that a DNN is more likely to encode simple interactive concepts. Here, we use the number of variables in an interactive concept S to represent the complexity of the concept, which is also termed the order of the concept. Thus, low-order interactive concepts usually represent simple AND interactions among a few input variables.

Specifically, we find that low-order interactive concepts encoded by DNNs are generally more stable to inevitable noises in data. In this way, low-order interactive concepts usually exhibit consistently positive (or negative) effects on the inference score of different samples in the same category. In comparison, a high-order interactive concept is more likely to have significantly diverse effects on inference scores of different samples. This explains the reason why DNNs easily learn simple interactive concepts.

In addition, we discover that our research may provide a new perspective to explain previous findings/understandings of the conceptual complexity [5, 34, 63, 28]. Besides, the conceptual complexity can directly explain the adversarial robustness of a DNN. Thus, our study is of considerable value in explaining the representation capacity of a DNN.

Interactive concepts vs. cognitive concepts. Although the Harsanyi interactive concepts visualized by [27] seem partially aligned with cognitive concepts to some extent, we do not think such interactive concepts fit humans' cognition [24, 53]. For example, the recognition of a big ball concept and that of a small ball concept have almost the same cognitive complexity for humans. In comparison, it is more difficult for the DNN to recognize a large ball, because it has to use a high-order interaction that contains a lot of pixels to examine whether all pixels within the ball have the same color. Thus, this study clarifies the exact formulation for the conceptual complexity on interactive concepts that boosts the learning difficulty, which is different from the cognitive difficulty.

2 Explainable AI (XAI) theories based on game-theoretic interactions

Explaining DNNs based on a system of game-theoretic interactions becomes an emerging direction in recent years. Specifically, such a system aims to tackle the following challenges in XAI.

Defining, extracting, and quantifying interactive concepts encoded by DNNs. Quantifying the interactions between different input variables [57, 58] is a new perspective to formulate the knowledge encoded by a DNN. Based on game theory, Zhang et al. [66, 68, 67] introduced multi-variate interaction and multi-order interaction to quantify the interactions encoded by the DNN. Ren et al. [45] empirically observed and Ren et al. [47] further proved that a DNN usually just encoded a small number of salient interactions. Ren et al. [44] defined the optimal baseline value for computing the Shapley value based on these interactions. Li and Zhang [27] found that salient interactions usually have high transferability over different samples and DNNs. These findings showed that these salient interactions could be viewed as interactive concepts encoded by a DNN.

Using interactive concepts to explain/measure the representation power of DNNs. The representation power of DNNs has been explained by game-theoretic multi-order interaction, including generalization power [67, 70], adversarial robustness [45, 70], and adversarial transferability [60]. Besides, Cheng et al. [8] used interactions to investigate how shapes and textures were encoded in DNNs. Cheng et al. [9] found that salient interactions often represented prototypical patterns encoded by DNNs. Deng et al. [11] derived that the DNN was less likely to encode mid-order interactions. In comparison, Ren et al. [46] proved that the Bayesian neural network was less likely to encode high-order interactions, thereby obtaining good adversarial robustness.

Unifying the common underlying mechanism shared by previous empirical findings. Interactions have superior representation power, and we find that many previous studies can be re-explained from the perspective of interactions. Deng et al. [12] proved that fourteen attribution methods could be reformulated and considered as a certain allocation of interaction effects. Besides, Zhang et al. [69] proved that reducing the interactions was the common underlying mechanism shared by twelve previous studies for improving adversarial transferability.

3 Explaining simple interactive concepts are easy to learn

3.1 Preliminaries: sparse interactive concepts emerge in DNNs

It is generally believed that the learning of DNNs is a fitting problem between the ground-truth label and the model output, instead of explicitly learning specific concepts like graphical models. However, Ren et al. [45], Li and Zhang [27] have empirically observed and Ren et al. [47] have mathematically proven the counter-intuitive emergence of concepts in DNNs, i.e. , it is proven that under a set of common conditions, a DNN encodes just a small number of interactive concepts for inference.

Specifically, given a pre-trained DNN v and an input sample x = [ x 1 , . . . , x n ] with n variables indexed by N = { 1 , . . . , n } , v ( x ) ∈ R denotes a scalar output of the DNN 3 . Each interaction in [45, 27, 47] is defined as Harsanyi dividend (or Harsanyi interaction) [18] in game theory, which represents a collaboration (AND relationship) between input variables in a specific set S ( S ⊆ N ). For example, as Figure 1(a) shows, the co-appearance of image patches forms a mouth interaction S = { x 1 , x 2 , x 3 } . Only when these three patches are all present, the mouth interaction will be triggered, and make a certain interaction effect I ( S ) on the network output. The absence (masking) of any patches of x 1 , x 2 , and x 3 will deactivate the mouth interaction and remove the interaction effect, i.e., I ( S ) = 0 . The interaction effect I ( S | x ) on the input sample x is computed as follows.

where x T denotes the masked input sample, when we mask input variables in N \ T and keep variables in T unchanged. Please see more properties of the Harsanyi dividend in the supplementary material.

Theoretically, there are 2 n potential subsets S of input variables ( S ∈ 2 N = { S | S ⊆ N } ). The proven concept-emergence phenomenon refers to that only a small number of subsets S ∈ Ω salient ⊆ 2 N make salient interaction effects I ( S | x ) on the network output, and can be considered as interactive concepts. Interaction effects of all other subsets are close to zero ( I ( S | x ) ≈ 0 ), which can be considered as ignorable noisy patterns . Therefore, the network output v ( x ) can be well approximated by interaction effects of a small number of interactive concepts, i.e. ,

We also conduct experiments to illustrate the concept-emergence phenomenon on various DNNs, including multi-layer perceptrons (MLPs), long short-term memory (LSTM), AlexNet [23], ResNet [19], convolutional neural networks (CNNs), and PointNet [41] trained on different types of data, including tabular data (UCI dataset [6]), natural language data (CoLA [61] and SST-2 [56]), image data (MNIST [26] and CelebA [31]), and point cloud data (ShapeNet [65]). We follow experimental settings in [45, 27], and Figure 1(b) shows that interactive concepts encoded by a DNN are usually indeed sparse.

Theorem 1. Given an input sample x ∈ R n with n variables, there are 2 n different masked samples x T w.r.t. all potential subsets T ⊆ N . Then, [45] has proven that

Theorem 1 means that we can use a small number of interactive concepts in Ω salient to well approximate network outputs on anyone x T of the 2 n masked samples.

Trustworthiness of interactive concepts. The mathematical proof in [47] makes I ( S ) become the first concepts whose emergence in DNNs is certificated. Equations (2) and (3) further guarantee that interactive concepts can faithfully explain the output of DNNs. In addition, Li and Zhang [27] have discovered that (i) interactive concepts are transferable across different samples; (ii) interactive concepts are discriminative, i.e. , if a concept S has salient interaction effects I ( S ) on a set of samples, then the concept tends to push these samples to be classified towards the same category. In this way, we can roughly consider such interactive concepts a relatively rigorous definition of concepts.

Complexity (order) of interactive concepts. The complexity of an interactive concept S is defined as the number of input variables involved in S , which is also termed the order of the interactive concept, i.e. , complexity ( S ) = order ( S ) = | S | . Thus, low-order interactive concepts usually represent simple AND relationships among a few input variables. In comparison, high-order interactive concepts often refer to as relatively complex AND relationships among a large number of input variables.

3.2 Simple interactive concepts in data are more stable and easier to learn

Unlike other interaction metrics [16, 32, 57], sparse interactive concepts are certificated to emerge in DNN, and can faithfully represent the inference score of DNNs. Therefore, in this study, we aim to further explore the learning difficulty of these certificated concepts. Specifically, we theoretically explain that a DNN is more likely to encode simple ( i.e. , low-order) interactive concepts.

In this subsection, we show that low-order interactive concepts encoded by DNNs are generally more stable to inevitable noises in data, compared to high-order interactive concepts. Specifically, we derive an approximate analytical solution to the variance (instability) of interactive concepts' effects w.r.t. data noise, and show that the variance (instability) increases along the order of concepts in an exponential manner.

Theorem 2 (proven in the supplementary material) . Given a neural network v and an arbitrary input sample x ′ ∈ R n , the network output can be decomposed by using the Taylor expansion i.e., v ( x ′ ) = ∑ S ⊆ N ∑ π ∈ Q S U S, π · J ( S, π | x ′ ) . In this way, according to Equation (1), the Harsanyi interaction effect I ( S | x ′ ) on the sample x ′ can be reformulated as follows.

Here, J ( S, π | x ′ ) = ∏ i ∈ S ( sign( x ′ i -b i ) · x ′ i -b i τ ) π i denotes a Taylor expansion term of the degree π , where the degree π ∈ Q S = { [ π 1 , . . . , π n ] |∀ i ∈ S, π i ∈ N + ; ∀ i ̸∈ S, π i = 0 } and b i is the baseline value to mask the input variable x i . In addition, U S, π = τ m ∏ n i =1 π i ! ∂ m v ( x ′ ∅ ) ∂x π 1 1 ··· ∂x πn n · ∏ i ∈ S [sign( x ′ i -b i )] π i , where x ′ ∅ denotes the sample whose input variables are all masked. m = ∑ n i =1 π i .

Theorem 2 rewrites the Harsanyi interaction effect I ( S | x ) of each concept from a new perspective, to facilitate the analysis of the instability of interactive concepts. The derivation of Theorem 2 is inspired by [12, 70, 46], in which the Harsanyi interaction effect is re-written as the sum of the Taylor interaction effects. Therefore, in this study, we define a new baseline value b i to further extend the finding in [12] and obtain Theorem 2, which significantly simplifies the further proof. Specifically, the new baseline value b i is set as follows to represent the masked state of the i -th input variable. i.e. ,

x i ← b i . Previous studies usually set b i = µ i = E x [ x i ] as the average value over different samples to represent the masked state [4, 12]. However, we consider that pushing the input variable x i to move a large distance τ ∈ R towards µ i has been significant enough to remove its information. To this end, we set b i = x i -τ 4 , if x i > µ i ; else, b i = x i + τ 4 . The above new setting ensures comparable perturbation magnitudes over different dimensions.

Next, we analyze the variance (instability) of the interaction effect I ( S | x ′ = x + ϵ ) when we add a Gaussian perturbation ϵ ∼ N ( 0 , σ 2 I ) to the input sample x . Here, the Gaussian perturbation ϵ ∈ R n is considered as a rough representation of inevitable variations in data, e.g. , shape deformation and object rotation variations in image classification. However, such variations are quite difficult to formulate, so we use a Gaussian perturbation as a rough representation. Fortunately, we find that our reformulation of interactions in Theorem 2 enables us to directly apply the finding in [46] to prove the variance (instability) of the interaction effect I ( S | x ′ = x + ϵ ) .

Theorem 3. Let us add a Gaussian perturbation ϵ ∼ N ( 0 , σ 2 I ) to the input sample x . Let us first consider the case with the lowest degree ˆ π = [ˆ π 1 , . . . , ˆ π n ] ∈ Q S , satisfying that ∀ i ∈ S, ˆ π i = 1; ∀ i / ∈ S, ˆ π i = 0 . The mean and variance of J ( S, ˆ π | x + ϵ ) over the Gaussian perturbation ϵ are given as

Furthermore, for the more general case with an arbitrary degree π ∈ Q S = { [ π 1 , · · · , π n ] |∀ i ∈ S, π i ∈ N + ; ∀ i / ∈ S, π i = 0 } , the mean and variance of J ( S, π | x + ϵ ) are computed as

Theorem 3 shows the (variance) instability of interactive concepts of different complexities (orders) w.r.t. data variations. It indicates that the variance of J ( S, π | x + ϵ ) roughly increases along with the order ( S ) = | S | in an exponential manner. Similar conclusions are also introduced in [70]. Furthermore, according to Equation (4), the interaction effect I ( S | x + ϵ ) of the concept S can be represented as the weighted sum of J ( S, π | x + ϵ ) , and coefficients U S, π w.r.t. different orders s and degrees π are usually chaotic. Hence, we can roughly consider that the variance (instability) of the interaction effect I ( S | x + ϵ ) increases along with the order | S | exponentially, as well. In other words, compared to high-order concepts, low-order concepts are much more stable to slight input perturbations.

Experimental verification. Here, we conduct experiments to verify whether the variance (instability) of interaction effects indeed increased along with the order in an approximately exponential manner. Specifically, to mimic variations in the data, we add Gaussian perturbations ϵ ∼ N ( 0 , 0 . 02 2 I ) to each training sample. Then, we compute the average mean E ( s ) = E x ∈ X [ E S ⊆ N, | S | = s [ | E ϵ [ I ( S | x + ϵ )] | ]] and the average variance V ( s ) = E x ∈ X [ E S ⊆ N, | S | = s [Var ϵ [ I ( S | x + ϵ )]]] of interaction effects of the s -th order concepts w.r.t. Gaussian perturbations. For verification, we calculate such metrics on DNNs trained for image classification and DNNs trained on tabular data. We train AlexNet [23], VGG-11 [55], ResNet-18/20 [19] on the CIFAR-10 dataset [22] and the Tiny-ImageNet dataset [25], respectively. We also train a five-layer MLP [45] on the UCI census dataset (namely census dataset ) and the UCI TV news dataset (namely TV news dataset ) [6], respectively. In addition, considering the computational cost of I ( S | x + ϵ ) in Equation (1) is intolerable if each pixel is considered as an input variable, we divide images into 8 × 8 patches [45] to calculate I ( S | x + ϵ ) . Please see the supplementary material for more implementation details.

Figure 2 shows that the variance (instability) V ( s ) of interaction effects increases in an exponential manner along with the order s of the interactive concept. In addition, the relative stability E ( s ) / √ V ( s ) of interaction effects decreases significantly along with the order s . Such phenomenon successfully verifies findings in Theorem 3.

3.2.1 Formulating the conceptual learning as a linear regression problem

Note that, Li and Zhang [27] have discovered that interactive concepts have high transferability over samples in the same category, i.e. , most interactive concepts extracted from different samples may

AlexNet on CIFAR-10

VGG-11 on CIFAR-10

AlexNet on Tiny-ImageNet

VGG-11 on Tiny-ImageNet

Five-layer MLP on Census

Five-layer MLP on TV news

Order

ResNet-20 on CIFAR-10

ResNet-18 on Tiny-ImageNet

log[

𝑉𝑉

(

𝑠𝑠

)

]

Order

𝐸𝐸

𝑠𝑠

𝑉𝑉

(

𝑠𝑠

)

0

200

0

-2

-4

0

.1n

0

.5n

0

.9n

0

.1n

0

.5n

0

.9n

all belong to the same subset Ω c ⊆ Ω salient. Therefore, we can consider the interactive concepts as common knowledge learned by the DNN, and we aim to analyze the difficulty of learning each interactive concept with different complexity.

To facilitate the analysis, we first simplify the conceptual learning as a linear regression problem. Specifically, we first rewrite the interaction effect of an interactive concept S . Given an input sample x , according to Equation (4), the interaction effect of the concept S on the sample x ′ (obtained by applying some transformations on x ), I ( S | x ′ ) , can be rewritten as

where the constant U S = I ( S | x ) denotes the interaction effect on the sample x , and the function for the triggering state on the transformed sample x ′ is given as C S ( x ′ ) = ∑ π ∈ Q S U S, π J ( S, π | x ′ ) /U S .

Theorem 4. Given an arbitrarily masked sample x T ( T ⊆ N ) , the function C S ( x T ) defined above can be computed as the binary triggering state of the concept S in the sample x T .

∀ T ⊆ N, C S ( x T ) = ∏ i ∈ S A i ( x T ) = ✶ ( S ⊆ T ) , (8) where A i ( x T ) ∈ { 0 , 1 } denotes whether the variable x i is present A i ( x T ) = 1 or being masked A i ( x T ) = 0 in the sample x T .

The function C S ( x T ) represents the triggering state of the concept S under an arbitrarily masking condition ∀ T ⊆ N . Only when all variables in S are present under the masking condition T , the concept S is triggered C S ( x T ) = 1 . If any of variables in S is masked, then the concept S will not be triggered C S ( x T ) = 0 , yielding zero interaction effect I ( S | x T ) = 0 . Theorem 4 shows that given a masked sample x T , the concept S is only triggered when the sample x T contains all variables in S , i.e. , T ⊇ S .

Thus, inspired by [46], we can extend the conclusion to a continuous version that explains the output of the DNN as a linear regression of very few salient interactive concepts based on Equation (3).

where the triggering state C S ( x ′ ) of each interactive concept S can be considered as an input dimension of the linear function, which reflects whether the input sample x ′ contains the concept S .

Therefore, the absolute value of the weight coefficient U S in Equation (9) can be viewed as the strength of the DNN encoding the interactive concept S . Considering the sparsity of interactive concepts discussed in Section 3.1, most interactive concepts have ignorable coefficients U S ≈ 0 , and not so many concepts S have large absolute value of | U S | . Thus, we can consider that the DNN only learns a small number of salient interactive concepts.

3.2.2 Explaining the learning difficulty of concepts

Equation (9) in the previous subsection enables us to understand a DNN for the classification task as a pseudo-linear function. Then, if an input feature dimension has a stable value ( i.e., the triggering state C S of an interactive concept S stably being present/absent) across all samples in the same category, then we consider this feature dimension ( i.e., the concept) is easy to learn. In comparison, when we extract the same concept from different samples in the same category, if this concept exhibits inconsistent interaction effects ( e.g. , such a concept does not consistently present or absent over different samples), then this feature dimension ( i.e., the concept) is hard to learn.

AlexNet on CIFAR-10

VGG-11 on CIFAR-10

AlexNet on Tiny-ImageNet

VGG-11 on Tiny-ImageNet

Five-layer MLP on Census

Five-layer MLP on TV news

Epoch

Epoch

Epoch

ResNet-20 on CIFAR-10

Epoch

ResNet-18 on Tiny-ImageNet

𝔼𝔼

𝑆𝑆⊆𝑁𝑁

,

𝑆𝑆 =𝑠𝑠

[

𝛽𝛽

(

𝑆𝑆

)]

𝑠𝑠

= 0.

1𝑛𝑛

𝑠𝑠

= 0.

2𝑛𝑛

𝑠𝑠

= 0.

9𝑛𝑛

𝑠𝑠

= 0.

8𝑛𝑛

𝔼𝔼

𝑆𝑆⊆𝑁𝑁

,

𝑆𝑆 =𝑠𝑠

[

𝛽𝛽

(

𝑆𝑆

)]

AlexNet on CIFAR-10

VGG-11 on CIFAR-10

AlexNet on Tiny-ImageNet

VGG-11 on Tiny-ImageNet

Five-layer MLP on Census

Five-layer MLP on TV news

Epoch

Epoch

Epoch

ResNet-20 on CIFAR-10

ResNet-18 on Tiny-ImageNet

Epoch

𝔼𝔼

𝑆𝑆⊆𝑁𝑁

,

𝑆𝑆 =𝑠𝑠

[

𝜅𝜅

(

𝑆𝑆

)]

𝔼𝔼

𝑆𝑆⊆𝑁𝑁

,

𝑆𝑆 =𝑠𝑠

[

𝜅𝜅

(

𝑆𝑆

)]

𝑠𝑠

= 0.

1𝑛𝑛

𝑠𝑠

= 0.

2𝑛𝑛

𝑠𝑠

= 0.

9𝑛𝑛

𝑠𝑠

= 0.

8𝑛𝑛

To analyze the consistency/stability of the concept w.r.t data variations, we use small perturbations ϵ to represent various inevitable variations in the data, such as shape deformation and object rotation variations. Theorem 3 shows that high-order interactive concepts are much more unstable to inevitable variations in the data than low-order interactive concepts. This makes high-order concepts more likely to be influenced by data variations and less likely to be consistently present or absent in samples of the same category, which boosts the learning difficulty.

Experimental verification. We conduct experiments to verify the claim that high-order interactive concepts are less stably extracted under data variations than low-order interaction. We use the following two metrics to evaluate the each interactive concept S ( i.e., each single input dimension of the above linear function) of a certain order. Specifically, we use the first metric β ( S ) = E c [ | E x ∈ X c [ I ( S | x )] | / Std x ∈ X c [ I ( S | x )] ] to measure the relative consistency of the interactive concept appearing over different input samples in a certain category c . Here, X c denotes a set of training samples belonging to the category c , and Std x ∈ X c [ I ( S | x )] indicates the standard deviation of I ( S | x ) over different input samples. A large β ( S ) value means that the interactive concept S in all samples of the category c has similar/consistent effects I ( S | x ) . It is easier for a DNN to learn such a consistent concept S .

In addition, we use another metric κ ( S ) to verify whether the interaction effect of the highorder interactive concept is usually less stably extracted than those of the low-order interactive concept. To this end, the metric κ ( S ) = E x ∈ X [ E ϵ [ | I ( S | x + ϵ ) -I ( S | x ) | ]] / E x ∈ X [ | I ( S | x ) | ] = E x ∈ X [ E ϵ [ | C S ( x + ϵ ) -C S ( x ) | ]] / E x ∈ X [ | C S ( x ) | ] measures the relative instability of the interactive concept S to the inevitable slight variations ϵ in the data. Here, C S ( x + ϵ ) = I ( S | x + ϵ ) /U S computed in Equation (7) denotes the trigger state of the interactive concept S on the sample x ′ = x + ϵ .

To this end, we use DNNs and experimental settings in the experimental verification paragraph of Section 3.2 for evaluation. Figure 3 and Figure 4 show the change of the average consistency β ( S ) and the average instability κ ( S ) of s -order interactive concepts through the learning process, respectively. At each training epoch, low-order concepts usually obtain higher consistency β ( S ) and

AlexNet on CIFAR-10

VGG-11 on CIFAR-10

ResNet-20 on CIFAR-10

AlexNet on Tiny-ImageNet

VGG-11 on Tiny-ImageNet

ResNet-18 on Tiny-ImageNet

Five-layer MLP on Census

Five-layer MLP on TV news

Epoch

Epoch

Epoch

𝑠 = 0.1𝑛

𝑠 = 0.2𝑛

𝑠 = 0.3𝑛

𝑠 = 0.4𝑛

𝑠 = 0.5𝑛

𝑠 = 0.6𝑛

𝑠 = 0.7𝑛

𝑠 = 0.8𝑛

𝑠 = 0.9𝑛

Epoch

Epoch

Epoch

Epoch

Epoch

sim(𝐼

𝑡

𝑠

, 𝐼

final

𝑠

)

sim(𝐼

𝑡

𝑠

, 𝐼

final

𝑠

)

lower instability κ ( S ) than high-order concepts. It means that low-order concepts usually are more consistent and more stable. Thus, this experiment explains the reason why low-order interactive concepts are easier to learn.

3.2.3 Fast learning of low-order concepts

In this subsection, we illustrate the phenomenon that low-order interactive concepts are usually learned faster than high-order concepts, as a support for the claim that low-order interactive concepts are easier to be learned. We have theoretically proven in Section 3.2 and experimentally verified that low-order interactive concepts are more consistently present or consistently absent in different samples of the same category, which makes low-order interactive concepts easier to be learned. Then, this experiment is conducted to check whether low-order concepts are really learned faster than high-order concepts.

Specifically, we examine whether interactive concepts encoded by the finally-learned DNN v final ( x ) have already been encoded by the DNN that is v t ( x ) trained after t epochs. If so, we consider such interactive concepts are learned fast. Specifically, let I ( s ) t = [ I t ( S 1 | x ) , · · · , I t ( S d | x )] ⊤ ∈ R d denote a vector of interaction effects for all d = ( n s ) interactive concepts of s -order. Then, we compute the Jaccard similarity between s -order interactive concepts encoded by the DNN v final ( x ) and those encoded by the DNN v t ( x ) , i.e. , sim ( I ( s ) t , I ( s ) final ) = ∥ min( ˜ I ( s ) t , ˜ I ( s ) final ) ∥ 1 / ∥ max( ˜ I ( s ) t , ˜ I ( s ) final ) ∥ 1 and ∥ · ∥ 1 denotes L1-norm. We extend the d -dimensional vector I ( s ) t into a 2 d -dimensional vector with non-negative elements ˜ I ( s ) t = [( I ( s ) , + t ) ⊤ , ( I ( s ) , -t ) ⊤ ] ⊤ = [max( I ( s ) t , 0) ⊤ , -min( I ( s ) t , 0) ⊤ ] ⊤ ∈ R 2 d . Similarly, we compute ˜ I ( s ) final with non-negative elements based on I ( s ) final . In this way, a large similarity sim ( I ( s ) t , I ( s ) final ) at an earlier epoch t indicates that interactive concepts are easier to be learned.

To this end, let us use DNNs and experimental settings introduced in the experimental verification paragraph of Section 3.2 for evaluation. Figure 5 shows that DNNs first learn low-order interactive concepts, and then learn high-order interactive concepts. Such a phenomenon verifies the conclusion that a DNN easily learns low-order (simple) interactive concepts.

4 Explaining findings in previous studies

4.1 Explaining adversarial robustness

Previous study [43] has discovered that low-order interactions are more robust to adversarial attacks than high-order interactions. Notice that their high-order (low-order) interactions are different from, but highly related to our high-order (low-order) interactive concepts. Specifically, our interactive concepts can be explained as elementary components for such multi-order interactions in [43]. Please see the supplementary material for the proof. Therefore, from this perspective, our conclusion that 'low-order interactive concepts are easy to learn' can also explain how a DNN encodes concepts of different adversarial robustness.

AlexNet on CIFAR-10

VGG-11 on CIFAR-10

AlexNet on Tiny-ImageNet

VGG-11 on Tiny-ImageNet

Five-layer MLP on Census

Five-layer MLP on TV news

ResNet-20 on CIFAR-10

ResNet-18 on Tiny-ImageNet

Order

𝐴𝐴

(

s

)

𝐴𝐴

(

𝑠𝑠

)

Order

Order

Order

Can we use interactive concepts in this paper to verify the heuristic findings of adversarial robustness in [43]? According to Equation (2), we can represent the inference logic of a DNN by a set of interactive concepts, i.e., v ( x ) ≈ ∑ S ∈ Ω salient I ( S | x ) . In this way, we conduct experiments to evaluate the sensitivity of each interactive concept S to adversarial perturbations. The symbolic conceptual representation of interactive concepts allows us to measure the adversarial robustness in a more direct way. That is, if the effect of an interactive concept does not significantly change under adversarial attacks, we consider this interactive concept is robust to adversarial attacks. In contrast, if the effect of an interactive concept changes significantly under adversarial attacks, we consider this interactive concept is sensitive to adversarial attacks.

To this end, we use the metric α ( S ) = E x ∈ X [ | I ( S | x + δ ) -I ( S | x ) | ] / E x ∈ X [ | I ( S | x ) | ] to evaluate the sensitivity of the interactive concept S to adversarial perturbations, where δ denotes the adversarial perturbation generated by the ℓ ∞ attack [33]. In this way, a small α ( S ) value indicates that the interactive concept S is robust to the adversarial attack.

We follow experimental settings in the experimental verification paragraph of Section 3.2 to evaluate different DNNs. Figure 6 shows that compared to high-order interactive concepts, low-order concepts usually obtain smaller A ( s ) = E S ⊆ N, | S | = s [ α ( S )] values. Such phenomena demonstrate that low-order interactive concepts are more robust to adversarial attacks.

4.2 Connections to existing findings on what are learned first by a DNN

In this subsection, we discuss some related studies on which kind of knowledge is usually first learned by a DNN. Most previous studies conducted experiments to explore the knowledge that was easier to be learned by a DNN, without providing much theoretical support. However, we find that our theorems can partially explain mechanisms behind some previous findings.

Arpit et al. [5] trained DNNs to classify both normal samples and white-noise samples to different object categories. In this way, they considered that the DNN encoded simple concepts to classify normal samples, but the DNN had to learn complex concepts to classify white noises to randomlyassigned labels. They observed that the DNN usually learned normal samples first, because the classification accuracy of normal samples increased before that of white noises. To this end, our research provides more insights into such an observation. Specifically, Cheng et al. [8] have proven that the classification of noisy data usually depends on high-order concepts, i.e. , the classification of noisy data forces the DNN to memorize each specific white-noise sample as a specific high-order interactive concept. Let us combine this conclusion with our finding that high-order interactive concepts are hard to learn. Then, we can easily owe the slow learning of white-noise samples observed in [5] to the difficulty of learning high-order concepts.

Mangalam and Prabhu [34] considered/defined easy samples as training samples that could be correctly classified by shallow machine learning models, such as support vector machine (SVM) and random forests (RF). They discovered that DNNs first learned easy samples, and then gradually learned more about hard samples. To this end, our research verifies such observation. Specifically, we claim that easy samples mainly contain low-order interactive concepts. Thus, hard samples mainly contain high-order interactive concepts corresponding to complex interactions between numerous

input variables, and thus are difficult to be classified by shallow models ( e.g. , the SVM and the RF). In this way, the fast learning of low-order concepts is another understanding of the finding in [34].

· Xu et al. [63] discovered that during the training process, DNNs usually first learned samples of low frequencies ( e.g., robust to noises), and then encoded samples of high frequencies ( e.g., sensitive to noises). However, the original design of DNNs is not towards learning specific spectrums, and techniques of deep learning are not developed by assuming a periodic loss landscape of training samples. Therefore, we believe there should be a more direct explanation for the spectrum-learning phenomenon discovered by [63]. To this end, our research explains this phenomenon as the difficulty of learning high-order concepts. According to Section 4.1, low-order concepts usually are less sensitive to input perturbations, thereby corresponding to low-frequency components defined in the loss landscape in [63]. Accordingly, high-order concepts correspond to high-frequency components.

· Liu et al. [28] discovered that during adversarial training, the training loss of the DNN trained on easy samples decreased faster than that of the DNN trained on hard samples. To this end, we consider easy samples in adversarial training mentioned by [28] may mainly contain low-order interactive concepts. It is because, as discussed in Section 4.1, [43] discovered that low-order interactions were robust to adversarial perturbations. Thus, the fast learning of easy samples in adversarial training can be roughly owing to the learning of low-order concepts.

5 Conclusion and discussion

In this paper, we theoretically explain the trend of DNNs learning simple concepts. Since the emergence of interactive concepts of DNNs has been observed [45, 27] and proved [47], we aim to theoretically explain the explicit theoretical connection between the conceptual complexity and the difficulty of learning concepts. In this way, we prove that low-order interactive concepts in the data are much more stable than high-order interactive concepts, which makes low-order interactive concepts more likely to be encoded. Besides, our research can also provide new insights into several previous empirical understandings [5, 34, 63, 28] w.r.t. the conceptual representation of DNNs.

There are very few ways to define and examine what is a 'concept.' In this paper, we only use the following three properties to support the faithfulness of using sparse salient interactions as concepts encoded by the DNN.

· Ren et al. [47] have proved that a well-trained DNN can encode just a small number of salient interactions for inference under some common conditions.

· Ren et al. [45] have proved that, such a small number of salient interactions extracted from a sample can well mimic DNN's outputs on numerous masked samples.

· Li and Zhang [27] have discovered that salient interactions have considerable transferability and strong discrimination power.

However, the above properties cannot guarantee a clear correspondence between an interactive concept and a concept in human cognition. Up to now, we cannot mathematically formulate what is a concept in cognitive science. Thus, there is still a long way to unify the learning difficulty of a DNN and the cognitive difficulty of human beings.

Acknowledgments

This work is partially supported by the National Nature Science Foundation of China (62276165), National Key R&D Program of China (2021ZD0111602), Shanghai Natural Science Foundation (21JC1403800,21ZR1434600), National Nature Science Foundation of China (U19B2043). This paper is also partially supported by Huawei Technologies Inc.

References

Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence , 40(12):2897-2905, 2018.

Sravanti Addepalli, Anshul Nasery, Venkatesh Babu Radhakrishnan, Praneeth Netrapalli, and Prateek Jain. Feature reconstruction from outputs can mitigate simplicity bias in neural networks. 2022.

Rana Ali Amjad and Bernhard C Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence , 42(9):2225-2239, 2019.

Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In International Conference on Machine Learning , pages 272-281. PMLR, 2019.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Arthur Asuncion and David Newman. Uci machine learning repository, 2007.

Chuanchuan Chen, Dongrui Liu, Changqing Xu, and Trieu-Kien Truong. Saks: Sampling adaptive kernels from subspace for point cloud graph convolution. IEEE Transactions on Circuits and Systems for Video Technology , 2023.

Xu Cheng, Chuntung Chu, Yi Zheng, Jie Ren, and Quanshi Zhang. A game-theoretic taxonomy of visual concepts in dnns. arXiv preprint arXiv:2106.10938 , 2021.

Xu Cheng, Xin Wang, Haotian Xue, Zhengyang Liang, and Quanshi Zhang. A hypothesis for the aesthetic appreciation in neural networks. arXiv preprint arXiv::2108.02646 , 2021.

Huiqi Deng, Na Zou, Mengnan Du, Weifu Chen, Guocan Feng, and Xia Hu. A general taylor framework for unifying and revisiting attribution methods. arXiv preprint arXiv:2105.13841 , 2021.

Huiqi Deng, Qihan Ren, Xu Chen, Hao Zhang, Jie Ren, and Quanshi Zhang. Discovering and explaining the representation bottleneck of dnns. International Conference on Learning Representations , 2022.

Huiqi Deng, Na Zou, Mengnan Du, Weifu Chen, Guocan Feng, Ziwei Yang, Zheyang Li, and Quanshi Zhang. Understanding and unifying fourteen attribution methods with taylor interactions. arXiv preprint arXiv:2303.01506 , 2023.

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning , pages 1019-1028. PMLR, 2017.

Stanislav Fort, Paweł Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: A new perspective on generalization in neural networks. arXiv preprint arXiv:1901.09491 , 2019.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory , 28(4):547-565, 1999.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

John C Harsanyi. A simplified bargaining model for the n-person cooperative game. International Economic Review , 4(2):194-220, 1963.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research , 2022.

S Kornblith, M Norouzi, H Lee, and G Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning , pages 3519-3529. PMLR, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Stephen Laurence and Eric Margolis. Concepts and cognitive science. 1999.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Mingjie Li and Quanshi Zhang. Does a neural network really encode symbolic concept? In International conference on machine learning . PMLR, 2023.

Chen Liu, Zhichao Huang, Mathieu Salzmann, Tong Zhang, and Sabine Süsstrunk. On the impact of hard adversarial instances on overfitting in adversarial training. arXiv preprint arXiv:2112.07324 , 2021.

Dongrui Liu, Shaobo Wang, Jie Ren, Kangrui Wang, Sheng Yin, Huiqi Deng, and Quanshi Zhang. Trap of feature diversity in the learning of mlps. arXiv preprint arXiv:2112.00980 , 2021.

Dongrui Liu, Chuanchaun Chen, Changqing Xu, Robert C Qiu, and Lei Chu. Self-supervised point cloud registration with deep versatile descriptors for intelligent driving. IEEE Transactions on Intelligent Transportation Systems , 2023.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision , pages 3730-3738, 2015.

Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888 , 2018.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? 2019.

Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems , 27, 2014.

Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems , 31, 2018.

Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08760 , 2018.

Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv preprint arXiv:1312.6098 , 2013.

Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. Advances in Neural Information Processing Systems , 34: 18420-18432, 2021.

Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems , 34:1256-1272, 2021.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems , 30, 2017.

Jie Ren, Die Zhang, Yisen Wang, Lu Chen, Zhanpeng Zhou, Yiting Chen, Xu Cheng, Xin Wang, Meng Zhou, Jie Shi, et al. A unified game-theoretic interpretation of adversarial robustness. 35th Conference on Neural Information Processing Systems (NeurIPS 2021) , 2021.

Jie Ren, Zhanpeng Zhou, Qirui Chen, and Quanshi Zhang. Can we faithfully represent absence states to compute shapley values on a dnn? In The Eleventh International Conference on Learning Representations , 2022.

Jie Ren, Mingjie Li, Qirui Chen, Huiqi Deng, and Quanshi Zhang. Defining and quantifying the emergence of sparse concepts in dnns. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Qihan Ren, Huiqi Deng, Yunuo Chen, Siyu Lou, and Quanshi Zhang. Bayesian neural networks avoid encoding complex and perturbation-sensitive concepts. Proceedings of the 40th International Conference on Machine Learning , pages 28889-28913, 2023.

Qihan Ren, Jiayang Gao, Wen Shen, and Quanshi Zhang. Where we have arrived in proving the emergence of sparse symbolic concepts in ai models. arXiv preprint arXiv:2305.01939 , 2023.

Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems , 34:4974-4986, 2021.

Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shortcut cues will dnns choose? a study from the parameter-space perspective. arXiv preprint arXiv:2110.03095 , 2021.

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems , 33:9573-9585, 2020.

Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games , 2(28): 307-317, 1953.

Wen Shen, Qihan Ren, Dongrui Liu, and Quanshi Zhang. Interpreting representation quality of dnns for 3d point cloud processing. Advances in Neural Information Processing Systems , 34: 8857-8870, 2021.

Thomas J Shuell. Cognitive conceptions of learning. Review of educational research , 56(4): 411-436, 1986.

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 , 2017.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. In International Conference on Machine Learning , pages 9259-9268. PMLR, 2020.

Che-Ping Tsai, Chih-Kuan Yeh, and Pradeep Ravikumar. Faith-shap: The faithful shapley interaction index. arXiv preprint arXiv:2203.00870 , 2022.

Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transferability. arXiv preprint arXiv:2010.04055 , 2020.

Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transferability. In International Conference on Learning Representations , 2021.

Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019.

Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. arXiv preprint arXiv:1801.10578 , 2018.

Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523 , 2019.

Zhiqin John Xu. Understanding training and generalization in deep learning by fourier analysis. arXiv preprint arXiv:1808.04295 , 2018.

Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG) , 35(6):1-12, 2016.

Die Zhang, Hao Zhang, Huilin Zhou, Xiaoyi Bao, Da Huo, Ruizhao Chen, Xu Cheng, Mengyue Wu, and Quanshi Zhang. Building interpretable interaction trees for deep NLP models. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021 , pages 14328-14337. AAAI Press, 2021.

Hao Zhang, Sen Li, YinChao Ma, Mingjie Li, Yichen Xie, and Quanshi Zhang. Interpreting and boosting dropout from a game-theoretic view. In International Conference on Learning Representations , 2020.

Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021 , pages 10877-10886. AAAI Press, 2021.

Quanshi Zhang, Xin Wang, Jie Ren, Xu Cheng, Shuyun Lin, Yisen Wang, and Xiangming Zhu. Proving common mechanisms shared by twelve methods of boosting adversarial transferability. arXiv preprint arXiv:2207.11694 , 2022.

Huilin Zhou, Hao Zhang, Huiqi Deng, Dongrui Liu, Wen Shen, Shih-Han Chan, and Quanshi Zhang. Concept-level explanation for the generalization of a dnn. arXiv preprint arXiv:2302.13091 , 2023.

A Literature on the representation capacity of DNNs

Formulating and evaluating the representation ability of DNNs is an emerging perspective to explain DNNs. Pascanu et al. [38] and Montufar et al. [35] evaluated the representation capacity of DNNs based on the number of linear response regions. Kornblith et al. [21], Raghu et al. [42], and Morcos et al. [36] analyzed representations similarity between DNNs by using canonical correlation analysis. Shen et al. [52], Chen et al. [7], and Liu et al. [30] measured the quality of knowledge representations encoded in DNNs for point cloud processing. The information-bottleneck theory [54] quantified information encoded in DNNs, and was extended to improve the representation capacity of DNNs [1, 3]. Xu [64] and Xu et al. [63] explained the generalization of DNNs from the perspective of Fourier analysis. Furthermore, several metrics were proposed to evaluate the robustness or generalization capacity of DNNs, including the flatness of loss functions at minima [13], the stability of optimization [17], the CLEVER score [62], the stiffness [14], and the sensitivity metrics [37].

In contrast to previous empirical studies, we mathematically formulate concepts encoded by DNNs and theoretically prove that DNNs mainly learn simple concepts.

B Literature on interactions

Many studies investigated interactions between input variables of DNNs in recent years. Grabisch and Roubens [16] proposed the Shapley interaction index to measure the interaction between players in a cooperative game, based on the Shapley value [51]. Lundberg et al. [32] used the Shapley interaction index to analyze tree ensembles. Sundararajan et al. [57] proposed the Shapley-Taylor interaction index, and Tsai et al. [58] defined Faith-Shap, which was another interaction index.

In this paper, we use interactions between input variables of a DNN to represent concepts encoded by the DNN. In this way, We theoretically explain and empirically verify that DNN is easier to learn simple interactive concepts.

C Literature on the shortcut learning and simplicity bias of DNNs

Many studies focused on the underlying principles and limitations of DNNs. Geirhos et al. [15], Robinson et al. [48], and Scimeca et al. [49] investigated the shortcut learning in DNNs. I.e. , DNNs may find decision rules that perform well on standard datasets, but fail to generalize to real-world scenarios. For example, DNNs successfully detected pneumonia by identifying hospital-specific metal tokens on the scan, but they failed to learn much about pneumonia and achieved low performance for scans from novel hospitals. Meanwhile, many studies proposed simplicity bias , i.e., standard training procedures for DNNs have a bias towards learning simple models [50, 20, 40]. Specifically, DNNs tend to learn low-rank embeddings [20] or simple features [50, 29, 40, 2].

From the perspective of the simplicity bias, our study considers the definition of interactive concepts in [47], and analyzes the bias towards learning simple concepts. This work clarifies an exact form of the complexity of concepts that a DNN is difficult to learn.

D Axioms and theorems for the Harsanyi dividend interaction

In this section, we introduce that the Harsanyi dividend interaction I ( S ) satisfies several desirable axioms and theorems.

The Harsanyi dividend interactions I ( S ) satisfies the efficiency, linearity, dummy, symmetry, anonymity, recursive and interaction distribution axioms, as follows.

Efficiency axiom (proved by [18]). The reward of a neural network can be decomposed into interaction effects of different contexts, i.e. v ( N ) = ∑ S ⊆ N I ( S ) .

Linearity axiom . If we merge rewards of two neural networks w and v as the reward of model u , i.e. ∀ S ⊆ N, u ( S ) = w ( S ) + v ( S ) , then their interaction effects I v ( S ) and I w ( S ) can be represented as ∀ S ⊆ N,I u ( S ) = I v ( S ) + I w ( S ) .

Dummy axiom . If a variable i ∈ N is a dummy variable, i.e. ∀ S ⊆ N \{ i } , v ( S ∪ { i } ) = v ( S ) + v ( { i } ) , then it does not interact with other variables, ∀∅ ̸ = S ⊂ N \ { i } , I ( S ∪ { i } ) = 0 .

frequency

0.4

0.3

0.1

0.0

⁄

𝑈𝑈

′

𝑈𝑈

′

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.6

0.4

0.2

0.0

⁄

𝑈𝑈

′

𝑈𝑈

′

0.0

0.2

0.4

0.6

0.8

1.0

frequency

census

TV news

𝒮𝒮

𝒮𝒮

𝒮𝒮

𝒮𝒮

𝒮𝒮

𝒮𝒮

Symmetry axiom . Given two input variables i, j ∈ N , if ∀ S ⊆ N \{ i, j } , v ( S ∪ { i } ) = v ( S ∪ { j } ) , then ∀ S ⊆ N \{ i, j } , I ( S ∪ { i } ) = I ( S ∪ { j } ) .

Anonymity axiom . For any permutations π on N , we have ∀ S ⊆ N,I v ( S ) = I πv ( πS ) , where πS { π ( i ) | i ∈ S } , and the new model πv is defined by ( πv )( πS )= v ( S ) . This indicates that interaction effects are not changed by permutation.

Recursive axiom . The interaction effects can be calculated recursively. Given i ∈ N and S ⊆ N \{ i } , the interaction effect of the pattern S ∪{ i } can be represented as the interaction effect of S with i minus the interaction effect of S without i , i.e. ∀ S ⊆ N \{ i } , I ( S ∪{ i } )= I ( S | i is always present ) -I ( S ) . I ( S | i is always present ) denotes the interaction effect when the variable i is always present as a constant context, i.e. I ( S | i is always present ) = ∑ L ⊆ S ( -1) | S |-| L | · v ( L ∪ { i } ) .

̸

Interaction distribution axiom . This axiom shows how interactions are distributed for 'interaction functions' [57]. The definition of the interaction function v T parameterized by a subset of variables T is given as follows. ∀ S ⊆ N , if T ⊆ S , v T ( S ) = c ; otherwise, v T ( S ) = 0 . The function v T models pure interaction among the variables in T , because only if all variables in T are present, the output value will be increased by c . The interactions encoded in the function v T satisfies I ( T ) = c , and ∀ S = T , I ( S ) = 0 .

E Sparsity of interactive concepts

In this section, we conducted experiments to verify the sparsity of interactive concepts. To this end, we computed effects U S of all 2 n interactive concepts encoded by a DNN following [45]. Specifically, we trained a five-layer MLP on the census dataset and the TV news dataset), respectively. For better visualization, we re-scaled effects of interactive concepts U S by | U S | / max S ′ ⊆ N | U S ′ | . Moreover, the strength of effects are average over different samples in each dataset. Fig. 7 shows histograms of absolute effects of interactive concepts.

F Complexity of interactive concepts

Many previous studies [11, 67, 59, 45, 43] used multi-order interactions to analyze DNNs. Specifically, Given a pre-trained DNN v and a masked sample x S , the multi-order interaction I ( m ) ( i, j ) used in [59, 11, 67] is given as follows:

where ∆ v ( i, j, S ) = v ( x S ∪{ i,j } ) -v ( x S ∪{ i } ) -v ( x S ∪{ j } ) + v ( x S ) . They consider that I ( m ) ( i, j ) reflects the collaboration with m contextual variables ( m pixels). In this way, the order m measures the number of variables in S . Therefore, a low-order interaction denotes a relatively simple collaboration between input variables with a small context S . In contrast, a high-order interaction represents a complex collaboration between input variables with a large context S .

In this paper, we consider that the number of variables in an interactive concept S can measure the complexity of an interactive concept encoded by a DNN, namely the order of the interactive concept , complexity ( S )= order ( S )= | S | . Thus, low-order concepts usually represent simple AND relationships among a few input variables. In comparison, high-order concepts often refer to as relatively complex AND relationships among a large number of input variables.

G Proof of Theorems

G.1 Proof of Theorem 2 in the main paper

Theorem 2. Given a neural network v and an arbitrary input sample x ′ ∈ R n , the network output can be decomposed by using the Taylor expansion [10], i.e., v ( x ′ ) = ∑ S ⊆ N ∑ π ∈ Q S U S, π · J ( S, π | x ′ ) . In this way, according to Eq. (1), the interaction effect I ( S | x ′ ) on the sample x ′ can be reformulated as follows.

Here, J ( S, π | x ′ ) = ∏ i ∈ S ( sign( x ′ i -b i ) · x ′ i -b i τ ) π i denotes a Taylor expansion term of the degree π , where the degree π ∈ Q S = { [ π 1 , . . . , π n ] |∀ i ∈ S, π i ∈ N + ; ∀ i ̸∈ S, π i = 0 } and b i is the baseline value to mask the input variable x i . In addition, U S, π = τ m ∏ n i =1 π i ! ∂ m v ( x ′ ∅ ) ∂x π 1 1 ··· ∂x πn n · ∏ i ∈ S [sign( x ′ i -b i )] π i , where x ′ ∅ denotes the sample whose input variables are all masked. m = ∑ n i =1 π i .

Proof. Let us denote the function on the right of Eq. (2) by ˜ I ( S | x ′ ) , i.e. ,

We need to prove that for any arbitrary input sample ∀ x ′ ∈ R n , ˜ I ( S | x ′ ) = I ( S | x ′ ) .

Actually, it has been proven in [16] and [45] that the Harsanyi dividend I ( S | x ′ ) is the unique metric satisfying the faithfulness requirement mentioned in the main paper, i.e. , satisfying

Thus, as long as we can prove that ˜ I ( S | x ′ ) also satisfies the above faithfulness requirement, we can obtain ˜ I ( S | x ′ ) = I ( S | x ′ ) .

To this end, we only need to prove ˜ I ( S | x ′ ) also satisfies the faithfulness requirement in Eq. (4). Specifically, given an input sample ∀ x ′ ∈ R n , let us consider the Taylor expansion of the network output v ( x T ) of an arbitrarily masked sample x ′ T ( T ⊆ N ) , which is expanded at x ′ ∅ = [ b 1 , . . . , b n ] . Then, we have

where π ∈ { [ π 1 , . . . , π n ] |∀ i ∈ N,π i ∈ N } denotes the degree vector of Taylor expansion terms, and m = ∑ n i =1 π i . In addition, b i denotes the reference value to mask the input variable x i .

According to the definition of the masked sample x ′ T , we have that all variables in T keep unchanged and other variables are masked to the reference value. That is, ∀ i ∈ T , ( x ′ T ) i = x i ; ∀ i ̸∈ T , ( x ′ T ) i = b i . Hence, we obtain ∀ i ̸∈ T, [( x ′ T ) i -b i ] π i = 0 . Then, among all Taylor expansion terms, only terms corresponding to degrees π in the set P = { [ π 1 , . . . , π n ] |∀ i ∈ T, π i ∈ N ; ∀ i ̸∈ T, π i = 0 } may not be zero. Therefore, Eq. (5) can be rewritten as

We find that the set P can be divided into multiple disjoint sets as follows, P = ∪ S ⊆ T Q S , where Q S = { [ π 1 , . . . , π n ] |∀ i ∈ S, π i ∈ N + ; ∀ i ̸∈ S, π i = 0 } . Then, we can derive that

where τ ∈ R is a pre-defined constant and δ i = sign( x i -r i ) ∈ {-1 , 1 } is a sign function satisfying δ 2 i m = 1 ( m = ∑ n i =1 π i ). Then, Eq. (7) can be re-written as

Thus, ˜ I ( S | x ′ ) satisfies the faithfulness requirement in Eq. (4) when Ω = 2 N .

Therefore, Theorem 2 holds.

G.2 Proof of Theorem 3 in the main paper

Theorem 3. Let us add a Gaussian perturbation ϵ ∼ N ( 0 , σ 2 I ) to the input sample x . Let us first consider the case with the lowest degree ˆ π = [ˆ π 1 , . . . , ˆ π n ] ∈ Q S , satisfying that ∀ i ∈ S, ˆ π i = 1; ∀ i / ∈ S, ˆ π i = 0 . The mean and variance of J ( S, ˆ π | x + ϵ ) over the Gaussian perturbation ϵ are given as

Furthermore, for the more general case with an arbitrary degree π ∈ Q S = { [ π 1 , · · · , π n ] |∀ i ∈ S, π i ∈ N + ; ∀ i / ∈ S, π i = 0 } , the mean and variance of J ( S, π | x + ϵ ) are computed as

Proof. If we only consider Taylor expansion term of the lowest degree, then I ( S | x ′ ) ≈ U S, ˆ π · J ( S, ˆ π | x ′ ) , where J ( S, ˆ π | x ′ ) = ∏ i ∈ S sign( x ′ i -b i ) · x ′ i -b i τ .

Let us add a Gaussian perturbation ϵ ∼ N ( 0 , σ 2 I ) to the input sample x . Then, we have

According to the setting of the baseline value, we have ∀ i ∈ S, x i -b i ∈ {-τ, τ } . In Section 2.2, we have assumed that the perturbation is small, i.e. , ∀ i ∈ S, | ϵ i | ≤ τ . In this way, we have sign( x i + ϵ i -b i ) = sign( x i -b i ) , and we can obtain

Since sign( x i -b i ) ∈ {-1 , 1 } , we have 1 + sign( x i -b i ) · ϵ i τ ∼ N (1 , ( σ/τ ) 2 ) , ∀ i ∈ S .

Proposition 1. If random variables X 1 , X 2 , · · · , X k are independent of each other, then E [ X 1 X 2 · · · X k ] = ∏ k i =1 E [ X i ] , and Var[ X 1 X 2 · · · X k ] = ∏ k i =1 ( E [ X i ] 2 + Var[ X i ] 2 ) -∏ k i =1 E [ X i ] 2 .

According to the above proposition, we have

Therefore,

According to Theorem 2, given an arbitrary input sample x ′ , we have

Let us add a Gaussian perturbation ϵ ∼ N ( 0 , σ 2 I ) to the input sample x . In this way, we have

According to the setting of the baseline value, ∀ i ∈ S, x i -b i ∈ {-τ, τ } . We assume that the perturbation is small, i.e. , ∀ i ∈ S, | ϵ i |≪ τ . In this way, sign( x i + ϵ i -b i ) = sign( x i -b i ) , and we can obtain

Since ∀ i ∈ S , ϵ i is independent of each other, according to Proposition 1 and Eq. (17), we have

Since sign( x i -b i ) ∈ {-1 , 1 } , we have E ϵ i [ ( 1 + sign( x i -b i ) · ϵ i τ ) k ] = E ϵ i [ ( 1 + ϵ i τ ) k ] , ∀ k ∈ N + . Therefore, we obtain

G.3 Proof of Theorem 4 in the main paper

Theorem 4. Given an arbitrarily masked sample x T ( ∀ T ⊆ N ) , the function C S ( x T ) defined above can well fit the binary activation state of the concept S in the sample x T .

∀ T ⊆ N, C ( S | x T ) = ∏ i ∈ S A i ( x T ) = ✶ ( S ⊆ T ) , (19) where A i ( x T ) ∈ { 0 , 1 } denotes whether the variable x i is present A i ( x T ) = 1 or being masked A i ( x T ) = 0 in the sample x T .

Proof. We know that the function of the activation state on an arbitrary sample x ′ is given by

Specifically, now we consider a masked sample x T , and we will prove that C S ( x T ) = ✶ ( S ⊆ T ) . We consider the following two cases.

Case 1: S ⊊ T . Then there exists some j ∈ S \ T . Since j / ∈ T , according to the masking rule of the sample x T , we have ( x T ) j -b j = 0 . Since j ∈ S , we have

In this way, we have C S ( x T ) = ∑ π ∈ Q S U S, π J ( S, π | x T ) /U S = 0 .

Case 2: S ⊆ T . In this case, ∀ i ∈ S , we have i ∈ T . According to the setting of the reference value in Section 2.3 of the main paper, we have ( x T ) i -b i ∈ { τ, -τ } . Then we have sign(( x T ) i -b i ) · ( x T ) i -b i τ = 1 , ∀ i ∈ S . This further implies that

Therefore, we can derive

Recall that U S = I ( S | x ) = ∑ π ∈ Q S U S, π J ( S, π | x ) , in which J ( S, π | x ) = ∏ i ∈ S ( sign( x i -r i ) · x i -b i τ ) π i = 1 , because x i -b i ∈ { τ, -τ } . So actually, we have

Therefore, C S ( x T ) = ∑ π ∈ Q S U S, π /U S = 1 .

Combining the two cases, we can conclude that C S ( x T ) = ✶ ( S ⊆ T ) = ∏ i ∈ S A i ( x T ) .

G.4 Relation between interactive concepts and multi-order interactions

In this section, we derive that high-order interactive concepts (computed via the Harsanyi dividend [18]) can be considered as elementary components for high-order interactions used in [43].

Given a pre-trained DNN v and a masked sample x S , the multi-order interaction I ( m ) ( i, j ) used in [43] is given as follows:

Let ∆ v T ( S ) = ∑ L ⊆ T ( -1) | T |-| L | v ( x L ∪ S ) denote the marginal benefit of variables in T ⊆ N \ S , given the environment S . In this way, ∆ v T ( S ) can be represented as the sum of interaction effects inside T and sub-environments of S , i.e. ∆ v T ( S ) = ∑ S ′ ⊆ S I ( T ∪ S ′ ) [45].

Thus, the I ( m ) ( i, j ) can be represented as follows,

G.5 Theoretical connection between variance and learning difficulty

There is a close connection between the variance of interactive concepts and the learning difficulty. In fact, a similar connection has been discovered by [46]. The theoretical connection is proved as follows. Specifically, we simplify the learning of a DNN into a linear regression problem. In this very simple setting, we assume each i -th concept encoded by the DNN to be a specific feature, and use f i to represent the triggering (presence) state of the i -th concept in a training sample. Because according to Eq. (9) in the main paper, the network output v ( · ) is the sum of a small number of salient interactive concepts, we roughly simplify and rewrite the inference output of the DNN as the following linear function

where f = [ f 1 , · · · , f d ] ⊤ . Then, w i can be viewed as the strength of the DNN encoding the i -th interactive concept. In this way, if w i ≈ 0 , the DNN does not learn the i -th interactive concept.

We suppose that training samples are sampled from f ∼ P ( f ) = N ( µ, Σ 2 ) and y ∗ denotes the ground truth label. Specifically, µ = [ µ 1 , µ 2 , ..., µ d ] ⊤ and Σ = diag( σ 2 1 , σ 2 2 , ..., σ 2 d ) .

The toy regression problem can be formulated as follows,

We derive the optimal weights to the above regression task in three steps.

Step 1. The optimal weight w i for the regression problem in Eq. (26) is computed by

where K = µµ ⊤ +Σ 2 , ζ = y ∗ µ , and K j denotes the j -th column of the matrix K .

Step 2 . We further prove that for the optimal w , we have

Step 3 . Based on Step 2, we can derive | w i | ∝ | 1 /σ 2 i | , where w i denotes the strength of the DNN encoding the i -th interactive concept. | w i | ∝ | 1 /σ 2 i | indicates that the learning effect of the i -th interactive concept is inversely proportional to the variance of the i -th interactive concept. Therefore, interactive concepts with high variances (high-order interactive concepts) are more difficult to learn.

H Experimental Settings

Training details. We trained AlexNet [23], VGG-11 [55], ResNet-18/20 [19] on the CIFAR-10 dataset [22] and the Tiny ImageNet dataset [25], respectively. We also trained a five-layer MLP on the UCI census dataset (namely census dataset ) and the UCI TV news dataset (namely TV news dataset ) [6], respectively. Each layer of the MLP contained 100 neurons. We trained each neural networks for 200 epochs with the SGD optimizer.

Sampling details. Since, the computational cost of I ( S | x ) was intolerable in real implementation, we applied the sampling-based approximation method in [67] to calculate I ( S | x ) . Due to the high dimension of image data ( e.g. 224 × 224 for ImageNet), we uniformly split the input image into 8 × 8 patches. Furthermore, we random sampled 12 patches and considered these patches as input variables for each image. The remaining 52 patches are set to the baseline value.

Implementations details. Here, we introduce how to measure β ( S ) and κ ( S ) in Section 2.3. On the Tiny ImageNet dataset, we randomly sampled 100 training images. These training images were randomly sampled from different 10 classes. On the CIFAR-10 dataset, we randomly sampled 10 training images from each class. For tabular datasets, we randomly sampled 50 training samples from each class. For image datasets, we set τ = 2 when the input data is normalized by its standard deviation. For tabular datasets, we set τ = 1 when the input data is normalized by its standard deviation. In this way, we set the baseline b i = max ( x i -τ, µ ) , if x i > µ i , and we set the baseline b i = min ( x i + τ, µ ) , if x i < µ i . For Gaussian perturbation ϵ , we set σ = 0 . 02 . Besides, for each training sample, we randomly sampled five Gaussian perturbation with five different seeds, respectively.

Adversarial attack. Here, we introduce how to measure A ( s ) in Section 3.1. For tabular datasets, we randomly sampled 50 training samples from each class from the training set. On the Tiny-ImageNet dataset, we randomly sampled 100 training images. These training images were randomly sampled from different 10 classes. On the CIFAR-10 dataset, we randomly sampled 10 training images from each class. We used the l ∞ untargeted PGD attack by following [33], in which the constraint ϵ = 16 / 255 , and the attack was conducted with 5 steps with the step size ϵ = 3 / 255 .

Peer reviews:
comments: SUMMARY:
Theoretical work using prior work defining "interactive concepts" to quantify difficulty of concepts and show why DNNs prefer learning simpler concepts by approximating the concept learning process for DNNs by linear regression.

STRENGTHS:
1) Useful theoretical contribution that clearly uses interactive concept idea to formally define "easy" concepts and show why DNNs prefer to learn these shortcuts 

2) Well-written and easy to follow

3) Empirical proof showing that higher order interactive concepts are more vulnerable to noise in the data

WEAKNESSES:
1) More discussion can be provided on why this particular experiment designed by the authors is a good way to verify the claims made. 

2) More clear explanation of what is the contribution of prior work and that of this work.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the problem of the learning difficulty of interactive concepts. The paper theoretically shows that a DNN is more likely to encode simple interactive concepts (with fewer variables in the interaction). Low-order interactive concepts are more stable to data noises and they exhibit consistent effects on the inference scores of different samples. Experiments on various DNN architectures using image and tabular datasets are done to validate the results. The paper also compares and relates their findings with existing theoretical and empirical works on trying to analyze and explain DNNs.\n\nSTRENGTHS:\n[S1] The paper presents a novel contribution by deriving an approximate analytical solution to the variance of interactive content's effects w.r.t data noise and showing that it increases along the order of concepts in an exponential manner.\n\n[S2] The proof provides novel insights explaining DNNS. The findings are useful in explaining existing empirical works such as adversarial robustness results for different ordered concepts, why adversarial training is faster for certain concepts, etc\n\n[S3] Experiments are conducted on four types of architectures and two datasets. The results are consistent with the theoretical findings.\n\n[S4] Paper is well-written and covers various existing works.\n\nWEAKNESSES:\n[W1] Authors claim that easy samples mainly contain low-order interactive concepts. Is there existing work showing that this claim is true or can authors show it? How is \"easy\" defined? Is it based on how \"easy\" (fast?) is it for a specific network to learn or is it a more universal concept?\n\n[W2] How was the variance value used in the experiments chosen? What is the impact if it is changed? What is the impact if a different type of noise is used?\n\n[W3] Could authors experiment on newer architectures such as transformers? What about textual data input?\n\n[W4] Can you provide a limitation/broader impacts section? E.g., Are there any assumptions? Does the definition of \"simplicity\" match real-world applicability? What can/cannot this new understanding do towards guiding the design of future networks, etc..\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe paper does not provide a limitations section. Perhaps authors can explain the assumptions behind the proof and its impacts on real-world applications. The paper does not provide any ethical statement in the paper nor do I think it is needed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper, the authors proved several theoretical results that formalized the idea that simple interactive concepts achieve a smaller variance in their interaction effects in the face of Gaussian perturbations, and are thus more stable and easier for deep neural networks (DNNs) to learn. The authors also provided experimental verifications to verify their theoretical results, and pointed out the connections between their theoretical work and previous findings on DNN learning.

STRENGTHS:
- The paper provides a rigorous theoretical framework for understanding concept learning of DNNs.
- The paper formalizes the idea of conceptual complexity and establishes a formal link between conceptual complexity and learning difficulty of DNNs.
- The paper sheds light on previous heuristic findings on DNN learning, and advances our understanding of DNNs.

WEAKNESSES:
- While the theoretical derivations of the results in the paper are impressive, I am not sure how the insights from a theoretical understanding of DNN learning could help us train better DNNs. In other words, I am not certain about the practical implications and contributions of this work.

QUESTIONS:
How can we leverage the results and theoretical insights from this work to design better algorithms for training DNNs, or help DNNs learn complex concepts?

LIMITATIONS:
Since I did not find a section on the limitations of their work, the authors are encouraged to include a discussion of the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
The paper provides theoretical results explaining why simpler concepts are easier to learn for neural networks. The paper refers to prior work on interactive concept models, which define a concept as a subset of input features, and defines the complexity of each concept as the number of features contained within it. The main theoretical results of the paper argue that the variance of a concept and its interaction under noisy input increases exponentially with the size or complexity of the concept. Hence, more complex concepts are more likely to be influenced by small variations in the data. The paper then presents empirical evidence supporting these claims, and shows empirically that lower order concepts are more stable and learnt quicker than higher order ones.

STRENGTHS:
1. The paper proposes an interesting analysis around interactive concepts.
2. The empirical evaluation backs the claims of the paper.

WEAKNESSES:
1. The theoretical analysis does not match the claims of the paper's abstract. The theory only explains higher variance for higher order concepts, but the connection between higher variance and difficulty of learning is not made explicit theoretically. 
2. Several lines of work around shortcut learning and simplicity bias of networks are missing from related works.

QUESTIONS:
1. Can the authors make the connection between variance and learning difficulty more concrete theoretically?

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07