Disentangling Regional Primitives for Image Generation

## Abstract

Abstract This paper presents a method to explain the internal representation structure of a neural network for image generation. Specifically, our method disentangles primitive feature components from the intermediate-layer feature of the neural network, which ensures that each feature component is exclusively used to generate a specific set of image regions. In this way, the generation of the entire image can be considered as the superposition of different pre-encoded primitive regional patterns, each being generated by a feature component. We find that the feature component can be represented as an OR relationship between the demands for generating different image regions, which is encoded by the neural network. Therefore, we extend the Harsanyi interaction to represent such an OR interaction to disentangle the feature component. Experiments show a clear correspondence between each feature component and the generation of specific image regions.

## 1 Introduction

The interpretability of deep neural networks (DNNs) has received increasing attention along with the fast development of deep learning. However, there is a clear technical boundary between techniques of explaining a single scalar output score of a DNN(^1^11Outputs of these models are supposed to be summarized as a single or very few scalar scores.) and methods of explaining the high-dimensional output (e.g., an image) of a DNN.
For example, for DNNs for classification, attribution methods (Simonyan, ; Shrikumar et al., ; Selvaraju et al., ) were developed to estimate attributions of input variables to the scalar classification confidence. Zeiler & Fergus () and desai & Ramaswamy () visualized inference patterns encoded by a DNN, which determined the scalar classification confidence. In contrast, for image generation, the generated image contains much richer information than a scalar classification confidence, so it is difficult to apply previous explanation techniques designed for a single scalar output of a DNN. Instead, image generation is usually explained by controlling image generation results via input engineering (Härkönen et al., ; Voynov & Babenko, ).

Therefore, the essence of such difference in the explanation techniques lies in
the two facts. (1) The explanation of a single scalar output(^2^22For example, people usually explain the scalar classification confidence $\log\frac{p\left(y=y^{\text{truth }}\mid\mathbf{x}\right)}{1-p\left(y=y^{\text {truth }}\mid\mathbf{x}\right)}$ of most discriminative models for multi-category classification) can be considered to explain the the structure of a DNN’s inference logic. (2) In comparison, there is no solid theory developed to explain the internal representation structure of the neural network for image generation. For example, as Figure 1 shows, both attribution/importance scores of input variables to the classification confidence (Selvaraju et al., ) and interactions between input variables for inference (Ren et al., , ) all reflect potential representation structure of the DNN.

However, how to explain the internal representation structure of an image-generation model has not been sophisticatedly formulated. This problem can be discussed in the following two aspects.

First, we can empirically say that most DNNs do not encode image information at a pixel level(^3^33PixelRNN and PixelCNN (Van Den Oord et al., ) are special cases, but these models still encode spatial relationships between pixels into larger patterns, instead of handling each individual pixel independently.). Instead, image generation of the DNN is conducted somewhat like pasting a set of pre-encoded regional patterns, rather than let each pixel be generated independently. Therefore, how to faithfully formulate and quantify the primitive regional patterns is the core of explaining the internal representation structure of image-generation models.

Second, is it possible for the image generation result to be mathematically represented as the superposition of different primitive regional patterns?

Therefore, in order to formulate the representation structure of a given DNN for image generation, in this study,
we disentangle the feature $f$ of an intermediate layer of the DNN into different feature components $f=f_{0}+\Delta f_{1}+\Delta f_{2}+...+\Delta f_{m}$, each being used to generate a specific subset of image regions. Specifically, as Figure 1 shows, the disentanglement of feature components should satisfy the following two requirements.

$\bullet$ Each feature component $\Delta f_{i}$ is responsible for generating a specific primitive regional pattern $S_{i}$.

$\bullet$ The generation of the entire image can be explained as the superposition of different primitive regional patterns.

In this way, above two requirements ensure the faithfulness of the explanation of the image generation. The generation of a certain subset of image regions is exclusively determined as the superposition of different primitive regional patterns, each being generated by a specific feature component.

To this end, we prove that the feature component $\Delta f_{i}$ represents a certain OR relationship between the demands for the generation of different image regions. We extend the theory of the Harsanyi interaction (Harsanyi, ) to mathematically formulate the OR relationship between different primitive regional patterns used by a given neural network. Our theory ensures that the generated image can be represented as a linear superposition of these primitive regional patterns.

As Figure 2 shows, the OR interaction means that when the DNN is required to generate any one region of $S_{i}$, $\Delta f_{i}$ must be added into the feature in the intermediate layer.

Figure: Figure 1: Different ways of explaining representation structures of a DNN. Both estimating the attribution of input variables (a) and extracting interactions between input variables encoded by the DNN (b) partially explain the representation structure of a DNN for image classification. (c) In comparison, we propose to decompose the feature in an intermediate layer $f$ into different feature components $\Delta f_{1},\Delta f_{2},...,\Delta f_{m}$. Each $i$-th feature component $\Delta f_{i}$ is exclusively used to generate an primitive regional pattern in $S_{i}$. Thus, the generation of an entire image can be explained as the superposition of all primitive regional patterns.
Refer to caption: x1.png

In sum, this paper proposes a new method to disentangle feature components from the intermediate layer of a DNN. It is theoretically guaranteed that each feature component is exclusively used to generate a specific set of image regions. Experiments have demonstrated the faithfulness of the proposed explanation of image generation.

## 2 Related Work

The interpretability analysis of GAN model.
In recent years, numerous studies have explored the interpretability of GAN models. Some studies explained the GAN models by analyzing the latent space of GANs (Abdal et al., ; Härkönen et al., ; Lang et al., ; Patashnik et al., ). For instance, Härkönen et al. () employed principal component analysis (PCA) to identify the main directions in the latent space. By making layer-wise perturbations along these directions, it achieved interpretable control of the image generation process. While most of these works focused on GAN interpretability from the perspective of input manipulation, some other works took a different approach by examining the role of intermediate neurons in the generation process. For example, Bau et al. () identified specific neurons associated with certain object categories. In this way, this method enabled people to control the presence of certain objects in the generated images by adjusting the activation of the neurons.

Figure: Figure 2: The selection of feature components reflects an OR relationship between demands of reconstructing different regions. Given the demand of reconstructing a certain set of image regions, a feature component $\Delta f_{k}$ is selected if its action region $S_{k}$ contains any one target image region for reconstruction.
Refer to caption: x2.png

Unlike previous output control based on input engineering, we aim to explore the internal representation structure of an image-generation network, which represents a new explanation perspective. We disentangle feature components that control the generation of specific regions, and we discover that the feature component can be formulated as the OR relationship between the demands of reconstructing different image regions.

Interaction-based DNN explanation. In the field of explainable AI, an emerging question is whether the decision-making process of DNNs can be interpreted as a set of sparse symbolic concepts. To address this, a theoretical system based on the Harsanyi interaction has been proposed to explain how symbolic concepts are encoded by DNNs. Over the past three years, about 20 articles have been published in the field of explainable AI, which aimed to tackle the mathematical possibility of explaining DNN inference logic through a limited set of logical patterns. Most of these studies were surveyed by Ren et al. (). Specifically, Sundararajan et al. (); Janizek et al. (); Tsai et al. () proposed different types of interactions between input variables of a DNN. Ren et al. () used the Harsanyi dividend (Harsanyi, ) to represent the AND interaction in a DNN. They also experimentally discovered that DNNs usually encoded a limited set of interactions between input variables, i.e., the sparsity of AND interactions. Li & Zhang () revealed that low-order interactions exhibited higher transferability across different input samples in discriminative neural networks. Ren et al. () proved the three common conditions under which the sparsity of interactions could be guaranteed. Ren et al. () proposed a method to learn optimal masked states of input variables based on interactions and alleviated the bias of the Shapley value caused by the sub-optimal masked states of input variables. Furthermore, Chen et al. () extracted common interactions across different neural networks, and interactions shared by different neural networks usually represented generalizable inference patterns.
Cheng et al. () proposed to extract the interactions from the intermediate layers of neural networks, so as to illustrate how DNNs gradually learned and forgot inference patterns during forward propagation.

In addition, the interaction theory can also explain the representation power of a neural network. Specifically, this can be elaborated as follows.
Wang et al. () discovered and proved the negative correlation between DNNs’ adversarial transferability and the interaction inside adversarial perturbations. Ren et al. () found adversarial attacks primarily affected the high-order interactions than low-order interactions. Similarly, Zhou et al. () revealed that low-order interactions tended to generalize better than high-order interactions. Liu et al. () explained the intuition that DNNs learned low-order interactions more easily than high-order interactions.
Deng et al. () proved a counter-intuitive bottleneck i.e., bivariate interactions of middle orders were usually not well encoded by a neural network. Ren et al. () found that Bayesian neural networks (BNNs) tended to avoid encoding complex Harsanyi interactions, compared to normal neural networks.
Ren et al. () and Zhang et al. () discovered and proved the dynamics of learning interactions in neural networks exhibited a two-phase phenomenon, which had been widely observed in different neural networks and tasks.
Deng et al. () proved that the mechanisms underlying fourteen different classical attribution methods could all be rewritten as different redistributions of interaction effects to input variables.

In sum, most previous studies used interactions to explain the scalar output of a neural network. In comparison, in this paper, we first attempt to apply interactions to explain high-dimensional image generation of a DNN, which proposes fully new challenges. To this end, we discover that each feature component disentangled from the neural network can be formulated as an OR interaction between demands of reconstructing different image regions. Experiments verified the effectiveness of the proposed method.

## 3 Explaining representation structure of image generation

### 3.1 Preliminary: AND interaction

As the theoretical foundation of the explanation of an image-generation network, let us first introduce the definition of interactions in the task of image classification. Given an input image $\mathbf{x}$, let $v(\mathbf{x})$ denote the output of a DNN for image classification. We can set $v(\mathbf{x})=\log\frac{p\left(y=y^{\text{truth }}\mid\mathbf{x}\right)}{1-p
\left(y=y^{\text{truth }}\mid\mathbf{x}\right)}$ to denote the classification confidence. Let us divide the image $\mathbf{x}$ into $n=H\cdot W$ regions, and therefore we rewrite $\mathbf{x}=[\mathbf{x}_{1},\mathbf{x}_{2},…,\mathbf{x}_{n}]^{T}$, where $\mathbf{x}_{k}$ denotes the $k$-th image region. We use $N=\{1,2…,n\}$ as the set of indices of all image regions. For each specific set $S\subseteq N$ of image regions, the numerical effect of the Harsanyi interaction between image regions in $S$ is computed as

$$ $I(S)\stackrel{{\scriptstyle\text{ def }}}{{=}}{\sum}_{T\subseteq S}(-1)^{|S|-| T|}\cdot u(T)$ (1) $$

where $u(T)\stackrel{{\scriptstyle\text{ def }}}{{=}}v\left(\mathbf{x}_{T}\right)-v
\left(\mathbf{x}_{\emptyset}\right)$, and $v\left(\mathbf{x}_{T}\right)$ denotes the classification confidence w.r.t. the true category of the masked sample $\mathbf{x}_{T}$, where regions of $N\backslash T$ are masked, while regions in $T$ remain unchanged. Therefore, $u(N)=v(\mathbf{x})-v(\mathbf{x}_{\emptyset})$ represents the overall effect of all the input variables.

Each Harsanyi interaction represents an AND relationship between input variables encoded by the DNN, and contributes a certain effect $I(S)$ to the network output $v(\mathbf{x})$. For example, as shown in Fig 1(b), the DNN encodes the non-linear relationship between $S=\{head,mantle,body...\}$ to form a bird pattern. The co-appearance of all images regions in $S$ triggers the AND relationship of the bird pattern and makes an effect $I(S|\mathbf{x})$ to the classification confidence $u(N)$. The absence of any region in $S$ will remove the effect $I(S|\mathbf{x})$ from $u(N)$.

Ren et al. () and Zhou et al. () have discovered and partially proven the sparsity property and the universal-matching property of the interactions, as the mathematical guarantee for taking interactions as faithful primitive inference patterns encoded by the DNN. According to Theorem 3.1, we can construct a surrogate logical model $h(\cdot)$ based-on the extracted interactions. This surrogate logical model can accurately fit the classification confidence $u(\cdot)$ of a DNN, no matter how the input is masked, i.e., $\forall T\subseteq N,u(T)=h(\mathbf{x}_{T})$. The above property is called the universal-matching property.

###### Theorem 3.1 .

(Universal-matching property (Ren et al., ), also proved in Appendix B). Given an input sample $\mathbf{x}$, the output $u(T)$ on each masked sample $\{\mathbf{x}_{T}\mid T\subseteq N\}$ can be well matched by a surrogate logical model $h(\mathbf{x}_{T})$. The surrogate logical model sums up effects of all interactions that are triggered by the masked sample $\mathbf{x}_{T}$ as the output score.

$$ $\displaystyle\forall T\subseteq N,u\left(T\right)=h\left(\mathbf{x}_{T}\right),\,$ (2) $\displaystyle\textit{subject to}\,h\left(\mathbf{x}_{T}\right)\stackrel{{ \scriptstyle\text{def}}}{{=}}\sum_{S\subseteq N,S\neq\emptyset}\mathbbm{1}( \begin{array}[]{c}\textit{$\mathbf{x}_{T}$ triggers AND }\\ \textit{relation $\,S$}\end{array})\cdot I\left(S\right)=\sum_{S\subseteq T,S \neq\emptyset}I\left(S\right)$ $$

Besides the universal-matching property, the sparsity property is another property that guarantees the faithfulness of the interaction-based explanation. When we enumerate all subsets $S\subseteq N$ and compute effects of all $2^{n}$ interactions, it is proved that for the DNNs with stable outputs on masked input samples,(^4^44Ren et al. () proposed the common conditions for smooth outputs on masked inputs, which could be satisfied by most well-trained DNNs.) most of the extracted interactions have negligible effects on the output, i.e., $I(S|\mathbf{x})\approx 0$. Only a small set of interactions, denoted by $\Lambda=\{S\subseteq N:|I(S)|>\tau\}$, have considerable effects , where $\tau$ is a small scalar threshold. Therefore, we consider such a small number of salient interactions as the faithful explanation of the DNN. As Corollary 3.2 shows, the surrogate logical model $h(\cdot)$ can be approximated by these salient interactions.

###### Corollary 3.2 ( Sparsity property (Ren et al., 2024a ) ) .

The surrogate logical model $h(\mathbf{x}_{T})$ on each randomly masked sample $\mathbf{x}_{T}$, $T\subseteq\mathcal{N}$, uses the sum of a small number of salient interactions to approximate the network output score $u(T)$.

$$ $\forall T\subseteq N,\qquad u(T)=h(\mathbf{x}_{T})\approx{\sum}_{S\subseteq T, S\neq\emptyset}I(S)$ (3) $$

###### Theorem 3.1 .

###### Corollary 3.2 ( Sparsity property (Ren et al., 2024a ) ) .

### 3.2 Two requirements to explain representation structure

In this paper, we aim to extract the internal representation structure encoded by a DNN for image generation. Instead of generating each pixel independently(^5^55Except for neural networks like PixelCNN and PixelRNN.), lots of empirical findings (Härkönen et al., ; Voynov & Babenko, ) all showed that a DNN usually encoded a set of regional patterns as the internal structure of an image, and the generation of an image could be considered as the superposition of the pre-encoded regional patterns.

Therefore, in this paper, we extend the Harsanyi interaction to represent such regional patterns from a trained DNN. Let a DNN generate an image $\mathbf{x}$ with a certain input code. We divide the generated image into $n=H\cdot W$ images, denoted by $\mathbf{x}=[\mathbf{x}_{1},…,\mathbf{x}_{n}]^{T}$. $N=\{1,2…,n\}$ denotes the set of indices for image regions. Let $f\in R^{D}$ denote the feature of an intermediate layer of the DNN. Then, the objective of this study is to decompose the feature $f$ into $m$ feature components, so as to let each feature component $\Delta f_{k}$ exclusively generate a specific regional pattern.

$$ $\mathbf{x}=g(f),\qquad f=f_{0}+\Delta f_{1}+\Delta f_{2}+…+\Delta f_{m}$ (4) $$

where $f_{0}$ represents a baseline feature as a non-informative feature state, and $M=\{1,2,...,m\}$ denotes the index set of all feature components. $f_{0}$ can be set as the average feature over all features given different input codes $\boldsymbol{z}$. i.e., $f_{0}=\mathbb{E}_{\boldsymbol{z}}[d(\boldsymbol{z})]$, where $d(\cdot)$ denotes the modules of the image-generation model between the input code $\boldsymbol{z}$ and the intermediate feature. We use $g(f)$ to represent the image generated by the feature $f$, i.e., $\boldsymbol{z}\xrightarrow{\text{$d$}}f\xrightarrow{\text{$g$}}\mathbf{x}$.

Two requirements for feature decomposition. In this way, each feature component $\Delta f_{k}$ added upon the baseline feature $f_{0}$ is supposed to exclusively generate a certain regional pattern $S_{k}\subseteq N$, and thus can be taken as the primitive patterns encoded by the DNN. $S_{k}$ is also termed the action region of the $k$-th feature component. In this way, the generation of specific image regions is controlled by a set $\Omega\subseteq M$ of feature components, denoted by $F(\Omega)\stackrel{{\scriptstyle\text{def}}}{{=}}f_{0}+{\sum}_{k\in\Omega}
\Delta f_{k}$. In particular, $F(M)=f$. To achieve this, the feature decomposition is conducted w.r.t. the following two requirements.

Requirement 1: Each feature component $\Delta f_{k}$ exclusively generates a specific set of image regions $S_{k}\subseteq N$, without affecting other image regions. I.e., the addition of $\Delta f_{k}$ to $F(\Omega)$ should not change the generation of other regions.

$$ $\begin{gathered}\forall k\in M,\quad\forall\Omega\subseteq M\setminus\{k\}, \quad g_{N\setminus S_{k}}(F(\Omega))=g_{N\setminus S_{k}}(F(\Omega\cup\{k\})) \\ \textit{s.t.}\quad F(\Omega\cup\{k\})=F(\Omega)+\Delta f_{k}\end{gathered}$ (5) $$

$g_{N\setminus S_{k}}(\cdot)$ denotes image regions in $N\setminus S_{k}$ selected from the generated image $g(\cdot)$.

Requirement 2-$\alpha$. The generation of each $i$-th image region can be fully determined by the superposition of all feature components that cover the $i$-th region.

$$ $\mathbf{x}_{i}=g_{S=\{i\}}(F(\Omega)),\quad\textit{subject to}\quad\Omega=\{k \mid i\in S_{k}\}$ (6) $$

i.e., $g(F(\Omega))$ well generates the $i$-th region $\mathbf{x}_{i}$ of the target image $\mathbf{x}$.

Requirement 2-$\beta$. The generation of image regions in $\hat{S}$ can also be exclusively determined by the superposition of feature components that intersect with $\hat{S}$, i.e.,

$$ $\mathbf{x}_{\hat{S}}=g_{\hat{S}}(F({\hat{\Omega}})),\quad\textit{subject to} \quad\hat{\Omega}=\{k\mid S_{k}\cap\hat{S}\neq\emptyset\}$ (7) $$

where $\hat{\Omega}$ denotes the set of feature components whose action regions partially cover regions in $\hat{S}$.

Requirement 1 shows that all regions in $S_{k}$ can be considered as a singleton visual pattern encoded by the DNN. The above requirements ensures that the feature decomposition is a faithful explanation of the representation structure of the DNN generating the image $\mathbf{x}$.

Minimal feature for regional generation. We can consider that $\Delta f_{1}+\Delta f_{2}+...+\Delta f_{m}$ are features required to be added to the baseline feature $f_{0}$ to generate the entire image $\mathbf{x}$, in Equation (4).
If we are only requested to reconstruct a subset $\hat{S}$ of image regions, then we only need to use feature components $F(\hat{\Omega})=f_{0}+\sum_{k\in\hat{\Omega}}\Delta f_{k}$, s.t. $\hat{\Omega}=\{k\mid S_{k}\cap\hat{S}\neq\emptyset\}\subseteq M$, and the addition of any further feature components will not affect the generation of regions in $\hat{S}$, according to above requirements. $\hat{\Omega}_{\hat{S}}$ denotes the set of feature components selected to reconstruct image regions in $\hat{S}$. In this way, we can consider $F(\hat{\Omega})$ as the minimal feature of generating image regions in $\hat{S}$, denoted by $u(\hat{S})=F(\hat{\Omega})$. On the other hand, the minimal feature $u(\hat{S})$ can be also estimated as follows.

$$ $u(\hat{S})=\operatorname{argmin}_{u(\hat{S})}\|u(\hat{S})\|_{\text{L-1}},\quad \textit{ w.r.t.}\quad\mathbf{x}_{\hat{S}}=g_{\hat{S}}(u(\hat{S})).$ (8) $$

Therefore, the decomposition of feature components $\Delta f_{k}$ w.r.t. above requirements can be written as follows.

$$ $\displaystyle\textit{Decomposion of }f=f_{0}+{\sum}_{k\in M}\Delta f_{k} \textit{ w.r.t. Requirement 1,2-$\alpha$,2-$\beta$}\quad$ (9) $\displaystyle\equiv$ $\displaystyle\quad\operatorname{min}_{\{\Delta f_{k}\}}\|u(\hat{S})\|_{\text{L -1}}\,\textit{s.t.}\,\forall\hat{S}\subseteq N,u(\hat{S})=F(\hat{\Omega})=f_{0 }+{\sum}_{k\in\hat{\Omega}}\Delta f_{k}$ $$

where $\hat{\Omega}=\{k\mid S_{k}\cap\hat{S}\neq\emptyset\}$

Implementation details. The computation of the minimal feature $u(\hat{S})$ w.r.t. image regions in $\hat{S}$ can be approximated as $\underset{\alpha}{\operatorname{min}}\,\|u(\hat{S})\|_{L-1}+\frac{\lambda}{|
\hat{S}|}\|g_{\hat{S}}(u(\hat{S}))-\mathbf{x}_{\hat{S}}\|_{L-2}^{2},\,\textit{
subject to}\,\|g_{\hat{S}}(u(\hat{S}))-\mathbf{x}_{\hat{S}}\|_{L-2}<\tau,u(
\hat{S})=f_{0}+\alpha\odot\Delta f$.
Here, we consider the minimal feature are all contained by the whole feature $f$, so we apply the empirical constrain $u(\hat{S})=f_{0}+\alpha\odot(f-f_{0}),\textit{w.r.t.}\,\alpha\in{[0,1]^{D}}$. Each feature dimension of $\alpha$ is in the range of [0,1]. $\odot$ is referred to as the element-wise multiplication. In this way, we constrain $u(\hat{S})$ strictly locates within the range between $f_{0}$ and $f$.

### 3.3 Using interactions for feature disentanglement

The above subsection introduces how to compute the minimal feature component, and clarifies the mathematical connection between the feature components $\{\Delta f_{k}\}_{k}$ and minimal features $\{u(\hat{S})\}_{\hat{S}}$. Then, in this subsection, we introduce how to disentangle feature components $\Delta f_{1},...,\Delta f_{m}$ from all minimal features, so as to let each feature component $\Delta f_{k}$ exclusively generate regions in $S_{k}$.

OR interaction. To this end, we prove that the proposed two requirements can be rewritten as follows.

$\bullet$ For each feature component $\Delta f_{l}$ whose action regions does not cover any regions in $S_{k}$, i.e, $S_{k}\cap S_{l}=\emptyset$, this component does not affect the generation of the image regions in $S_{k}$.

$\bullet$ $u(\hat{S})=F(\hat{\Omega})=f_{0}+\sum_{k\in\hat{\Omega}}\Delta f_{k},\quad\hat
{\Omega}=\{k\mid\hat{S}\cap S_{k}\neq\emptyset\}$

Clearly, above two terms reflects an OR relationship in the selection of feature components towards the reconstruction of the target regions in $\hat{S}$. When we need to reconstruct image regions in $\hat{S}$, then the feature component $\Delta f_{k}$ should be added to $f_{0}$ if and only if $\hat{S}$ covers any regions in $S_{k}$. For example, as Figure 2 shows, let us consider a feature component $\Delta f_{k}\,\textit{w.r.t.}\,S_{k}=\{1,3\}$. Then, this feature component must be added to $f_{0}$ when the reconstruction demand includes either the \nth1 or the \nth3 image region. If the construction demand does not contains any region in $S_{k}$, then $\Delta f_{k}$ should not be added. Therefore, we can consider the selection of the feature component $\Delta f_{k}$ reflects an OR relationship between the target reconstruction regions.

Each feature component $\Delta f_{k}$ can be formulated as an OR interaction between reconstruction demands of different regions in $S_{k}$, as follows. Fortunately, we realize that we can extend the Harsanyi interaction theory (Ren et al., ) to represent such OR interaction. It is because the OR relationship is equivalent to a specific AND relationship, i.e., a feature component $\Delta f_{k}$ is not added if and only if all regions in $S_{k}$ are not in demand for reconstruction.

$$ $\Delta f_{k}=\operatorname{\boldsymbol{I}_{or}}(S_{k})=-{\sum}_{S^{\prime} \subseteq S_{k}}(-1)^{|S_{k}|-|S^{\prime}|}u\left(N\backslash S^{\prime}\right ),S_{k}\neq\phi$ (10) $$

Please see Algorithm 1 for the pseudo-code of disentangling feature components.

Figure: Algorithm 1 Disentangling feature components and find salient feature components

Just like the Harsanyi interaction (or the AND interaction), the above definition of the OR interaction satisfies the following universal property.

###### Theorem 3.3 .

(Universal matching property of the OR interaction, proof in Appendix D). Given the demand of reconstructing regions in $\hat{S}$ in the target image $\mathbf{x}$, the minimal feature $u(\hat{S})$ for image reconstruction can be well estimated by a surrogate logical model $h(\mathbf{x}_{\hat{S}})$. The surrogate logical model sums up feature components corresponding to all interactions that are triggered by the reconstruction target $\hat{S}$.

$$ $\displaystyle\forall\hat{S}\in N,u(\hat{S})=h(\mathbf{x}_{\hat{S}}),\,\textit{ subject to}\,$ (11) $\displaystyle h(\mathbf{x}_{\hat{S}})\stackrel{{\scriptstyle\text{def}}}{{=}}f _{0}+{\sum}_{S_{k}\subseteq N,S_{k}\neq\emptyset}\mathbbm{1}(\begin{array}[]{c }\textit{$\mathbf{x}_{\hat{S}}$ triggers OR}\\ \textit{relation $\,S_{k}$}\end{array})\cdot\Delta f_{k}=f_{0}+{\sum}_{S_{k} \cap\hat{S}\neq\emptyset,S_{k}\neq\emptyset}\Delta f_{k}$ $$

Theorem 3.3 ensures that OR interactions for feature decomposition satisfy the Requirement 2, i.e., when the demand of reconstructing regions in $\hat{S}$ will trigger all feature components $\Delta f_{k}$, as long as $\hat{S}$ partially covers any regions in $S_{k}$. Please see Algorithm 2 in Appendix E for a pseudocode of selecting a specific set $\hat{\Omega}_{\hat{S}}$ of feature components to reconstruct the target image regions in $\hat{S}$.

In addition, because the OR interaction can be considered as a special AND interaction that reverses the definition of the masked state and the unmasked state, the OR interaction also satisfies the sparsity property. The sparsity property shows that the minimal feature is the sum of a few non-zero feature dimensions in feature components, while other dimensions in feature components have negligible effects. The action region of a salient feature component can be view as a primitive regional pattern. The entire image $\mathbf{x}$ is generated by superimposing these primitive regional patterns.

###### Theorem 3.3 .

## 4 Experiment

We tested our method on the BigGAN-128 model (Brock, ). We used the intermediate features in the layer2 for the feature decomposition, i.e., the output of the first ResBlock in BigGAN. The baseline feature component $f_{0}$ was computed as the average feature over different input codes $\boldsymbol{z}$, i.e., computing $f_{0}=\mathbb{E}_{\boldsymbol{z}}[d(\boldsymbol{z})]$, when we set the DNN to generate images in different categories. Given an input code $\boldsymbol{z}$, the DNN generated the image $\mathbf{x}$, and the image was segmented into a $6\times 6$ grid. Because the computation cost of interactions was NP-complete, we randomly selected $n=9$ grids on the foreground object, and analyzed the OR interactions between these image regions encoded by the DNN.

### 4.1 Visualizing feature components (OR interactions)

Given a random input code $\boldsymbol{z}$, we used the DNN to generate an image $\mathbf{x}$, and extracted feature components used for image generation. We followed the feature decomposition method described in Section 3. In order to obtain the baseline feature $f_{0}$, we used the DNN to generate different images in different categories, and extracted the average of intermediate features over all the generated images as the baseline feature $f_{0}$ in experiments. In order to compute the OR interaction corresponding each feature component, we divided the original images into 6$\times$6 regions in grids, and randomly chose 9 regions of the main part of the image as input variables. The parameter $\alpha$ was initialized to a vector with all-ones elements, and $\lambda$ was set to 10000. We optimized $u(N)$, i.e., the corresponding minimal feature of all regions in $N$. For each other set of regions $S\subseteq N$, the solution corresponding to $u(N)$ was set as the starting point of further computation of $u(S)$. Then we followed the algorithm in Equation 10 to compute $I_{\text{or}}(S_{k})$. We used the L-2 norm $\|\Delta f\|_{L-2}$ to rank the interaction strength of each feature component (please see Appendix H for more detailed settings). Figure 6 shows the sparsity of the extracted interactions. This figure visualizes all elements of all the extracted interactions (feature components) by sorting their absolute values in a descending order. Most elements of feature components were almost zero, which verified the sparsity of the feature components.

Figure 4 verifies that the different feature components were exclusively responsible for the reconstruction of their own action regions. The interpretability of feature components enabled us to control the reconstruction of different image regions by adding different feature components.

### 4.2 Verifying the disentangled feature components (OR interactions)

Towards the reconstruction of a single image region. In order to verify whether the disentangled feature components were exclusively responsible for the reconstruction of specific image regions, we randomly selected a region $i\in N$ of the image $\mathbf{x}$. We chose all the feature components whose action regions included the $i$-th region. Then, we added these feature components into the baseline feature $f_{0}$, and obtained $\hat{f}=f_{0}+\sum_{k\in\Omega}f_{k},\textit{s.t.}\,\Omega=\{k|i\in S_{k}\}$. Figure 7(a) shows image reconstruction results when we added the ratio $p$ of feature components that covered the $i$-th image region. We discovered that the feature $\hat{f}$ only well constructed the $i$-th region in the image $\mathbf{x}$. The target image region was gradually reconstructed when we added increasing numbers of feature components.

Towards the reconstruction of a set of image regions. We further evaluated our method in the reconstruction of a set of image regions. We randomly selected a set of regions $S\subseteq N$ in the image $\mathbf{x}$. We chose all the feature components whose action regions partially covered $S$. Then, we added these feature components into the baseline feature $f_{0}$, and obtained $\hat{f}=f_{0}+\sum_{k\in\Omega}f_{k},\textit{s.t.}\,\Omega=\{k|S\cap S_{k}\neq\emptyset\}$. Figure 7(b) shows image reconstruction results when we added the ratio $p$ of feature components whose action regions partially covered $S$. We discovered that the feature $\hat{f}$ only well constructed the regions in $S$ in the image $\mathbf{x}$. The target image regions were gradually reconstructed when we added increasing numbers of feature components.
Examining image reconstruction effects of irrelevant feature components. From another perspective, we also examined whether irrelevant feature components would affect the generation of a region. We randomly selected
the $i$-th region in image $\mathbf{x}$. Then, we gradually added all the feature components whose action region did not cover the $i$-th region, i.e., $\hat{f}=f_{0}+{\sum}_{k\in\Omega}\Delta f_{k}\,\textit{s.t.}\,\Omega=\{k|i
\notin S_{k}\}$. Figure 5 shows images reconstructed by adding the ratio $p$ of irrelevant feature components in $\Omega$. We found that although we had added all irrelevant feature components, the target image region was not changed.

Figure: Figure 3: Action regions of the seven most salient feature components.
Refer to caption: x3.png

Figure: Figure 4: Incremental reconstruction of different image regions when we gradually added feature components. We added all feature components corresponding to the target image regions (in red boxes) for image reconstruction. It shows that different regions in the target object were sequentially reconstructed, but these feature components did not reconstruct the background. The heatmap shows the distribution of the overlapping action regions of the added feature components.
Refer to caption: x4.png

## 5 Conclusion

In this paper, we introduce a new method to explain the internal representation of a image generation neural network. We use the OR interaction to disentangle components from the intermediate feature of the neural network. Our theory ensures that each component exclusively generates a primitive regional pattern, and the generation of the whole image can be explained as the superposition of all the extracted primitive regional patterns. Experiments have validated the faithfulness of the explanation of the representation structure of the neural network. I.e., each component is only responsible for the generation of a set of regions. People can control the neural network to exclusively reconstruct a specific set of image regions by adding feature components corresponding to these regions. As a limitation of the current theory, there is no strictly theory to constrain all feature components within the manifold of the intermediate-layer features that are generated by the input code $\boldsymbol{z}$, although experiments have verified the effectiveness of using these feature components to control image generation. Thus, ensuring and computing the input code that corresponds to each combination of these feature components are the future work of this study.

Figure: Figure 5: Validation of Requirement 1, i.e., irrelevant feature components do not reconstruct the target region. We added different ratios $p$ of irrelevant feature components whose action regions did not cover the target image regions (in red boxes). We found that the target image region was not reconstructed, when we added these irrelevant feature components. Very weak reconstruction effects are caused by small computational errors.
Refer to caption: x5.png

Figure: Figure 6: Sparsity of the extracted interaction
Refer to caption: x6.png

Figure: Figure 7: Validation of Requirement 2, i.e., relevant feature components reconstruct the target region. We added different ratios $p$ of feature components whose action regions covered the target image regions (in red boxes). We found that the target image region was gradually reconstructed when we added increasing number of feature components.
Refer to caption: x7.png

## Appendix A Axioms and theorems for the Harsanyi dividend interaction

The Harsanyi dividend was designed as a standard metric to measure interactions between input variables encoded by the network. In this section, we present several desirable axioms and theorems that the Harsanyi dividend interaction $I(S)$ satisfies. This further demonstrates the trustworthiness of the Harsanyi dividend interaction.

The Harsanyi dividend interactions $I(S)$ satisfies the *efficiency, linearity, dummy, symmetry, anonymity, recursive* and *interaction distribution* axioms, as follows. We follow the notation in the main paper to let $u(S)=v(\mathbf{x}_{S})-v(\mathbf{x}_{\emptyset})$.

$\bullet$ Efficiency axiom. The output score of a model can be decomposed into interaction effects of different patterns, i.e., $u(N)=\sum_{S\subseteq N}I(S)$.

$\bullet$ Linearity axiom. If we merge output scores of two models $u_{1}$ and $u_{2}$ as the output of model $u$, i.e. $\forall S\subseteq N,u(S)=u_{1}(S)+u_{2}(S)$, then their interaction effects $I_{u_{1}}(S)$ and $I_{u_{2}}(S)$ can also be merged as $\forall S\subseteq N,I(S)=I_{u_{1}}(S)+I_{u_{2}}(S)$.

$\bullet$ Dummy axiom. If a variable $i\in N$ is a dummy variable, i.e., $\forall S\subseteq N\setminus\{i\},u(S\cup\{i\})=u(S)$ then it has no interaction with other variables, $\forall\emptyset\neq S\subseteq N\setminus\{i\},I(S\cup\{i\})=0$.

$\bullet$ Symmetry axiom. If input variables $i,j\in N$ cooperate with other variables in the same way, i.e., $\forall S\subseteq N\setminus\{i,j\},u(S\cup\{i\})=u(S\cup\{j\})$, then they have same interaction effects with other variables, $\forall S\subseteq N\setminus\{i,j\},I(S\cup\{i\})=I(S\cup\{j\})$.

$\bullet$ Anonymity axiom. For any permutations $\pi$ on $N$, we have $\forall S\subseteq N,I_{u}(S)=I_{{\pi}u}({\pi}S)$ where ${\pi}S=\{{\pi}(i)|i\in S\}$ and the new model ${\pi}u$ is defined by $({\pi}u)({\pi}S)=u(S)$. This indicates that interaction effects are not changed by permutation.

$\bullet$ Recursive axiom. The interaction effects can be computed recursively. For $i\in N$ and $S\subseteq N\setminus\{i\}$, the interaction effect of the pattern $S\cup\{i\}$ is equal to the interaction effect of $S$ with the presence of $i$ minus the interaction effect of $S$ with the absence of $i$, i.e., $\forall S\subseteq N\setminus\{i\},I(S\cup\{i\})=I(S|i\,\text{is always
present})$ - $I(S).I(S|i\,\text{is always present})$ denotes the interaction effect when the variable $i$ is always present as a constant context, i.e. $I(S|i\,\text{is always present})$ = $\sum_{S\subseteq I(S)}(-1)^{|S|}\cdot u(L\cup\{i\})$.

$\bullet$ Interaction distribution axiom. This axiom characterizes how interactions are distributed for “interaction functions”. An interaction function $u_{T}$ parameterized by a subset of variables $T$ is defined as follows. $\forall S\subseteq N$, if $T\subseteq S$, $u_{T}(S)=c$; otherwise, $u_{T}(S)=0$. The function $u_{T}$ models pure interaction among the variables in $T$, because only if all variables in $T$ are present, the output value will be increased by $c$. The interactions encoded in the function $u_{T}$ satisfies $I(T)=c$, and $\forall S\neq T,I(S)=0$.

## Appendix B Prove for AND-interaction universal matching property

In this section, we proof the univeral matching properties for the Harsanyi interaction.

###### Theorem B.1 .

(Universal-matching property).Given an input sample $\mathbf{x}$, the output classification confidence $u(T)$ on each masked sample $\{\mathbf{x}_{T}\mid T\subseteq N\}$ can be well matched by a surrogate logical model $h(\mathbf{x}_{T})$. The surrogate logical model sums up effects of all interactions that are triggered by the masked sample $\mathbf{x}_{T}$ as the output score.

$$ $\displaystyle\forall T\subseteq N,h\left(\mathbf{x}_{T}\right)$ $\displaystyle=u\left(T\right)$ (12) $\displaystyle={\sum}_{S\subseteq N,S\neq\emptyset}\mathbbm{1}\left(\mathbf{x}_ {T}\text{ triggers AND relation }S\right)\cdot I\left(S\right)$ $\displaystyle={\sum}_{S\subseteq T,S\neq\emptyset}I\left(S\right)$ $$

proof According to the definition of the Harsanyi interaction, we have $\forall S\subseteq N$,

$$ $\displaystyle\sum_{T\subseteq S}I(T)$ $\displaystyle=\sum_{T\subseteq S}\sum_{L\subseteq T}(-1)^{|T|-|L|}u(L)$ $\displaystyle=\sum_{L\subseteq S}\sum_{T\subseteq S:T\supseteq L}(-1)^{|T|-|L| }u(L)$ $\displaystyle=\sum_{L\subseteq S}\sum_{t=|L|}^{|S|}\sum_{\begin{subarray}{c}T \subseteq S:S\supseteq L\\ |T|=t\end{subarray}}(-1)^{t-|L|}u(L)$ $\displaystyle=\sum_{L\subseteq S}u(L)\sum_{m=0}^{|S|-|L|}\binom{|S|-|L|}{m}(-1 )^{m}$ $\displaystyle=u(S)$ $$

Therefore, we have $u(S)={\sum}_{T\subseteq S}I(T)$.

###### Theorem B.1 .

## Appendix C Proving that the OR interactions can be considered as a specific AND interaction

The OR-interaction is defined as follow.

$$ $\Delta f_{k}=\operatorname{\boldsymbol{I}_{or}}(S_{k})=-\sum_{S_{l}\subseteq S _{k}}(-1)^{|S_{k}|-|S_{l}|}u\left(N\backslash S_{l}\right),S_{k}\neq\phi$ (13) $$

Here, $u\left(N\backslash S_{l}\right)$ denotes the minimal feature of the set $N\backslash S_{l}$, and regions in the set $S_{l}$ is not in the action region. Let us denote $\mathbf{x}_{T}$ as a masked state of $\mathbf{x}$, where only the regions in $T$ are presented. Furthermore, we define $\mathbf{x}^{\prime}_{T}$ as the masked state where regions in $T$ are removed. Therefore, the definition of OR interaction can be rewritten as follows:

$$ $\displaystyle I_{\text{or }}(S\mid\mathbf{x})$ $\displaystyle=-\sum_{T\subseteq S}(-1)^{|S|-|T|}v(\mathbf{x}_{N\backslash T}), \quad T\neq\phi$ (14) $\displaystyle=-\sum_{T\subseteq S}(-1)^{|S|-|T|}v\left(\mathbf{x}_{T}^{\prime} \right),\quad S\neq\emptyset$ $\displaystyle=-I_{\text{and }}^{\prime}\left(S\mid\mathbf{x}^{\prime}\right), \quad S\neq\emptyset$ $$

where $v(\mathbf{x}_{N\backslash T})=u(N\backslash T)$. Therefore, we can consider the OR interaction as a specific AND interaction.

## Appendix D Proof of Theorem 3.3 in the main paper

According to Appendix C, we reconsider the definition of the masked board state $\mathbf{x}_{T}$ as $\mathbf{x}^{\prime}_{T}$. $\mathbf{x}_{T}$ denotes the masked state where regions in the set $T$ are inside the action region, and regions in the set $N\setminus T$ are not considered. In comparison, $\mathbf{x}^{\prime}_{T}$ denotes the masked state where regions in the set $T$ are not considered, and regions in the set $N\setminus T$ are inside the action region.

In this way, the effect of an OR interaction based on the definition of $\mathbf{x}$ can be represented as the effect $I_{\text{or}}(S|\mathbf{x})$ of an AND interaction based on the definition of $\mathbf{x}^{\prime}$.

$$ $\displaystyle I_{\text{or}}(S|\mathbf{x})$ $\displaystyle=w_{S}^{\text{or}}\cdot\left[-\prod_{i\in S}\neg\textit{exist}(x_ {i})\right]$ (15) $\displaystyle=-w_{S}^{\text{or}}\cdot\prod_{i\in S}\neg\textit{exist}(x_{i})$ $\displaystyle=-\frac{w_{S}^{\text{or}}}{w_{S}^{\text{and}}}\cdot I_{\text{and} }^{\prime}(S|x^{\prime})$ $$

where the function $exist(x_{i})$ represents whether $x_{i}$ is in the action region. Therefore, as we have proven that the universal-matching property of the AND-interaction, the OR interaction still satisfies this property.

## Appendix E A pseudocode of reconstructing image regions

Algorithm 2 presents a pseudocode of selecting a specific set $\hat{\Omega}_{\hat{S}}$ of feature components to reconstruct the target image regions in $\hat{S}$.

Figure: Algorithm 2 Reconstructing the regions in $\hat{S}$ based on all the salient feature components in $\hat{\Omega}_{N}$

## Appendix F Discussion for the influence of the input code

The BigGAN model uses ResNet GAN architecture, i.e., uses residual blocks, and the input code (a concatenated vector of the latent vector and the class vector) will be fed into the following modules. Therefore, after modifying the intermediate feature of a specific layer, the generated image might still change if the input code changed, due to the residual structure.

Therefore, to validate the influence of the input code is small, we randomly selected input codes, and modify the intermediate feature to a same $f$. The results are shown in Figure 8.

Figure: Figure 8: Images generated by same intermediate feature but different input codes
Refer to caption: x8.png

The experiment result showed that the influence of the input code is relatively small (the background might change a little, but the main parts of the image didn’t change). Therefore, we can consider that most of the information is contained in the intermediate feature.

## Appendix G The implementation detail of extracting faithful feature components

Although we proved the sparsity of the OR interaction in the main paper, it is still challenging to regard those extracted regional patterns as the faithful primitive regional patterns encoded by the model. We find that the OR interaction is quite sensitive to the minimal feature and a small change to the minimal feature might not have a significant change in the generated image, but it will cause a change to the feature components we extracted.

Therefore, we ensured the interaction stability by removing the noise in the minimal feature. We use a vector $\boldsymbol{q}_{S}$ to model the noise in the minimal feature $\Delta f_{k}=I_{o}r(S)$. Each element in $\boldsymbol{q}_{S}$ is bounded in the range of $[-\tau,\tau]$, where $\tau$ is a threshold to avoid large noise.

Primitive regional patterns with stability is extracted by solving the following optimization problem:

$$ $\min_{\{\boldsymbol{q}_{S}\}}\sum_{S\subseteq N,S\neq\emptyset}\|I_{or}(S| \mathbf{x},\{\boldsymbol{q}_{S}\})\|_{L-1}$ (16) $$

In this way, we can ensure the interaction stability by removing noises in the minimal feature. Experiments in the main paper showed we can still well reconstruct the image after removing noises.

## Appendix H Experiment settings

This section includes detailed experiment settings in this paper.

In our experiment, we use BigGAN-128 to generate the image categories of ImageNet. Therefore, when calculating the baseline intermediate feature, we randomly generated images of all the 1000 classes in ImageNet. And the truncation of the BigGAN model was set to 0.4. After disentangling feature components, we ranked those feature components by their L-2 norm. In order to find out which components are salient, we removed the feature components from the intermediate feature in descending order, utill the generated image is similar to the baseline image. Empirically, we found that for most samples, the top 60% feature components were salient. And in Figure 6 we also found that in log space, about 40% interactions had relatively low interaction strength. When calculating the distribution of action regions, we performed a weighted sum of the number of feature components contained in each region, where the weight of a single feature component in a region is its L-1 norm divided by the number of its action regions. When we create heat map visualizations, we apply an exponential operation based on the weights mentioned above to enhance the contrast between different regions.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the Weakness section.

Specifically, W2, W3, W5, and W8.

Please provide an answer in the rebuttal.
soundness: 3
strengths: Strengths:

1. It seems that the OR definition introduced in this paper is dual to the AND definition in previous works, which successfully extends previous framework for generative models.
summary: This paper proposes a method to interprete the GAN model by disentangling regisonal primitives for image generation. Following the similar AND definition proposed in previous methods for classification models, this paper introduces OR definition for generation models. Some analysis validates that it shares some properties in common with AND. The authors also propose an algorithm to find salient feature components for an image. Some experiments demonstrate the methods can extract sparse interaction of regions within an image and yield plausible reconstruction results using extracted primitives.
weaknesses: Weaknesses:

1. To be honest, I cannot digest many parts of the article. This includes, but is not limited to, the tutorial of introduction, the method, and the experiments. I encourage the authors to rephrase some mathmatical descriptions with intuitive explanations.
2. The term $(-1)^{|S|-|T|}$ in many equations appear strange to me, e.g., Eq. 1 and Eq. 10. Why should we make the score for $T$ negative when the difference of the number of regions in $S$ and $T$ is odd? Please give more details on this -- it is important to understand its connection to the core idea of the submission.
3. I am not sure whether the extracted regional primitives are also applicable to generate other images. Otherwise, what is the purpose of extracting primitives for a specific image? The authors are encouraged to discuss the use cases of the method more clearly. Please provide  specific examples.
4. It seems that the algorithm needs enumerate each subset for a given set, which may be inefficient in practice. The authors are encouraged to discuss the time efficiency and hardware requirements specifically.
5. Is the addition defined in Fig. 1(c) element-wise? What is the tensor shape of each $\Delta f$? Does it represent a patch of it has the shape of a full image feature map, i.e., $C\times H\times W$? From my understanding, $f_0$ should have the shape $C\times H\times W$ and each $\Delta f$​ should also have the same shape for element-wise summation. If so, why does the visualization only contain a patch?
6. There is almost no quantitative results. Since reconstruction is one target, at least there should be some numerical evaluation regarding this, e.g., PSNR, SSIM, LPIPS, etc.
7. There is no comparison with other methods that also focusing on interpreting GAN. Please compare to the SOTA methods.
8. Why do the authors choose layer2 of BigGAN for validation? How does this choice influence the results?  How the results might change if different layers were used
9. The writing of the beginning part of Sec. 4 is largely overlapping with that of Sec. 4.1. Overall, the presentation, in terms of clarity, organization, and distinction with previous works, can be enhanced a lot.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: * What benefits would the current method add to the overall image generation task?  
* On what tasks could the proposed algorithm provide significant improvement?
* I don't understand the motivation of choosing specific regioins of the image to perform the analysis in Figure 3? 
* Could you explain the definition of the "action region" in Figure 3?
soundness: 2
strengths: * The paper analyzes the primitive component of the internal representation of a Deep Generation Network. 
* The research falls in-line with the work of explaining the generation process of a distributed neural network via the idea of AND-OR graph [1] which is exciting. It contributes to the foundation of understanding the image generation process as hierarchical compositional theory. 
* To model the OR interaction, the paper proposed to model it via Harsanyi's interaction theory. The paper provides formal writing about the theory that backs up the proposed algorithm. 

[1] Xing, X., Wu, T., Zhu, S.C., Wu, Y.N., 2020. Inducing hierarchical compositional model by sparsifying generator networks. CVPR 2020.
summary: The paper aims to provide an explanation framework for interpretation of the image synthesis process. The approach focuses on decomposing intermediate-layer features into "feature components," each responsible for generating specific image regions. By isolating these components, the method enables a new level of interpretability, where the image can be viewed as the superposition of "primitive regional patterns," each linked to distinct feature components.
weaknesses: * The author shows primarily the visual explanation of the generated image, however, it’s unclear how each “component” interacts with another component and how the components together compose the entire image in a hierarchy. It would be better if the authors can clearify what does mean by interaction?
* The authors also claim that the primitives interactions are minimized, however, Figure 4 shows that these components are heavily overlapped. Further clarification on the experiment part would be helpful.
* When explaining the feedforward process of a decision making network, it can make people understand how the network derived the conclusion. Although understanding unconditional image synthesis is interesting, it becomes much more impactful if the author could apply the proposed technique on image generation tasks that requires more precise control, such as image editing or image generation based on textual input, etc. 
* The presentation of Algorithm 1 depends on the understanding of Equation 9 and 10. It would become much clearer if the author can decouple Equation 9 / 10 and provide an algorithm by simply showing the essential operational building blocks. 
* A more quantitative evaluation protocol is needed to rigorously evaluate the effectiveness of the proposed method.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Is the method applicable to other architectures, or does it depend heavily on the specific properties of BigGAN?
soundness: 2
strengths: Fresh and Interesting Idea: The paper brings a new approach to explainable AI by breaking down image generation into basic regional patterns using OR interactions. 

Strong Theoretical Foundation: The use of Harsanyi interaction theory is thoughtful and well-developed. It’s clear the authors put serious effort into the math, giving the approach a solid backbone.

Promising Results: While the evaluation could be better, the visual experiments do show the method can isolate specific image regions effectively. There’s a real potential here for fine-tuning or controlling generative models more precisely.
summary: The paper introduces a new approach to explaining the internal workings of neural networks for image generation. The method breaks down intermediate-layer features into distinct components that correspond to specific image regions, aiming to interpret the model's internal representation as a combination of "primitive" regional patterns. By extending Harsanyi interaction theory to handle OR relationships between regions, the authors provide a novel way to link feature components with specific areas of the generated image. The experiments primarily use visual results from the BigGAN model to demonstrate the method's validity.

The paper does present a promising idea, but issues with the evaluation overshadow the strengths in problem formulation and theoretical discussion. Without more rigorous quantitative analysis, the contributions remain less convincing.
weaknesses: Despite I really like the first half of the paper, I would be somehow sceptical of the overall contribution for the following reasons.

1. Limited Quantitative Evaluation: ALL evaluations are visualisations. Quantitative metrics is indeed needed, like visual quality measurement, disentanglement testing. As it stands, we’re left guessing about the true effectiveness beyond what's visually apparent. Also, the absence of comparisons with other explainable methhods weakens the case for this approach.

2. Coarse Image Division: Dividing the images into a 6x6 grid feels too crude for explainability. There’s likely a lot of information lost with such a brutish approach. Why not use finer subdivisions or superpixels that adapt to the image content? That could make the whole decomposition more meaningful.

3. Unclear Visualization: The current visualizations fall short because the gray map overlays fail to highlight background changes, making it hard to see which regions are kept. Transparent overlays, heatmaps, or color-coding could make the results much clearer.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. Computation time - as mentioned, an analysis of the computational time about the number of regions/number of images is useful here and can tell how practical this approach is for finding disentangled directions.

2. Empirical analysis of the extracted components across a dataset - Do the extracted components create a pattern in the image space (e.g., some components are responsible for multiple regions while others are responsible for fewer regions)? How many components are needed to get different levels of disentanglement and fewer intersections between the corresponding regions (a graph that ablates it will be useful here)? Do more components correspond to the center of the image rather than the background?
soundness: 3
strengths: The paper presents an interesting approach for extending Harsanyi interaction for the image generative case. It provides a theory and builds an algorithm for matching the individual regions to disentangled representations. It finds and proves an interesting OR relationship that is implemented by the model via the disentangled features. The qualitative results show a clear correspondence between image regions and extracted features. This is a very interesting complementary result to the "AND" interaction (Ren et al., 2024a) that is implemented in discriminative models.
summary: The paper presents an approach for extracting disentangled components that correspond to specific image regions. The paper provides a method that ensures that each such component is a minimal component that exclusively generates a specific set of image regions. Thus, the paper shows that the different components constitute an OR relationship for generating different parts of the image.
weaknesses: 1. Presentation - My main concern is the clarity of the paper, which needs significantly more work. The paper is full of details and notations and is hard to follow. The presentation of the AND interaction is not directly related to this paper (e.g., Theorem 3.1/Corollary 3.2 are not mentioned anywhere after the preliminaries) and introduces more notation that is reintroduced later anyway (e.g., defining N twice). On the other hand, some of the notations are not defined (e.g., lambda in line 294). The introduction includes a lot of related work, and in general, there are a lot of repetitions in the text (e.g., between 4 and 4.1.). Given the current state of the paper, I couldn't verify all the proofs due to the difficulty of following the text. I recommend the authors significantly revise the paper to simplify a lot of the notations and remove redundant text. 

2. Usefulness - Given that the extraction procedure is computationally expensive (worked on 6x6 patches, but what about larger images), it is not clear how practical this method is for extracting the disentangled representations. An analysis of the computational cost, the number of images, and other parameters that can make this approach practical is needed. 

3. Interpretability - it is not clear that the disentangled representations are more interpretable to humans in any way. Although they are "minimal" features for regional generation, it does not mean that they are more interpretable -- they still have a complex functionality. Applying the same approach for being minimal about human-interpretable concepts (e.g., disentangling the minimal features that are responsible for foreground/background generation) will provide better interpretability and controllability to the generation process.