Interpreting Class Conditional GANs with Channel Awareness
Authors: Yingqing He
Zhiyi Zhang
Jiapeng Zhu
Yujun Shen
Qifeng Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: class-conditional GANs
representations
interpretation

Abstract: Understanding the mechanism of generative adversarial networks (GANs) helps us better use GANs for downstream applications. Existing efforts mainly target interpreting unconditional models, leaving it less explored how a conditional GAN learns to render images regarding various categories. This work fills in this gap by investigating how a class conditional generator unifies the synthesis of multiple classes. For this purpose, we dive into the widely used class-conditional batch normalization (CCBN), and observe that each feature channel is activated at varying degrees given different categorical embeddings. To describe such a phenomenon, we propose channel awareness, which quantitatively characterizes how a single channel contributes to the final synthesis. Extensive evaluations and analyses on the BigGAN model pre-trained on ImageNet reveal that only a subset of channels is primarily responsible for the generation of a particular category, similar categories (e.g., cat and dog) usually get related to some same channels, and some channels turn out to share information across all classes. For good measure, our algorithm enables several novel applications with conditional GANs. Concretely, we achieve (1) versatile image editing via simply altering a single channel and manage to (2) harmoniously hybridize two different classes. We further verify that the proposed channel awareness shows promising potential in (3) segmenting the synthesized image and (4) evaluating the category-wise synthesis performance.

Under review as a conference paper at ICLR 2023
INTERPRETING CLASS CONDITIONAL GANs WITH CHANNEL AWARENESS
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding the mechanism of generative adversarial networks (GANs) helps better use GANs for downstream upplications. Existing efforts mainly target interpreting unconditional models, leaving it less less explored how conditional GAN learns to render images regarding various categories. This work fills in this gap by investigating how class conditional generator unifies the synthesis of multiple classes. For this purpose, we dive into the widely used class-conditional batch normalization (CCBN), and observe that each feature channel is activated at varying degrees given different categorical embeddings. To describe such a phenomenon, propose channel awareness, which quantitatively characterizes how single channel contributes to the final synthesis. Extensive evaluations and analyses the BigGAN model pre-trained on ImageNet reveal that only a subset of channels is primarily responsible for the generation of particular category, similar categories (e.g., cat and dog) usually get related to some same channels, and some channels turn out to share information across all classes. For good measure, our algorithm enables several novel applications with conditional GANs. Concretely, we achieve (1) versatile image editing via simply altering a single channel and manage to (2) harmoniously hybridize two different classes. We further verify that the proposed channel awareness shows promising potential in (3) segmenting the synthesized image and (4) evaluating the category-wise synthesis performance. Code will be made publicly available.
INTRODUCTION
The past few years have witnessed the rapid advancement of generative adversarial networks (GANs) in image synthesis (Karras et al., 2021; Brock et al., 2019). Despite the wide range of applications powered by GANs, like image-to-image translation (Isola al., 2017), super- resolution (Chan et al., 2021; Menon et al., 2020), and image editing (Ling et al., 2021), it typically requires learning separate model for recent studies have confirmed that a well-trained GAN model naturally supports various downstream applications, benefiting from the rich knowledge learned in the training process (Bau et al., 2019; Shen et al., 2020). Therefore, to make sufficient use of GAN, it becomes crucial to explore and further exploit its internal knowledge.
task, which can be time and resources consuming. Some
Many attempts have been made to understand the generation mechanism of GANs. It is revealed that, produce fair synthesis, the generator is required render multi-level semanties, such as the overall attributes (e.g., the gender of face image) (Shen et al., 2020), the objects inside (e.g., the bed in bedroom image) (Bau al., 2019; Yang et al., 2020), the part-whole organization (e.g., the segmentation of the synthesis) (Zhang et al., 2021). 1), etc. However, existing efforts mainly focus on interpreting unconditional GANs, leaving conditional generation as a black box.
Compared with unconditional models, a class conditional model is more informative and efficient in that it unifies the synthesis of multiple categories, like animals, vehicles, and scenes (Brock et al., 2019). Figuring out how it manages the class information owns much great potential yet rarely explored. To fill in this gap, we take close look at the popular class-condee conditionn batch normalization (CCBN) (Brock et al., 2019), which is one of the core modules distinguishing conditional generators from unconditional ones. Concretely, CCBN learns category-speciff parameters to scale and shift the input features, such that the output features developed with different class embeddings can be

---

Under review as a conference paper at ICLR 2023
(c)
(I) Single-chinnel Imugee Editing
Inputs
Conditional Generator
Various Categories
(2) Category Hybridization
(3) Object Part Segmentation
w
(4) Synthesis Performance Evaluation
H
Channel Awareness
Lutbery ai1
Figure 1: Novel applications enabled by interpreting class conditional GANs. Given a conditional generator in (a), we propose channel awareness to quantify the contribution of each feature channel to the output image, as shown in (b), which reveals how the categorical information is handled by different channels. Red, green, and blue channels are primarily responsible for the synthesis of a particular category, while yellow ones are shared by all classes. (c) Such an interpretation facilitates range of applications, including single-channel image editing, category hybridization, fine- ana-ggai semantic segmentation, and category- synthesis performance evaluation. (Zoom in for better view.)
easily told apart from each other, eventually resulting in the synthesis of various categories. We notice from such process that, under the perspective of the ReLU activation (Nair Hinton, 2010) following CCBN, different feature channels present varying behaviors given different embeddings.
To quantify the aforementioned channel effect, we propose channel awareness that characterizes how single channel contributes the final synthesis. Through in-depth analyses the BigGAN (Brock et al., 2019) model pre-trained on ImageNet (Deng et al., 2009), we have the following key findings, which are also illustrated in Fig. lb. First, only portion of channels active in rendering images for a particular class while the remaining channels barely affect the generation. Second, more similar categories tend to share more relevant channels. For instance, channels regarding dog synthesis intersect with those of cats but disjoint from those of buses. Third, some channels highly response to the latent code instead of the class embedding and hence appear to deliver knowledge to all classes
Beyond model interpretation, our proposed channel awareness facilitates range of novel applica- tions with class conditional GANs, as shown in Fig. lc. First, after identifying the relevant channels through awareness ranking, we realize versatile image editing by simply altering a single feature channel (Sec. 5.1). Second, through mixing the channels that are related to two classes respectively, we achieve harmonious category hybridization (Sec. 5.2). Third, we verify that intermediate feature maps from the generator, after weighted by our channel awareness, fine-grained semantic segmentation (Sec. 5.3). Fourth, we empirically demonstrate the potential of channel awareness in evaluating the category-wise synthesis performance (Sec. 5.4).
be convincingly used for
RELATED WORK
Among various types of generative models, such as variational auto-encoder (VAEs) (Kingma Welling, 2013; Razavi et al., 2019), flow-based model (Kingma & Dhariwal, 2018), diffusion model (Ho et al., 2020; Dhariwal & Nichol, 2021), etc., GAN (Goodfellow et al., 2014) has received wide attention due to impressive performance on both unconditional synthesis (Karras et al., 2019; 2020; 2021) and conditional synthesis (Zhang al., 2019; Brock et al.. 2019; Sauer et al., 2022). Early studies on interpreting GANs (Bau et al., 2019; Shen et al., 2020) suggest that, a well-learned GAN generator has encoded rich knowledge that can be promising applied to various
2

---

Under review as a conference paper at ICLR 2023
downstream tasks, including attribute editing (Zhu al., 2022; Yang et al., 2020; Ling al., 2021), image processing (Pan et al., 2020; Zhu et al., 2020), super-resolution (Menon et al., 2020; Chan et al., 2021), image classification (Xu et al., 2021), semantic segmentation (Zhang et al., 2021; 1; Xu & Zheng, 2021), and visual alignment (Peebles et al., 2022). Existing interpretation approaches usually focus the relationship between the latent space and the image space (Shen et al., 2020; Zhu et al., 2021; Yang et al., 2020; Wu et al., 2021), hence commonly evaluated on unconditional models. Some attempts are made to also analyze class conditional models (Jahanian et al., 2020; Voynov & Babenko, 2020; Härkönen et al., 2020; Shen & Zhou, 2021), but they still target the latent space, leaving unclear how the generator leverages the categorical information. This work clearly differs from prior arts from the following aspects. (1) We inspect the conditional generator from the channel perspective, which aggregates the messages from both the lutent code and the class embedding. To our knowledge, this is the first attempt on understanding the function of embedding space in conditional generation. (2) We demonstrate the editability of altering a single channel of the conditional generator. Different from the single-channel editing in unconditional GANs (Wu et al., 2021), our approach identifies different relevant channels with respect to different categories in unsupervised manner. (3) We achieve fine- grained semantic segmentation by paying more attention to some "important" channels. Unlike the existing efforts on single-object generation (Zhang al., 2021; Xu & Zheng, 2021), our method does not require data-driven learning and can be robustly generalized to all classes. (4) We also enable some applications that are peculiar to conditional models, including the category hybridization and the category- -wise synthesis performance evaluation.
3 METHODOLOGY
In this section, we introduce the proposed channel awareness. Specifically, we re-examine the class- conditional batch normalization (CCBN), which is widely used in class conditional generation, and investigate how it helps the generator with the categorical information provided by the class embedding. It is noteworthy that our approach is fully unsupervised, without relying on any segmentation masks annotations.
3.1 PRELIMINARIES
Unlike unconditional GANs, where the generator takes the latent code as the only input, class conditional generator employs an additional embedding vector, e, to provide the categorical information. Accordingly, the generation process can be formulated as I = G(z, where I and G(.,-) are the output image and the generator, respectively. That way, given a different embedding, the generator is able produce images for that specific category.
There are many ways of integrating e into G(-,)), where the most popular one is to adopt the class- conditional batch normalization (CCBN) (Brock al., 2019). In particular, CCBN learns class- specific parameters to scale and shift the input feature maps as
y = > (z, e)) o x -x(x)
+ #((oocat(z,,e)
(I)
(x)
where x and y, both with shape C H x W, denote the input and output features. nx()) and a()) compute the mean and variance of tensor along the spatial dimensions (i.e., H and W). concat(-,") :) stands for the concatenation operation. 7(-) and (-)) outputs the C-dimensional
scale and bias by learning from both z and O represents the element-wise multiplication with broadcasting.
3.2 CHANNEL AWARENESS
Channel Probe. From Eq. (1), we can tell that both the latent code and the class embedding act on the generation through CCBN. In other words, their messages through the learning of 7(-) and ((-). Now, we take a look at how each single channel contributes to the synthesis. For a particular channel with index c, Eq. (1) can be simplified as
delivered to the feature channels
y x 1
3

---

Under review as a conference paper at ICLR 2023
CCBN y = Vexx + +Be
ReLU
Channel Probe
PC
Figure 2: Concept diagram of channel probe, from whose statisties develop the channel awareness. For a particular feature channel, it first transformed by class-conditional batch normalization (CCBN) with category-specific scale %6, and bias 3% then activated by ReLU to filter out negative neurons. Such process is reformulated as a combined operation (as shown at the bottom), where we define the channel probe as t CCBN and ReLU.
A This value reflects the integrated action of 72
where x and yf. both with shape 1 x H x W, denote the normalized input (i.e., subtracting mean and dividing by standard deviation) and the output. 7. and BE are scalars for the e-th channel. Here, stands for the embedding index, carrying the categorical information, and the effect of z is omitted for simplicity.
CCBN is usually followed by ReLU (Brock et al., 2019). Such an activation controls the information flow in that negative values y are cut off to zero. According to Eq. (2), is equivalent cutting off the values in x" that are smaller than
3
12
(3)
In this way, we manage to directly relate the output feature channel to its corresponding input channel, as shown in Fig. 2. The value 1. acts like channel probe since it measures the channel-wise activation of a particular synthesis. Generally, for a certain channel, it presents different behaviours (i.e., with different values) for different samples and different categories.
Category-oriented Channel Awareness. As discussed above, given well-trained generator with CCBN, Eo is strictly determined by the latent code, z, and the class embedding, Recall that the channel probe t is instance-awarg which fluctuates along with the synthesis varying. To get a more reliable understanding of the function of single feature channel, we derive channel awareness from the statistics of te. Oriented to the class embedding, e, we would like to eliminate the impacts caused by the randomness of To this end, we first sample number of latent codes with the embedding fixed, then calculate the & value for each synthesis and perform averaging. The resulting averaged score can be used evaluate how feature channel is responsible for the synthesis of particular category. Recall that lower Ex suggests that more information will be preserved along this channel. Therefore, channels with lower mean values are more likely to be used for generating the target class. We define category-oriented channel awareness as [t]-
Latent-oriented Channel Awareness. We further study the contribution of the latent code, z, to each channel. Similarly, we sample a collection of latent codes with a fixed embedding and calculate E.. Differently, this time we are interested in the variance instead of the mean, which gives us the latent-oriented channel awareness. A larger variance indicates that the randomness has a stronger influence on the synthesis regarding this channel. Hence, the latent-oriented channel awareness, Var(t reflects how sensitively chamnel reacts to the latent code with the given embedding.
4 EVALUATION AND ANALYSIS
4.1 CATEGORY-ORIENTED CHANNEL AWARENESS
Qualitative Evaluation. We measure the causal channel effect on image generation via masking the target channels zero during the forward process (Bau et al., 2019). Then compare the
4

---

Under review as a conference paper at ICLR 2023
Masking 5% channels
Masking 10% channels
Original
Lowest CCA
Random
Highest CCA
Lowest CCA
Random
Highest CCA
Difference Genertedd map
image
3
Figure 3: Qualitative results of masking channels with the lowest (the 1st column) and the highest (the 3rd column) category-oriented channel awareness (CCA) as well as masking randomly selected channels (the column, as baseline). Masking channels with the highest CCA results in the most noticeable class-relevant features lost, while masking channels with the lowest CCA causes only mild changes in the outputs. Results of more classes are shown in Appendix D.
generation results between masking channels with the highest and lowest channel awareness, and masking random channels, which is served as baseline experiment. The qualitative results are shown in Fig. which demonstrate that channels are not equally contributing to the generation of a particular class: Only partial channels (with the highest class awareness) primarily contribute to it, and some channels barely affect its generation (with the lowest class awareness).
Quantitative Evaluation. We further quantitatively verify the category-oriented channel awareness regarding every single channel in all 1,000 classes. We utilize a pre- trained classifier as an channel effect evaluator by assessing generated class images with and without certain channel. Specifically, we measure the classification score drop after masking the target channel, which we refer to as categorical score change. We then estimate the correlation coefficient between the categorical score change and the proposed category-oriented channel awareness. We show four classes with the highest correlation in Fig. 4. For all 1,000 classes, the averaged correlation is 0.503 for BigGAN and 0.603 for BigGAN-Deep respectively. Details of evaluation on BigGAN-deep can be found in Appendix B. Note that a high correlation indicates the effectiveness of our method.
Correlatin:
Correaton:
Consistion 0,904
Corrilatonn 0.322
Figure 4: Quantitative results of the correlation between category-orientee channel awareness (blue line) and cat- egorical score change (red line) of four classes. Top: The correlation between the two curves; Bottom: the corresponding class name. (The chan- nel indices are sorted by the aware- ness scores for better display).
Implementation details. We exploit the representative BigGAN (Brock et al., 2019) and BigGAN-ddeep condition- ally trained on the large-scale ImageNet (Deng al., 2009) as the target models, which can generate realistic images of 1,000 various categories 256 256 resolution. We use the Inception-v3 (Szegedy et al., 2016) model trained on ImageNet as the evaluation classifier, which is a widely used model for assessing generated images (Heusel et al., 2017). Then we calculate each channel's categorical score change before and after masking and take the average among 1,000 generated samples. The sampling process is without truncation for fair estimation. To reduce data noise, we smooth the raw data of the categorical score change (see those pink dots in Fig. 4) via the Savitzky-Golay filter (Luo et al., 2005) with a window length 51. The smoothed curves are shown in red line in Fig. 4. The channel awareness (including both category-oriented and latent-oriented) is computed by sampling 10,000 samples per class, and then taking the average (for category) and variance (for latent) among samples of the per-channel channel probe scores. Please note that, the awareness scores are fixed after computation on target generator, and can be directly used for the following analysis and applications.
Channel overlap estimation. After discovering high class-related channels, we wonder how distinct those channels are regarding different classes? Therefore, we estimate the channel overlap of two classes as [8,08,1, where S, and Sj represent the set of channel indices with top k highest awareness of class i and j, and k is the number of selected channels. Results of channel overlaps
5

---

Under review as a conference paper at ICLR 2023
Tarpet
Source
0.47
0.2
0.07
0.07
0.07
0.13
0
Loro Poitted
D.47
0.47
0.07
0.2
0.47
0.07
0.07
0
0.07
0
0.07
0.27
0.6
0.07
0.07
0.07
0.6
1
0.2
0.07
0
WhiaFu
Vater Anaa Car
0.13
0.07
0.07
0.07
0.07
0
0.6
0.67
0
0
0.6
1
0.73
0.67
0.73
000
(a)
(b)
Figure 5: Cross-class analysis on category-oriented channel awareness. (a) Ratio of overlapped channels between 9 different classes. select the top 100 channels with the highest category-oriented channel awareness for each class. A high ratio indicates more shared channels between the two classes. (b) Single-channel manipulation results on different classes. The semantics discovered from the source class (1st and 2nd columns) are applied to other target classes (3 7th columns). (Zoom in for better view.)
Table 1:
Cross-class analysis latent-oriented channel awareness. Results of the number of shared channels among 1,000 classes in different layers (layer - layer 11) in three types of channel selection criterion. For each class, select only 10 channels (k=10) and perform intersection among all classes. The calegory-oriented channel awareness is served as baseline. Channel overlapping results of more ks, well an extreme case of k=1 can be found in Appendix C.
Selection Criterion
LO
L1
L2
L3
L4
L5
L6
L7
L8
L9
L10
L11
highest awareness w.r.t category 0
0
0
0
0
0
0
lowest awareness w.e.. category 0
0
0
0
0
0
0
0
highest awareness w.r.f. latent 3 3
3
1
3
0
4
2
1
4
of 81 class pairs and = 100 are shown in Fig. 5a. This matrix indicates that visually similar classes(c.g. two different kinds of dogs) share more related channels, while distinct classes (c.g.. dogs with cars) have nearly zero channel overlap.
Single-channel manipulation. Furthermore, we discover that channels with the highest category- oriented channel awareness control meaningful semantic information for that class, thus simply manipulating one target channel can achieve semantic editing. We provide the single-channel manipulation results in Fig. 5b, where each manipulation is achieved by multiplying a scale factor (=3) on the target channel during the forward process of generator. However, channels discovered by the source class can only manipulate itself as well as similar classes, while hard to manipulate other classes which do not have such attributes, as shown in Fig. 5b.
4.2 LATENT-ORIENTED CHANNEL AWARENESS
Discovery of class-shared channels. Different from the category-oriented channel awareness, which is used for finding highly class-related channels, the latent-oriented channel awareness is used for finding class-shared channels (even shared among all the 1,000 classes). Results in Tab. demonstrate that, when only selecting 10 channels for each class of the highest latent-oriented channel awareness, then performing intersection on the sets of channel indices among all the classes, there still exists many channels left (the 3dd row of Tab. 1), while other two baseline experiments both have no overlapped channels (the 1s and 2nd row of Tab. 1). This implies that a class-conditional generator can learn unified representations provided to all classes. Besides, these class-shared channels can further enable semantic manipulation with highly similar effect on disparate classes. Visual editing results with respect to these channels provided in Fig. 6b.

---

Under review as a conference paper at ICLR 2023
Odiaid
Stetic Fit
Donas
Naw
Dark
Stel
Origial
Nilt
Syla
Lihe
Oxst
Caleegrryorizntted Attriuues
(b) Latent-ariceted Attributes
Figure 6: Versatile attributes discovered by the channel awareness. (a) Attributes discovered via the category- oriented channel awareness, which are highly relevant to a particular class. (b) Autributes discovered via the latent-oriented channel awareness, which are shared among various classes. Each editing result is achieved by manipulating single channel. (Zoom in for better view.) More results can be found in Appendix and Appendix F.
APPLICATIONS WITH CLASS CONDITIONAL GANS
5.1
IMAGE EDITING VIA SINGLE CHANNEL MANIPULATION
With our method, we can identify meaningful channels through awareness ranking, from which we discover that simply manipulate a single channel can achieve versatile effects for image attribute editing. Such a manipulation can be either multiplying or adding a constant of manipulation magnitude. Therefore, with channels that are relevant to particular class, we can discover and then manipulate various attributes of this class, such as fur, mouth, and nose for dogs and cats, explosion for volcanic. Results of image editing about ditegory-oriented attributes are shown Fig. 6a. Meanwhile, with channels that respond latent, we can edit meaningful class-shared semantics controlled by the latent vector, which therefore manipulate all kinds of classes in BigGAN. Results of image editing about latent-oriented attributes are shown in Fig. 6b.
5.2 CATEGORY HYBRIDIZATION VIA CHANNEL MIXING
In this section, we define a novel image editing task called category hybridization, which aims to synthesize realistic images while preserving meaningful attributes from more than categories. This task could benefit novel content creation and produce unnatural yet plausible combinations of two categories which do not exist in the training set. To achieve this, transplant the channels of the reference class to the corresponding channel positions in the feature of the input class, which we refer as channel mixing. To activate the transplanting channels, exploit the class embedding of the reference class for the generation of next layers. Results of category hybridization are shown in Fig. With channel mixing, we could harmoniously fuse attributes from two categories into one realistic image which cannot achieve by style mixing (i.e., simply mixes layer-wise class embeddings).
5.3 OBJECT PART SEGMENTATION VIA AWARENESS WEIGHTED PIXEL CLUSTERING
Existing works (Xu & Zheng, 2021; Zhang et al., 2021; Abdal al., 2021: Li et al., 2022; Tritrong et al., 2021; Ling et al., 2021) revealed that internal features in GANs can enable object part segmentation of generated images, which facilitates semantic annotation synthesis and local semantic editing tasks. Here, we present a simple unsupervised approaches for object part segmentation with the proposed channel awareness. Specifically, we collect the internal feature activations from multiple layers of the generator, then upsample each feature to be the same resolution as the target image H' x W' After upsampling, features from multiple layers could be concatenated along the channel dimension into total feature volume C" H' W where C' is the total number of channels. A pixel-wise feature vector in the volume, in size of C' x x 1, is considered one sample for performing the clustering. Then, simply perform K-Means
7

---

Under review as a conference paper at ICLR 2023
Input
Reference
Output
Output
Input
Reference
Output
Oulput
w/o channel mixing w/ channel mising
w/o Channel Mixing w/ Channel Mixing
Figure 7: Category hybridization by mixing the channels that are relevant to two categories. Compare with the conventional style mixing (Karras et al., 2019) (the 3rd column), our approach (the 4th column) better fuses the characteristics (including both shape and appearance) both classes. More results be found in Appendix G.
Clusters
5 Clusters
Features from all layers
Features from latter half layers
Fentures from all layers
Features from latter half layers
Input
without
with
without
with
without
with
without
with
Figure Object part segmentation with and without awareness weighting. We showcase that simple clustering on the feature space of the class-conditional GAN can obtain object part segmentations. Our channel awareness can be used for weighting channels and push them to be more class-related. Results of three different categories, two kinds of feature selecting layers, and two different amounts of clusters reveal that, all settings, the channel awareness helps the clustering to focus more the class-relevant representations, hence it recovers more detailed segmentations.
clustering on all pixel-wise feature vectors to segment the image into K parts. Segmentation results without and with channel awareness are shown in Fig. 8.
5.4 SYNTHESIS PERFORMANCE EVALUATION VIA TOTAL CHANNEL AWARENESS
Assumption and method. For synthesis performance evaluation, our key assumption is that if one class has the awareness of consistently activating certain channels for producing category- attributes while deactivating certain channels for suppressing irrelevant attributes, then the category will provide faithful generation results with high probability. For example, generating flower needs to suppress those channels responsible for eyes and nose (mainly used by other classes like animals) and activate those related petals. Thus, we measure the overall selection and suppression awareness as total channel awareness, by summing the absolute values of category-oriented channel awareness over all layers, which can denoted as follows: a. = E.ecr [Ez [t]]] Here C" stands for the number of total channels among all layers and is the given class.
Results. With total channel awareness, we empirically find that classes with the higher total channel awareness exhibit better quality yet low diversity of generated samples. Qualitative results of generated samples of classes with the highest and lowest total channel awareness are shown in Fig. 9. For quantitative verification, and other four widely used metrics for GANs among all 1k classes. Specifically, we estimate the quality via precision (Kynkäänniemi et al., 2019), diversity via recall (Kynkäänniemi et al., 2019)
calculate the correlation between total channel awareness
8

---

Under review as a conference paper at ICLR 2023
classes with highest total channel awareness
Manhole Cover
Rapeseed
Odometer
Website
Cypripedium parviflorum
Total Awaresioss S6R;
Total Channel Awaremess: 527;
Total Channel Awarenesc 512;
Total Channel Awareness: 510;
Total Channel Awaroness:
Precision: 0.98; Recall: 0.08;
Precisiom: 0.85;
Recall: 0.14;
Precasio:: 0.93;
Recall: 0.13;;
Precasion: 0.98;
Recall: 0,04;
Precision: 0.54, Recall: 0.05;
5 classes with lowest total channel awareness
Chainsaw
Stretcher
Reel
Plastic Bag
Rarrow
Total Awareness: 205;
Tocal Awarcness:: 199; Tocal Awareness: 199; Total Awareness:: 197;; Channel Awareness: 194;
Precision: 0.65; Recall: 0.20;
Precision: 0,70: Recall: 0.20;
Precision: 0.73;
Recall: 0.13;
Precision: 0.71; Recall: 0.20;
Preciston: 0.73; Recall: 0.15;
Figure 9: Qualitative results of evaluating category-wise synthesis performance, which is enabled by the total channel awareness. For each class, we show 6 generated samples, well as total channel awareness, precision, and recall on below. We observe that classes with high total awareness tend to have high synthesis quality yet low diversity.
Total Channel Awareness
Precision
Recall
MS-SSIM
FID
(a) Precision()), correlation-0.71
Recall(1). conreltionn-
(c) MS-SSIMML) comrllaionn-..9
FIXX correlation-
Figure 10: Quantitative results of evaluating category-wise synthesis performance, including
the correlation between our total channel awareness and the category-wise precision, recall, MS- SSIM, and FID. X-axis: Class indices sorted by the total awareness. Y-axis: Normalized total channel awareness and other four metrics. The correlations between total channel awareness and each other metric are shown at the bottom. t: higher is better; 4: lower is better. Note that precision evaluates quality, recall and MS-SSIM evaluates diversity and FID measures both quality and diversity.
and MS-SSIM (Odena et al., 2017), and general metric FID (Heusel al., 2017) for each class. Implementation details are in Appendix H. The quantitative results are shown in Fig. 10.
6 CONCLUSION AND DISCUSSION
This work takes the first step towards understanding the generation mechanism of conditional GANs from the channel perspective. Concretely, we propose simple yet effective channel awareness, by which we successfully identify that channels that response to the latent code and to different class embedding. Extensive analyses shed light on how a conditional GAN manages the categorical information with different channels. More importantly, our channel awareness enables four novel applications with class conditional generators, which are rarely explored by prior work. In particular, we achieve single-channel attribute editing as well as harmonious category hybridization in fully unsupervised manner. We also demonstrate the promising potential of our channel awareness in fine-grained semantie segmentation and category-wise synthesis performance evaluation.
Despite the appealing results, there are still some directions worth exploring. Recall that this work primarily targets the BigGAN generator (Brock et al., 2019), which learns the categorical information through CCBN with ReLU activation. Investigating other architectures, like the style modulation layer (Karras et al., 2019) with Leaky ReLU activation (Xu et al., 2015), can be one of the future works. Besides, approach mainly focuses interpreting and utilizing well-learned model for downstream tasks. How the insights provided in this work can inspire the design of a more powerful GAN model would be of more significance.

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement. To ensure the reproducibility of work, provide the pseudo- code of the core implementation of our channels awareness in Appendix A. In addition, we provide implementation details of quantitative evaluation and synthesis performance evaluation in Sec. 4.1 and Appendix H, respectively. Implementation details of other analyses and applications can be found in the corresponding paragraph. We will also make source code publicly available for all the core experiments.
REFERENCES
Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using StyleGAN. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. In Int. Conf. Learn. Represent., 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent. 2019.
Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. GLEAN: Generative latent bank for large-factor image super-resolution. In IEEE Conf. Comput. Vis. Puttern Recog., pp. 14245-14254, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Adv Neural Inform. Process. Syst., 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Adv. Neural Inform. Process. Syst., 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANSpace: Discovering interpretable GAN controls. In Adv. Neural Inform. Process. Syst., 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. Adv. Neural Inform. Process. Syst., 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adv. Neural Inform. Process. Syst., pp. 6840-6851, 2020.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-inage translation with conditional adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 1125-1134, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In Int. Conf. Learn. Represent., 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Compur. Vis. Pattern Recog., 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Adv. Neural Inform. Process. Syst., 2021.
Diederik Kingma and Max Welling. 1312:6 2013.
Auto-encoding variational bayes. arXiv preprint
10

---

Under review as a conference paper at ICLR 2023
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Adv. Neural Inform. Process. Syst., 2018.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Adv. Neural Inform. Process. Syst., 2019.
Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, Sanja Fidler, and Antonio Torralba. BigDatasetGAN: Synthesizing imagenet with pixel-wise annotations. arXiv preprint arXiv:2201.04668 2022.
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. EditGAN: High-precision semantic image editing. In Adv. Neural Inform. Process. Syst., 2021.
Jianwen Luo, Kui Ying, and Jing Bai. Savitzky-gobe smoothing and differentiation filter for even number data. In Signal processing, pp. 1429-1434, 2005.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PLUSE: Self- supervised photo upsampling via latent space exploration of generative models. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 2437-2445, 2020.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Int. Conf. Mach. Learn., 2010.
Augustus Odena, Christopher Olah, and Jonathon Shlens. auxiliary classifier GANs. In Int. Conf. Mach. Learn., 2017.
Conditional image synthesis with
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In Eur. Conf. Comput. Vis., 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Adv. Neural Inform. Process. Syst., 2019.
William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman. GAN-supervised dense visual alignment. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-V-E-2. In Adv. Neural Inform. Process. Syst., 2019.
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. arXiv preprint arXiv22202.00273. 2022.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in GANs. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. InterFaceGAN: Interpreting the disentangled face representation learned by GANs. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In IEEE Conf. Comput. Vis. Pattern Recog., 2016.
Nontawat Tritrong, Pitchaporn Rewatbowomwong,, and Supasorn Suwajanakorn. Repurposing GANs for one-shot semantic part segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In Int. Conf. Mach. Learn., 2020.
Zongze Wu, Dani Lischinski, and Eli Shechtman. StyleSpace analysis: Disentangled controls for StyleGANN image generation. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
11

---

Under review as a conference paper at ICLR 2023
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv: 2015.
Jianjin Xu and Changxi Zheng. Linear semantics in generative adversarial networks. In IEEE Conf.
Comput. Vis. Pattern Recog., 2021.
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. Generative hierarchical features from synthesizing images. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 4432-4442, 2021.
Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative representations for scene synthesis. Int. Comput. Vis., 2020.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. Int. Conf. Mach. Learn., pp. 7354-7363, 2019.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Caan-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. DatasetGAN: Efficient labeled data factory with minimal human effort. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain GAN inversion for real image editing. In Eur. Conf. Comput. Vis., 2020.
Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zhengjun Zha, Jingren Zhou, and Qifeng Chen. Low-rank subspaces in GANs. In Adv. Neural Inform. Process. Syst., 2021.
Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, and Qifeng Chen. Region-based semantic factorization in GANs, arXiv preprint arXiv:2202.096499 2022.
APPENDIX
A PSEUDO-CODE AND RUNNING TIME
Given a ire-trained class conditional model, category -rriented channel awareness and latent- oriented channel awareness, can be computed unsupervisedly and efficiently. The below PyTorch- style (Paszke al., 2019) pseudo-code provides the core implementation.
Import torch
compute compute_channel_ Sthaanne__wwaenese (scin_scsie, scbn_bias, e, aum_samples, z_dim): ""*oomputes the channel awareness with respect to a particular category.
Azge:
ocon_set sealeellee e: A PyTorch module (usual a ful cy-ooneede layer) used compute calegory 'scale' coe:Ticients. cebn_bias: & Pyferch module (usually a colly-coonected layer) used in CUSN compute
in
catagory-spesi.ik 'bias' coefficients. the enbedding vactor for the target category.
11
12
rum_eamples: Number random samples used for computation. z_dim: Dimension of the latent space.
13
14
Returns:
16
à txc-elemeat tuple, indicating the cetegory-ariented channel awaraness and latens-oriented channel sxareness, respectivoly.
17
18
***
19
Randon sample.
20
torch. randn (num_samples, =_dim) e.reshape (1, -1), fnum_samples, 1)
21
torch.cat(1e, el, dim=l)
23
channal probe,
shope [N, C].
25
probe cobn_biaalcode (code) cobn_soale (code) Category oriented channel awareness, with shape [C).
26
categosy_ewareness torch. dim=() Latent-orianted channel avareness, with shape (C).
28
latent_ewereness ver (probe, dime0)
30
return entegary_awarenee latent awareness
12

---

Under review as a conference paper at ICLR 2023
Running Time. Computing channel awareness is highly efficient. With NVIDIA Tesla V100 GPU, it costs 0.0017s to interpret all channels in the first layer of BigGAN regarding certain class. For all layers, the average running time is 0.0012s. The computing time for each layer is averaged using 200 classes.
B QUANTITATIVE EVALUATION ON BIGGAN-DEEP
This section provides additional evaluations for Sec. 4, including implementation details and results for evaluating the category-orienter channel awareness the BigGAA-Deepp model (Brock al., 2019). Unlike BigGAN, BigGAN-Deeep has twice the number of channels (e.g., 2,048 channels in layer 0), making the single-channel evaluation more time-consuming. Besides, zeroing out single channel makes an invisible change on the output image. Thus in this experiment, we zero out five channels for each intervention, and every five channels are selected along with the order of sorted channel indices by the category-oriented channel awareness. Results of top four results of four classes are shown Fig. A1. Here we scale the two curves into the range of 0-1 for better view.
Feather Correlation: 0,864
cream
Orange
African Chameleon
Correlation: 0.852
Correlation: 0.851
Correlation: 0.827
14
63
05
Figure A1: Quantitative evaluation of cafegory-oriented channel awareness on BigGAN-Deep with five-channel modulation. The correlation between awareness score (blue) and categorical score change (red) appears on top of each figure. X-axis: intervention order following the sorted awareness score. Y-axis: Scaled values of both lines. The average correlation of all 1,000 classes is 0.603.
C MORE ANALYSIS OF LATENT-ORIENTED CHANNEL AWARENESS
top-1
overtagped among 1,000 classes
1.000
12-
leyart
layezi
10
lapend
700
Jnyer
800
lapor9
laper10
layer1
300
200
2
3
100
10
13
34
16
10
Layer
Number of selecttd cbanzels
Figure A2: In-depth analyses on the number of shared channels among all classes. (a) We count how many classes share the top-1 channel at different layers, where the most popular channel at all layers serves over 450 classes from 1.000 classes in total. (b) Number of overlapped channels (X-axis) regarding different top-k values (Y-axis). Along with more channels incorporated for cach class, the number of overlapped channels increase. (Served additional results of Sec. 4.2 in the main paper.)
13

---

Under review as a conference paper at ICLR 2023
D
MORE QUALITATIVE RESULTS OF CHANNEL MASKING
Masking 5% channels
Masking 10% channels
Original
Lowest CCA
Random
Highest CCA
Lowest CCA
Random
Highest CCA
Generated
image
Difference
map
Generated image
Difference map
Generated image
Difference map
Generated image
Difference
map
Generated
map image
Difference
Figure A3: More results of channel masking for quanlitative evaluation (Fig. the main paper), which show comparisons among masking lowest (left) and highest (right) crtegtry-oriented channel awareness (CCA) as well as masking randomly selected channels (middle). The four class shown here are mushroom, castle, komondor, house finch, and alp.
14

---

Under review as a conference paper at ICLR 2023
E
MORE RESULTS OF CATEGORY-ORIENTED ATTRIBUTE EDITING
Front board
Snow
S Ghowploo
Head Pose
Head Size
JA
Brambling
Leg Length
Beak Length
Redshank
Shape
Ground
Triumphal Arch
Figure A4: dategory-oriented attribute editing (Sec. 5.1 in the main paper). Given a target class, our category-oriented channel awareness can unsupervisedly identify channels responsible for the class-oriented attributes, based on which image editing can be achieved via altering single channel. For each group of images, the middle one is input, while the left and the right one are manipulating the single channel along the positive and the negative direction. Corresponding class names are annotated on the left.
15

---

Under review as a conference paper at ICLR 2023
F
MORE RESULTS OF LATENT-ORIENTED ATTRIBUTE EDITING
Original
Zoom
Plants
Night
Style
Color
Great Grey Owl
Bee Eater
Alp
Castle
Sand Electric Locomotive
Figure A5: Latent-oriented attribute editing (Sec. 5.1 in the main paper). Our latent-oriented
channel awareness unsupervisedly identifies channels that deliver information to all classes. Thus altering one channel can edit images from multiple classes. Corresponding class names are annotated the left.
16

---

Under review as a conference paper at ICLR 2023
G
MORE RESULTS OF CATEGORY HYBRIDIZATION
Input
Reference
Output
Input
Reference
Output
Figure A6: Category hybridization by mixing the channels that are relevant to two different categories (Sec. 5.2 in the main paper). For each group, the first two columns present the original syntheses, while the third column shows the hybridization result, which successfully fuses the characteristics (including both shape and appearance) of both classes.
17

---

Under review as a conference paper at ICLR 2023
H IMPLEMENTATION DETAILS OF SYNTHESIS PERFORMANCE EVALUATION
This section gives the implementation details for category-wise synthesis performance evaluation via total channel awareness (Sec. 5.4 in the main paper).
Data preparation. For precision, recall, and FID, which need to compare fake images with real images during estimation, we firstly pre-process real images from ImageNet (Deng al., 2009) following the procedure in BigGAN training which is cropping and resizing to 256 x 256. The number of real images in ImageNet around 1.300 for each class. We randomly sample all generated images without truncation for fairly measuring the synthesis performance.
Precision and Recall. Precision (Kynkäänniemi al., 2019) is used for measuring sample quality by retrieving fake images and checking whether it locates within the manifold of real images. Recall (Kynkäänniemi et al., 2019) is used to measure sample diversity by retrieving real images and checking whether it is within the manifold formed by fake images. The ranges of these two metries are both between zero to one. higher value indicates better performance. We calculate precision and recall with the number of fake images equal to the number of real images of each class followed the suggested setting (Kynkäänniemi et al., 2019). The feature extraction network the pre-trained VGG-16. We set the neighborhood size k equal 3, which is more robust choice(knnkännniem et al., 2019).
MS-SSIM. We calculate the similarity between fake image pairs via the image similarity metrics MS-SSIM following (Odena et al., 2017). Lower image similarity indicates better diversity. For each class, randomly sample 100 fake images to construct 10,000 image pairs for calculating similarity, then an averaged similarity score for the class can be obtained as the final result for the intra-class diversity.
Fréchet Inception Distance (FID). FID is a general metric that considers both diversity and quality by directly estimating the distance between the feature distribution of fake and real images. We calculate the FID with 50,000 generated images for each class. Then we exploit a pre-trained Inception-v3 the feature extractor for estimating the feature distance between real and fake images.
18


Reviews:
[{"review_id": "r4-TN_cX0VI", "reviewer": "Reviewer_dhhh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the conditioning mechanism of BigGANs, i.e., Class-Conditional Batch Normalization (CCBN), and proposes \"channel awareness\" in order to quantify the impact of each feature channel in the the final synthesis. Using channel awareness, the paper asserts that a) only a subset of channels is primarily responsible for the generation of a particular category, b) that similar categories (such as cats and dogs) share some channels (they are activated jointly), and c) that some channels are \"global\" in the generation  process of all classes. Finally, the paper presents results on a number of downstream tasks (image editing, hybridization between two classes, image segmentation).", "summary_of_the_review": "The main reason I cannot suggest acceptance of the paper at this time is the limited applicability of the proposed method, since it only studies CCBN (BigGAN), but not other class-conditional generators, mainly StyleGAN-XL.", "strength_and_weaknesses": "Strengths\n---\n\nThe paper presents a very simple method for quantifying the impact of each feature channel in the generation process of a BigGAN (by studying its conditioning mechanism, i.e., CCBN). Its simplicity is its main strength, while the provided applications are useful and potentially useful.\n\n\n\nWeaknesses\n---\n\nMy main point of criticism concerns the generality of the proposed method in conditional GANs that do not use CCBN as a conditioning mechanism. The paper investigates the use of CCNB in BigGANs, but leaves without any discussion another recent and powerful conditional GAN, namely StyleGAN-XL [1]. To the best of my knowledge, BigGAN is the only relatively recent conditional generator to use CCBN as a conditioning mechanism. This renders the paper BigGAN-specific, and potentially reduces its applicability in other class conditional architectures, mainly StyleGAN-XL that gives significantly better results, as far as I know.\n\nMoreover, another weakness of the paper concerns its novelty in the reported applications. The paper states that the proposed method enables several novel applications (such as image editing and segmentation) with conditional GANs, but this is not accurate. Conditional GANs have been studied in terms of finding interpretable directions (e.g., [2,3]), as well as in the applications of image editing [2,3,4] and segmanation [3,5].\n\nFinally, whilst the above applications do seem interesting and the provided method helps towards, the \"key findings\" of the paper, as stated in the manuscript (and listed above), are not surprising or totally new.\n\n\nReferences\n\n[1] Sauer, Axel, Katja Schwarz, and Andreas Geiger. \"StyleGAN-XL: Scaling stylegan to large diverse datasets.\" ACM SIGGRAPH 2022 Conference Proceedings. 2022.\n[2] Tzelepis, Christos, et al.. \"WarpedGANSpace: Finding non-linear RBF paths in GAN latent space.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n[3] Voynov, Andrey, and Artem Babenko. \"Unsupervised discovery of interpretable directions in the gan latent space.\" International conference on machine learning. PMLR, 2020.\n[4] Oldfield, James, et al. \"Tensor Component Analysis for Interpreting the Latent Space of GANs.\" arXiv preprint arXiv:2111.11736 (2021).\n[5] Andrey Voynov, et al. Big GANs are watching you: Towards unsupervised object segmentation with off-the-shelf generative models. arXiv preprint arXiv:2006.04988, 2020", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eePrZRDT4aI", "reviewer": "Reviewer_vDMD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work tries to understand what a particular generative adversarial network (GAN) learns and how we can use a pretrained GAN. Specifically, this model mainly analysis one of the class-conditional GANs named BigGAN with a channel probing technique. Although its findings and application look attractive at first, it is hard to estimate the true effect before comparing them with the previous work or other models that meet the class-conditioning + ReLU requirement.", "summary_of_the_review": "This model tries to understand what class-conditional GAN learns and how we can play on top of the pretrained models. Although the results and the applications are fun to see, I would like to see a more direct comparison with other methods and other GAN models.", "strength_and_weaknesses": "#### Strength\n- This paper is well-written and easy to follow. \n- Carefully drawn figures help the reader catch what this paper is about and the implications of their findings. \n- Application in various ways, including segmentation examples, sounds interesting to explore.\n\n#### Weakness\n- Although a few methods are mentioned in related work, estimating the performance gap between this method and the rest of the literature is tough. \n- Since this work is mainly based on BigGAN (+ although supplementary presents few details about BigGAN-Deep), it is hard to catch whether this method generally works for other class-conditional GANs beyond BigGAN. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3nIYr_oyN-q", "reviewer": "Reviewer_EESf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes channel awareness based on class-conditional batch normalization (CCBN) to study how a single channel contributes to the final synthesis. Specifically, they conduct experiments on BigGAN pre-trained on ImageNet. The authors show that only a subset of channels primarily contributes to a specific category and some channels are shared by similar categories or all classes. Finally, they show the applications by editing specific channels for image editing, category hybridization, segmentation and image synthesis evaluation. \n", "summary_of_the_review": "The presentation of this paper is clear. My concerns are mainly about the novelty and the quality mentioned in *Strength And Weaknesses. Specifically, the authors conduct experiments only on BigGAN on ImageNet, which is not convincing enough. The authors should also provide comparisons with some SOTA approaches (e.g., StyleGAN). If all the concerns mentioned in Weaknesses are addressed, I will raise my rating. ", "strength_and_weaknesses": "### Strengths \nThis paper is well-written and it is easy to follow. \nThe application of category hybridization, segmentation and synthesis evaluation is interesting. \nThe authors promise open-source code. \n\n\n### Weaknesses \n\n**Clarity**\n\nThere are many missing details. For example, \n1. the authors forget to introduce the images that appeared in Figure 1(b). What do they mean? What are the differences between red, green, and blue bounding boxes? \n2. The authors labeled the edited images in Figure 6. Are they labeled manually or by pre-trained classifiers? \n3. The authors should provide more details about how to achieve category hybridization and image editing in Section 5. \n4. In Figure 7, the authors mixed similar categories, e.g., monkey and dog are both animals, How about mixing the monkey and a car? Or mixing a tree and a dog? \n\n**Novelty** \n\n1. There are some missing references that have studied the relationships between the channels and the categories (i.e., semantics), and they also achieved image editing by editing channels [A]. The authors should discuss and compare with them. \n\n[A] Learning semantic-aware normalization for generative adversarial networks. H Zheng, J Fu, Y Zeng, J Luo, ZJ Zha. Advances in Neural Information Processing Systems 33, 21853-21864. \n\n**Quality** \n\n1. The authors study channel awareness only on one model (i.e., BigGAN) on one dataset (i.e., ImageNet), which limits the credibility of channel awareness. We suggest the authors study it on more GAN models and more datasets to verify it. \n2. In Section 3, the authors calculate category-oriented channel awareness on sampled images. How many samples? 1,000 or 10,000 samples mentioned in the implementation details are not convincing enough since ImageNet has 14 million images. \n3. In Figure 7, the authors claimed they perform better style mixing than StyleGAN, however, they didn’t provide the results of StyleGAN for comparisons. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject