Improved Image Generation via Sparsity
Authors: Roy Ganz
Michael Elad
Venue: ICLR 2022 Submitted (2022)
Keywords: Sparse Modeling
Image Generation

Abstract: The interest of the deep learning community in image synthesis has grown massively in recent years. Nowadays, deep generative methods, and especially Generative Adversarial Networks (GANs),  are leading to state-of-the-art performance, capable of synthesizing images that appear realistic. While the efforts for improving the quality of the generated images are extensive, most attempts still consider the generator part as an uncorroborated ``black-box''. In this paper, we aim to provide a better understanding and design of the image generation process. We interpret existing generators as implicitly relying on sparsity-inspired models. More specifically, we show that generators can be viewed as manifestations of the Convolutional Sparse Coding (CSC) and its Multi-Layered version (ML-CSC) synthesis processes. We leverage this observation by explicitly enforcing a sparsifying regularization on appropriately chosen activation layers in the generator, and demonstrate that this leads to improved image synthesis. Furthermore, we show that the same rationale and benefits apply to generators serving inverse problems, demonstrated on the Deep Image Prior (DIP) method.

Under review as a conference paper at ICLR 2022
IMPROVED IMAGE GENERATION VIA SPARSITY
Anonymous authors
Paper under double-blind review
ABSTRACT
The interest of the deep learning community in image synthesis has grown sively in recent years. Nowadays, deep generative methods, and especially Gener- ative Adversarial Networks (GANs). are leading to state-of-the-art performance, capable of synthesizing images that appear realistic. While the efforts for im- proving the quality of the generated images extensive, most attempts still con- sider the generator part as an uncorroborated "black-box". In this paper, we aim provide better understanding and design of the image generation process. We interpret existing generators as implicitly relying on sparsity-inspired mod- els. More specifically, we show that generators can be viewed as manifestations of the Convolutional Sparse Coding (CSC) and its Multi-Layered version (ML- CSC) synthesis processes. We leverage this observation by explicitly enforcing a sparsifying regularization appropriately chosen activation layers in the gener- ator, and demonstrate that this leads to improved image synthesis. Furthermore. we show that the same rationale and benefits apply to generators serving inverse problems, demonstrated on the Deep Image Prior (DIP) method.
INTRODUCTION
The use of Generative Adversarial Networks (GANs) for image synthesis is one of the most fasci- nating outcomes of the emerging deep learning era, c.g.. Goodfellow al. (2014); Radford et al. (203)): Zbu et al. (2017); Ledig et al. (2017); Karras et al. (018). GAN IS a machine-leariin apparatus built of two competing neural networks, a generator and a discriminator. The generator produces samples, while its adversary, the discriminator, attempts to distinguish between them and training ones. The resolution and quality of images produced by these intriguing game-theoretic machines have improved significantly in recent years, e.g.. Karras et al. (2019): Brock et al. (2019).
Although leading to impressive results, GANs are difficult to train and prone to undesired phenom- ena, such mode collapse, failure converge and vanishing gradients (Thanh- Tung & Tran 2020).
Much of the research in this field has been focusing on mitigating the above difficulties and on sta- bilizing the training process. Most of this progress has been achieved by heuristically modifying the architectures of the generator and the discriminator, and by exposing new and better behaved training losses (Salimans et al.) 2016 Arjovsky et al. 2017; Gulrajani et al.) [017] Mao al. (017)) As such, while GANs in general have been extensively studied and redesigned, the generator itself still operates as "black-box" of unjustified architecture nor meaning.
Image generators within GANs are deep neural networks, comprised of a sequence of convolutional- based blocks. Distinct from other types of deep learning architectures, generators learn a transfer function from a (lypically simple) multivariate source distribution to a target one. To do so, each of the generator's blocks performs a transitional step from its input to a latent representation. The desired overall mapping is thus achieved by chain of these incremental steps. As mentioned above, previous work has considered the generator's mapping as an end-to-ent process, with hardly any [ to propose model that provides a possible explanation to the true nature of it.
Motivated by the sparse modeling literature (Elad, 2010), we aim to propose novel interpretation that sheds some light on the architecture of image generators and provides meaningful and effective regularization to such networks. We interpret generators as implicitly relying on sparse models in general, and on the Convolutional Sparse Coding (CSC) and its Multi-Layered (ML-CSC) version in
A notable exception to this rule is the work reported in Mahdizadehaghdam et al. 2019 which will be described and tied our contribution in Section

---

Under review as a conference paper at ICLR 2022
particular (Szlam al. 2010; Bristow et al., 2013 Chalasani al. 2013 Grosse et al. 2012 Heide 2018 This observation provides possible explanation for the generator's intermediate mappings. We harness this insight by proposing a general model-basee approach to regularize image generators, which can be easily applied to various architectures.
et al. 2015; Papyan et 201 2017b] Papyan et al. 2016: Sulam et al. 2018 Sulam et al.
More specifically, we refer to the generator as a machine that synthesizes a sparse representation, followed by multiplication by a dictionary. Accordingly, we propose several versions of sparsity- insipred regularizations, posed both as constraints or penalties, injected into the GAN training pro- cess. Under the CSC interpretation. the last layer of the generator consists of the learned synthesis dictionary, built as set of filters. ML-CSC further assumes that a sequence of such dictionaries multiplying the representation vector for generating the output image. We validate our proposed view by conducting extensive experiments on variety of well-known GAN architectures, from rel- atively simple to up-to-date ones. Experimenting with a wide range of such networks demonstrates the versatility and generality of our approach. All the conducted tests show performance improve- ment (assessed via Fréchet Inception Distance and Inception Score), achieved by our simple yet effective method.
We further extend our contribution by demonstrating that the same rationale and improvement are valid for other image generator neural networks. More specifically, apply the proposed regular- izations to the Deep Image Prior (DIP) algorithm (Ulyanov et al. 2018), which utilizes an image generator as implicit image prior for solving various inverse problems. We examine the effects of our approach, and show that in restoration tasks as well, performance improvement is attained.
Adopting different perspective towards our contribution, image generators are highly expressive overparametrized neural networks (Balaji et al., (021) that solve what seems to be a severely ill- posed problem. Using overparametrized models with a large capacity might lead a poor gen- eralization and inferior results, and especially so when trained with relatively small datasets. As such, properly chosen regularization applied to the generator may lead benefits in both stability and overall performance. Indeed, driven by similar reasons, regularization has been adopted for the discriminator, being overparametrized as well, e-g.. Wasserstein and Spectral Normalization GANs (Arjovsky et al. 2017, Gulrajani et al. 2017 Miyato et al.) 2018 Zhang al. 2019). Our work of- fers to close this gap by introducing a meaningful such regularization for generators, thereby further stabilizing and boosting the synthesis process.
To summarize, this work focuses on deep neural networks that serve as image generators, and its key contributions are the following:
We propose a novel sparse modeling interpretation of image generators for better under- standing them.
We leverage this interpretation for proposing appropriate sparsity-inspired regularizations to generators.
We demonstrate the validity of our view by showing a performance boost in image synthesis with various GAN architectures.
We show that similar performance improvement is applicable to image generators serving inverse problems, in the context of DIP.
2 BACKGROUND
2.1 SPARSE MODELING
Sparse Modeling has been proven be highly effective in signal and image processing applications.
e.g., DDbov et al. 2007
Bruckstein et al. 2009 Yang et al.) 2010 Elad 2010; Dong et al., 2011 2013 Mairal et al. 2014). This model assumes an underlying linear generative model, according to RN can be described as linear combination of few columns from a dictionary E R NxAl i.e. x = DT, where a sparse vector. The columns of D are referred to as atoms and they may form overcomplete set, i.c., M > N. Retrieval of sparse vector I', corresponding to given a signal x and a dictionary D. is referred to as sparse coding, formulated as
which, signal
min || s.t. Dr x,
(1)
2

---

Under review as a conference paper at ICLR 2022
where || counts the non-zeros in Г. Thus, synthesizing signal according this model is done by generating a sparse representation vector and multiplying it by D.
2.2 CONVOLUTIONAL SPARSE CODING MODEL
When handling images, sparse modeling has been usually deployed on small overlapping patches due to the complexity of handling general content dictionary. The Convolutional Sparse Coding
(CSC) model (Szlam et al. 2010; Grosse al. 2012, Bristow et al. 2013 Chalasani et al. 2013
Heide et al. 2015 Papyan al.) 2017b) is a global variant of the above model, which bypasses the
need to operate on patches. The CSC's dictionary is structured, being concatenation of banded
circulant matrices containing small support filters, each appearing in all possible shifts. Thus, D E
IR Nxm where N is the size of the signal x and m is the number of filters, each of length n < N.
According to this model, signal
be expressed by
Dr, described above (see Figure T.
The CSC has demonstrated superb performance in image processing tasks, such as denoising, sepa- ration, fusion, and super-resolution (Gu et al.] 2015 Liu al. [0166] Papyan al. [201a]] Simon &
Elad 2019 Zisselman et al. 2019). A recent work (Papyan et al. (017b) has proposed a theoretical analysis of this model, exposing the need for redefinition of the sparsity measure to be used r, in order to account for local use of atoms instead of globally counting non-zeros. We shall get back
this in the next section, when using the CSC model.
Tg
T2x
T2s
Figure 1: CSC visualization: signal x generated by superposition of few atoms from a convo- lutional dictionary D. Each entry of T corresponds to certain shift of a limited support filter.
81
(4g
Y
Ya
Q
YK
Y2
Yk
Yc
Ye
ag
Figure 2: 2D CSC visualization: I is HxWxC tensor, composed of HxW "needles". denoted as y, each of size 1xlxC. Since I is sparse, the bulk of its entries are zeros. A given y at location patch located in the corresponding location in the output image æ. Each
i in I' contributes non-zero entry in y defines the coefficient for the filter a; in a superposition that creates a patch. Note that the patches created by nearby needles may overlap, depending on the stride and the spatial size of the atoms in D.
3

---

Under review as a conference paper at ICLR 2022
2.3 MULTI-LAYER CONVOLUTIONAL SPARSE CODING (ML-CSC)
ML-CSC (Papyan et al. 2016 Sulam et al. 2018 Sulam et al. 2018) is an extension of the CSC, which generates cascade of sparse representations {T. corresponding CSC dictionaries (D. M
This model assumes that signal E EN can be represented as
I = D s.t. [[T lla S A1,
T1 D2T2 s.t. [T2loo X2,
TT-1 = DrTL s.t. [T S AL.
where {1: are sparsity thresholds. Thus, while the first equation perfectly aligns with the regular CSC model, the additional equations add further structure by suggesting that each sparse representa- tion vector is by itself a CSC signal. Note that by substituting the above equations, signal also be described as a D1 D,I,, = Des S i S L, with intermediate sparse representations.
IMPROVED IMAGE SYNTHESIS VIA GANS
Deep generative models are neural network-based architectures that synthesize signals, such that their output follows the probabilistic distribution of given data source. In computer vision, these generators are used successfully for synthesizing natural images, by training on massive datasets of photos. To this end, they map given source distribution P. (typically a simple and canonical multivariate Gaussian) to a data distribution of interest Pa,
((=) Tgen s.1. P2 and Tgen Pa-
Due to the complexity of image synthesis, the above mapping function is usually modeled by highly expressive reed-forward deep convolutional neural network, consisting of several consecutive layers. In its simplest and most common form, an image generator with K layers can be rephrased as forward CNN of the form
GK (-)( Jgen
where G; represents the 1hh layer of the generative model, applying convolutions, normalizations and a non-linearity (typically ReLU). Thus, the overall mapping is attained in tandem of K transitional mappings. Despite the incremental nature of the deep image generator, it is treated and trained in an "end-to-end" mode, without understanding the purpose of the inner activations, nor enforcing a specific structure properties on the intermediate activations.
In this work, we interpret existing image generators as implicitly relying on sparse modeling, and propose novel model-based approach to describe and improve the synthesis process of such architectures. Since these generators are highly expressive and overparametrized, confining them can lead to superior results and facilitate the training process. To do so, recall the convolutional sparse coding (CSC) and its Multi-Layer version (ML-CSC) models, as discussed in Section 2
According the CSC model, image can be represented sentation vector I' by a convolutional dictionary D. i.c. = DI'. The ML-CSC further assumes that is achieved by multiplication of L dictionaries: Defs = D,D2 Dr. Note, however, that although both are generative models, there is a substantial gap between their description and the process described in Equation P Whereas the CSC synthesis starts with sparse representation vector, the image generation setup in Equation 2 bridge this gap, we propose to interpret image generators as performing two consecutive tasks:
multiplication of sparse repre-
starts with a dense latent random vector 2. To
1. G Map the input vector sparse latent vector I (done by the generator's first K - layers).
2. G Multiply I' by a convolutional dictionary D, i.e., DT ~ Ps (performed by the genera- tor's Tthh layer that learns a convolutional dictionary for this purpose).
The second task exactly the CSC synthesis process. More specifically, if D is a single convolu- tional dictionary, the above formulation is equivalent to the CSC, whereas if D is comprised of two
4

---

Under review as a conference paper at ICLR 2022
more dictionaries, this becomes the ML version it. By splitting the generation process into two parts, we identify the role of I' as the sparse representation in a (ML-)CSC-based model. This way, the image synthesis process can be described
(²) (2)=-1 - -
s.t. P2. is sparse, Ps.
namely, the end-to-end mapping can be separated into two consecutive mappings, where each may be easier to learn. This interpretation is further shown in Figure In the ML-CSC context, G! is comprised of two or more global convolutional dictionaries.
6C = (o,(G,))))
( 6 =x
12]-100
8A.256
16,06,
End-to-end prespective Proposed prespective
G(+)
Figure 3: Our interpretation (green) of generators as handling two separate sub-tasks G produces sparse representation vector and G' maps it into image. The depicted architecture contains blocks and the stated tensors' dimensions are typical for synthesizing 32 x 32 x images.
We have analyzed the sparsity of I' in regular adversarial training and found out that it is not suffi- cient. For the purpose of designing more compatible generators with both the CSC and the ML-CSC models, we encourage G map 2 to truly sparse representation r. To this end, we propose few explicit techniques, serving as regularization forces applied in the training phase:
(1) L1 regularization: Adding an L1 based penalty on the representations to the overall loss of the image generator.
Eliminating small non-zero entries in I to satisfy predefined sparsity constraint: IITll S A. Namely, the amount of non-zero entries in I should be less or equal to X. This can be viewed as a projection to a constraint-satiiffingg tensor, obtained by zeroing the smallest absolute values of the representation.
(2) Lo constraint:
(3) Lo,co inspired constraint: This pseudo-norm is based on new sparsity measure related to the CSC Papyan et al. (2017b). [|Tilo, is the maximal number of non-zero coefficients affecting any pixel in the image Thus, foreing | À restricts the number of local atoms to X. While this constraint is theoretically justified in the context of CSC, projection onto it is known to be challenging (Plaut Giryes] 2019). To approximate in a computationally plausible manner, sneedle"-baseee sparsity measure (Papyan et al. 2017a Zisselman et al. 2019). In the general 2D CSC case, I is 3D tensor, of size HxWxC. where H and W define the image size to be synthesized, and the number of filters. We define a needle as 1x1xC tensor, contained in l'. Hence, I' contains HXW needles, each contributes to patch of pixels in a corresponding position in the output image. In this configuration, every pixel is affected by several adjacent needles. This setup is described in Figure 2 We propose to limit the amount of non-zero entries in every such needle. To this end, for given representation tensor T, we zero the smallest absolute values in each needle of the representation to satisfy the relaxed constraint.
Although the above three options all promote sparsity T, they are substantially different, as indeed will become evident in our experiments. While L2 and Lp consider global sparsity, the Lo,c0 forces a local balance in the use of the atoms, i.e., limiting the local density in the representation I. As for the difference between Lo and L,, the first is deployed as a penalty. Since both L and Ly are constraints, we implemented them as projection layers which map input tensor constraint-satisee illustrates the effect of applying the proposed regularization techniques and the differences between them in inducing sparsity.
constraint, while the latter is used
one, by zeroing its smallest entries. Figure
5

---

shall see in the next section, theme extends to other generators Under review as a conference paper at ICLR 2022
FID
25
Sparsity
37%
37%
Epoc1
(a) Sparsity levels and FID comparison
based
fg
based
Agee
based
0:5
D34
632
632
433
838
418
111
038
606
204
(b) Sparsification methods' comparison
Figure 4: An illustrative experiment on the effects of applying method during the training of simple DCGAN (Radford et al. 2015) on CIFAR-10 10 dataset. (a): Sparsity levels and FID compari- son between non-regularized, L1 based regularization and constraint beastraint-baeed regularization (Lo and Lu,ce) on a single-layer CSC and Lo,c0 constraint applied on the ML-CSC. The sparsity level is the percentage of non-zeros in the representation r. As can be seen, promoting sparsity leads to im- proved performance. (b): A spatial sparsity distribution comparison of r, obtained by the different sparsifying techniques. Each pixel in the above figure represents the mean sparsity attained in the corresponding needle of the sparse tensor I. As demonstrated above, Lo and L1 lead to global sparsity which is imbalanced locally, while Lo forces such balance. Since most of the objects in CIFAR- 10 are centered, applying Lo or L, regularizations leads to denser needles at the center.
We should note that connection between sparse modeling and GANs' generators has been already proposed in /Maddizadehaghdam al.) (2019). In their work, they introduced sparsity GANs for boosting performance, and achieved their goal by designing specific architecture that utilizes patch-based modeling and a pre-trained dictionary. While inspired by their view, our work differs substantially,
propose more general strategy that leverages more advanced models (CSC and ML-CSC), does not require a pre- pre-training of any sort, and can be easily applied to various existing GAN architectures. Indeed, that go beyond GANs.
To summarize the proposed theme, we argue that learning direct mapping from random noise natural images' distribution an extremely hard task that can be mitigated by enforcing a model that provides meaningful regularization. As in every model, the CSC introduces specific structured assumptions about the data. Since this model has been shown to be highly compatible with natural images, we believe that these assumptions empirically hold, and hence, the suggested approach will lead to better results. To verify the above statement, we conduct experiments in various types of tasks, compare and dissect the results of regularized and non-ragularized generators.
4 IMPROVED SOLUTION OF INVERSE PROBLEMS
Inverse problems involve the process of recovering data from degraded observations. There are various well-known such problems in image processing, e.g., denoising, deblurring, inpainting and super-resolution, all of which are ill-posed, as they may have unique solution. In the field of image processing, solving inverse problems is one the most important and studied topics, and there are thousands of papers dedicated to the derivation of various algorithms and strategies for handling such tasks. In this paper, we focus on one specific and fascinating method, DIP (Ulyanov al. 2018), which uses image generator for handling inverse problems.

---

Under review as a conference paper at ICLR 2022
According to DIP, deep image generator should be trained (i.e., adapt its parameters 8) to map fixed random tensor 2 to given corrupted image 20, and the solution to the inverse problem would be the generator's output. Formally.
arg min L(Go(2))zo), x² = G (z);
(4)
where L(x(2zg) is task-dependant loss term. This way, much of the information about the images'
distribution P, is derived from the generator's architecture.
DIP is appealing unsupervised technique for handling inverse problems, and find it is ideal for testing our core hypothesis the main claim advocated in our work is that CSC based models are adequate for generating images. If indeed correct, one would expect that CNN architectures with induced sparsity could serve better as prior for inverse problems. Recall that both Lo and L0,ee sparsity constraints are enforced via an architectural modification (a projection layer). DIP puts forward an exact such test, as its essence is reliance solely on the model's architecture itself for regularizing inverse problems. Therefore, image restoration via DIP serves as different yet meaningful setup for verifying our sparse-modeling assumptions. To this end, we aim to inject well-justified sparsity-inspired regularizations to specific activation layers within the generator for improving the final outcome.
Although DIP a generic concept that can be applied to various generator architectures, we exper- iment with the same architecture in the original paper encoder-decoder model that maps latent vector drawn from P. into an image of the same spatial extent. More specifically, DIP uses U-Net-like architecture (Ronneberger et al. 2015) with stip-connections. Due to the significant structural difference between the encoder-decoder and standard generators, applying our method in the same way in both is unjustified. To better understand the proper context of sparsity in such archi- tectures, we focus on the encoder part of the overall system. We interpret this part as transforming its input into set of transitional representations {T', I',, 1',' where K the number scales in the architecture, as can be seen in Figure 5 These are injected to the decoder, from which it constructs the output image. As these representations are attained by CNN, which resembles multi-layered thresholding algorithm (Papyan et al. 2016 Sulam et al. 2018 Sulam et al., (018)) we interpret this process as sparse coding serving
ML-CSC model. According to this perspective, the encoder maps the input random vector to a dense signal and proceeds by performing multi-layer pursuit to obtain [T,,..,, I',,, ). in the spirit of the ML-CSC model. Using this observation, we propose to apply sparsity- ceersitt-induiin regularization, described in section 3 all these representations.
Encoder
Decoder
L
Figure 5: A visualization of the intermediate representations in an encoder-docoder architecture with ckincconnettions
EXPERIMENTS
In this section we experimentally examine the proposed method, comparing its performance with non-regularized ("vanilla") image generator's 's architectures. First, we evaluate our method on image synthesis via GANs. We conduct extensive study variety of GAN architectures and explore the effect of applying our suggested regularizations to them. In addition, we study the effect of applying approach limited data sources of different sizes. We also show that -anduity
7

---

Under review as a conference paper at ICLR 2022
regularization is versatile and can be applied more general image generators. To this end, apply our method to the Deep Image Prior algorithm, which handles restoration problems. both image synthesis and in solving inverse problems, dissect the effects of the proposed regularization techniques that induce sparsity, as described in the previous sections.
5.1 IMPROVED IMAGE SYNTHESIS
For experimenting with the suggested regularizations for image synthesis, we apply these on GANs during training. The proposed approach is enforced only on the generator part, as the discrimi- nator remains untouched. We exhaustively evaluate the effect these regularizations on various GAN variants using the CIFAR-10 dataset (Krizhevsky) 2012), since it is one of the most popular benchmarks for image synthesis.
We apply method to several GAN architectures, conditional and unconditional, in order to show its compatibility with different synthesis processes. We experiment with wide range of architec- tures, considering both the CSC and the ML-CSC variants: DCGAN (Radford et al. 2013). condi- tional GAN (cGAN) (Mirza & Osindero 2014). Wasserstein GAN with Gradient Penalty (WGAN- GP) (Gulrajani et al. 2017), Mode Seeking GAN (MSGAN) (Mao al. (209)), Spectral Normal- ization GAN (SNGAN) (Miyato et al., 2018), Self Attention GAN (SAGAN) (Zhang et al.
2019)
and Improved Self Supervised GAN with Multiclass minimax game (SSGAN*) (Tran 2019)
BigGAN (Brock et al. 2019) and Differentiable Augmentation BigGAN DigiCRBigGAN KZhao
al.
2020).
To evaluate the synthesis results, we use the Fréchet Inception Distance (FID) (Huuell et al. 2017
which widely used metric for assessing the quality of generated images. Lower FID scores indicate better quality of the synthesized images. In addition to the FID, we also evaluate the performance of the conditional BigGAN models using Inception Score (IS) (Salimans al. 2016).
where higher values are better.
The results of the conducted experiments are summarized in Tables 2 The findings attest that us- ing our proposed regularization techniques significantly enhances the performance across all exam- ined GAN models, from simple to more sophisticated up-to-date architectures, which demonstrate the versatility and the generality of the proposed regularizations. Furthermore, the tested ML-CSC approach using two layers (i.e. Defs = D2 leads to improved performance, when compared to the single-layered CSC.
Table I: CIFAR-IO image synthesis FID results (lower is better). The ML column corresponds to GAN variants modified to fit the ML-CSC approach, explained in the proposed method section, using the Lu method.
Architecture
Vanilla
Lg
Lo
L1
ML
DCGAN
37.75
34.45
35.53
35.52
32.30
cGAN
27.64
26.43
26.06
26.48
26.41
WGAN-GP
30.06
28.97
28.51
30.59
28.04
MSGAN
24.32
21.72
23.80
23.91
21.86
SNGAN
25.50
25.11
24.81
24.85
23.63
SAGAN
18.39
18.23
18.37
18.48
18.09
SSGAN*
11.40
11.19
11.57
11.11
10.92
BigGAN
8.23
7.58
7.50
7.68
7.66
DifICRBigGAN
6.66
6.22
6.20
6.31
5.95
Table 2: IS results for conditional image synthesis of CIFAR-10 10 using BigGAN (higher is better).
Architecture BigGAN
Vanilla
Lo
Lo
L
ML
9,28
9.33
9.33
9.22
9.31
DifNCRBigGAN
9.36
9.39
9.48
9.42
9.48
8

---

Under review as a conference paper at ICLR 2022
We turn to study the effects of proposed regularizations when operating with limited datasets. To this end, we use 10% (5,000 images) and 20% (10,000 images) of the CIFAR-I0 dataset and experiment with BigGAN architecture. We also examine the effects of applying our method in addition to the differentiable augmentation method (Zhao et al. 2020), which provides state-of-the- art results limited data. Table demonstratess the performance improvement attained by applying our method in both of the setups.
Table 3: Image generation results (FID) limited data sources of regularized and non-regularized BigGAN architecture.
BigGAN
DigfCRBigGAN
Dataset
Vanilla
To,se
Lo
MI.
Vanilla
To,
To
MI.
20% CIFAR-10
20.25 38.33
19.78
19.96
17.72
11.29
10.93
10.62
10.33
10% CIFAR-II 10
42.52
35.72
37.77
16.80
15.30
15.82
16.20
5.2 IMPROVED SOLUTION OF INVERSE PROBLEMS
We proceed by experimenting with our regularizations on standalone image generators, in the con- text of the DIP algorithm. Our goal is to compare the ability of regularized and non-regularized image generators to serve as an implicit prior for solving inverse problems.
In the conducted experiments, we solve denoising problem with regularized and non-regularized image generators. As already mentioned in Section we use an U-Net-like "hourglass" architecture with skip-connections, in DIP (Ulyanov et al. 2018). Excluding the regularization, the compared generators have the same architecture and are being trained with exactly the same hyperparameters and setup. We conduct similar experiment to the in DIP, using the standard denoising dataset.
In this setup, an observed noisy image Zo is created by adding an additive white Gaussian noise (AWGN) with standard deviation 7n (0n 25 in experiments). To quantitatively evaluate the denoising performance, we use the Peak signal-to-noise ratio (PSNR).
The results are reported in Table where "Single" refers to the top PSNR value achieved by single output, and "Average" is the highest PSNR value of an averaged output (obtained by an exponential sliding window over past iterations, performed in DIP). We the experiment multiple times to verify the statistical significance of our results.
Vanilla
Lo
Lo
Single
28.74 + 0.03
29.03
0.03
29.22 + 0.06
Average
29.88 0.02
29.94
0.04
30.21
0.05
Table 4: Denoising results of regularized and non-regularized U-Net generator using DIP.
As can seen, enforcing sparsity by our proposed methods outperform the non-regularized model. These results demonstrate that the CSC modeling assumption is valid and contributes to better reg- ularizing the inverse problem.
CONCLUSIONS
In this work we have described simple yet effective regularization techniques for image generator architectures, which rely on sparse modeling. We demonstrate that such regularizations yield sub- stantial improvements across a wide range of GAN architectures. In addition, we show the versatility of the approach by applying it to image generators for solving inverse problems using DIP. In this context, our regularization improves the denoising results achieved by DIP. The enhanced perfor- mance achieved by promoting sparsity in image generators, in general, testifies the relevance of -arrstt-inspired models in image synthesis. We believe that this connection can be further learned and utilized to obtain even promising results.
http: hetp://www.cr cs. tut. fi/ foi /GC////F---- HM30/1 html #ref_ #ref_resss seee_res_ll_

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
In this work we have examined our proposed method both for boosting image synthesis via GANs and in improving the solution of inverse problems via DIP. both settings, implemented our approach using the official software in each of the papers we have referred to, and using the same hyperparameters as stated in their experiments. In addition, in all of our tests, we evaluated the performance multiple times using different seeds, so as to factor out randomness in the attained improvement. complete code package for running all the conducted experiments and reproducing the reported results of this paper will be published upon acceptance of the paper.
REFERENCES
M. Arjovsky, S. Chintala, and L.. Bottou. Wasserstein generative adversarial networks. In Interna- tional Conference on Machine Learning (ICML), 2017.
Y. Balaji, M. Sajedi, N. Mukund Kalibhat, M. Ding, D. Stöger, M. Soltanolkotabi, and S. Feizi. Understanding over-parameteriiztion in generative adversarial networks. In International Con- ference Learning Representations (ICLR), 2021.
H. Bristow, A. Eriksson, and S. Lucey. Fast convolutional sparse coding. In Conference on Com- puter Vision and Pattern Recognition (CVPR), 2013.
A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
A. Bruckstein, D. Donoho, and M. Elad. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Review, 2009.
R. Chalasani, J. C. Principe, and N. Ramakrishnan. A fast proximal method for convolutional sparse coding. In International Joint Conference on Neural Networks (IJCNN), 2013.
K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d transform- domain collaborative filtering. IEEE Transactions on Image Processing, 2007.
W. Dong, L. Zhang, G. Shi, and X. Wu. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing, 2011.
W. Dong, L. Zhang, G. Shi, and X. Li. Nonlocally centralized sparse representation for image restoration. IEEE Transactions Image Processing, 2013.
M. Elad. Sparse and redundant representations: from theory to applications signal and image processing. Springer Science & Business Media, 2010.
I. Goodfellow. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
R. Grosse, R. Raina, H. Kwong, and A.Y. Ng. Shift-invariance sparse coding for audio classification. arXiv preprint 120.v:1206.5222 5241, 2012.
S. Gu, Zuo, Q. Xie, D. Meng. X. Feng, and L. Zhang. Convolutional sparse coding for image super-resolution. In International Conference Computer Vision (ICCV), 2015.
Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of wasserstein gans. In Advanced in Neural Information Processing Systems (NeurIPS), 2017.
Heide, W. Heidrich, and G. Wetzstein. Fast and flexible convolutional sparse coding. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2015.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
10

---

Under review as a conference paper at ICLR 2022
T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations (ICLR), 2018.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
A. Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto Tech- nical Report, 2012.
C. Lediig L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz,
Z. Wang, and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Y. Liu, X. Chen, R. K. Ward, and Z. Jane Wang. Image fusion with convolutional sparse represen- tation. IEEE Signal Processing Letters, 2016.
S. Mahdizadehaghdam, A. Panahi, and H. Krim. Sparse generative adversarial network. In Interna- tional Conference on Computer Vision Workshops (ICCV), 2019.
J. Mairal, F. Bach, and Ponce. Sparse modeling for image and vision processing. Foundations and Trends in Computer Graphics and Vision, 2014.
Q. Mao. H.Y. Lee. H.Y. Tseng, S. Ma, and M.H. Yang. Mode seeking generative adversarial net- works for diverse image synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
X. Mao. Q. Li, H. Xie, Y. Lau, Z. Wang, and S. P. Smolley. Least squares generative adversarial networks. In International Conference on Computer Vision (ICCV), 2017.
M. Mirza and S. Osindero. Conditional generative adversarial nets. CoRR, 2014.
Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversar- ial networks. In International Conference on Learning Representations (ICLR), 2018.
V. Papyan, Y. Romano, and M. Elad. Convolutional neural networks analyzed via convolutional sparse coding. Journal of Machine Learning Research, 2016.
V. Papyan, Y. Romano, M. Elad, and J. Sulam. Convolutional dictionary learning via local process- ing. In International Conference on Computer Vision (ICCV), 2017a.
V. Papyan, Sulam, and M. Elad. Working locally thinking globally: Theoretical guarantees for convolutional sparse coding. IEEE Transactions on Signal Processing. 2017b.
F. Plaut and R. Giryes. A greedy approach to fo convolutional sparse coding. SIAM Journal on Imaging Sciences, 2019.
A. Radford, L. Metz, and Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. International Conference on Learning Representations, 2015.
O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 2015.
T. Salimans, I. Goodfellow, W. Zaremba, Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
D. Simon and M. Elad. Rethinking the ese model for natural images. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Sulam, Papyan, Y. Romano, and M. Flad. Projecting on to the multi-layer convolutional sparse coding model. In International Conference Acoustics, Speech and Signal Processing (ICASSP), 2018
Sulam, V. Papyan, Y. Romano, and M. Elad. Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Transactions Signal Processing, 2018.
11

---

Under review as a conference paper at ICLR 2022
A. Szlam, K. Kavukcuoglu, and Y. LeCun. Convolutional matching pursuit and dictionary training. arXiv preprint arXiv: 1010.0422, 2010.
H. Thanh-Tung and T. Tran. Catastrophic forgetting and mode collapse in gans. In International Joint Conference on Neural Networks (IJCNN), 2020.
Tran, V.H. Tran, N.B. Nguyen, Yang, and N.M. Cheung. Self-supervised GAN: analysis and improvement with multi-class minimax game. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Computer Vision and Pattern Recognition (CVPR), 2018.
J. Yang, Wright, T. S. Huang, and Y. Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010.
H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning (ICML), 2019.
Zhao, Z. Liu, Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient gan training. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
J.Y. Zhu, T. Park, P. Isola, and A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International conference on computer vision (ICCV), 2017.
E. Zisselman, J. Sulam, and M. Elad. A local block coordinate descent algorithm for the esc model.
In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
12

---

Under review as a conference paper at ICLR 2022
APPENDICES
A. VISUALIZATION OF THE ML-CSC ATOMS
In the field of sparse coding is a common practice visualize the learned atoms in the CSC model in order to demonstrate their variety and richness. In Figure A, show the atoms of the ML-CSC dictionary as obtained for DCGAN trained on the CIFAR-10 database. In order visualize the atoms in this two-layered ML-CSC case, need to observe both the columns of Dj and Deff D, D2 (Payynn et al. 2016 Sulam al. 2018), our setup, D contains 128 atoms of size x 4 x whereas D2 contains 128 atoms of size 3 x 128. According to the ML-CSC, each atom in D2 specifies how to combine atoms from D1 to form an atom in Dess- Therefore Dess contains 128 atoms, where each is of size & x 8 3 and created by sparse combination of atoms from D visualization of the dictionaries D1 and Deff- trained as part of regularized ML-CSC based DCGAN the CIFAR-10 10 dataset, can be viewed in the Figure A. As expected, the atoms of Deff are more complex than the atoms of D
(a)
Figure A: visualization of the dictionary atoms the ML-CSC setup obtained for DCGAN trained on the CIFAR-I0 database: (a) The 128 atoms of D,, each of size 4 x 3. (b) The 128 atoms of Deff = D,D,, each of size x 8 x 3.
B. ADDITIONAL ANALYSIS OF SPARSITY GANS
We turn to present additional graphs, similar to the one given in Figure 4a that demonstrate the effect of promoting sparsity in different GAN architectures. As can be seen Figure B, inducing sparsity consistently improves the performance of the tested models.
13

---

Under review as a conference paper at ICLR 2022
30
35
36-
34-
32-
Spersity
0.4
100
DCGAN
WGAN-GP
FID
20
Sari Ln
72
21-
20
Sparsity
Sparsity
0.4
0.2
100
150
200
250
300
100
150
200
250
300
400
SNGAN
MSGAN
Figure B: A visualization of the performance improvement obtained by the proposed sparsity- inducing methods across various GAN architectures.
14


Reviews:
[{"review_id": "uqv938AK9n", "reviewer": "Reviewer_ob62", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengthes:\n- The proposed method is neat and did improve the performance of other GANs with the proposed structure and the loss.\n\nWeaknesses:\n- It is not easy to see clear differences between the work of Mahdizadehaghdam and the proposed work. Even though the authors argued that they are different, but there is no clear supporting arguments theoretically and experimentally. CSC / ML-CSC can be seen as an advanced version of other dictionary based methods, but it has been used in other deep learning related works (see LISTA-CPSS, ALISTA) and employing CSC / ML-CSC into the framework does not seem like a major contribution.\nSee also the following work:\n[Zhou] Zhou et al., SPARSE-GAN: SPARSITY-CONSTRAINED GENERATIVE ADVERSARIAL NETWORK FOR ANOMALY DETECTION IN RETINAL OCT IMAGE, IEEE ISBI 2020.\n- It was claimed that \"We propose a novel sparse modeling interpretation of image generators for better understanding them.\", but the work of Papyan et al., 2017b does not have to be limited to non-generators, so it is hard to see this as a contribution of this work.\n- Results on CIFAR-10 seem too weak considering recent advanced in GANs for high resolution images. It is not easy to be convinced that the proposed method works well since images are too small (is the small image sparse?)\n- DIP is not a good method for denoising and usually requires multiple realizations for ensembling to boost the performance. Moreover, DIP requires early stopping for good denoising performance, while the sparsity constraint may affect this issue instead of purely enforcing sparsity. There are also prior works on DIP using other regularizers such as the following:\n[Liu] Liu et al., IMAGE RESTORATION USING TOTAL VARIATION REGULARIZED DEEP IMAGE PRIOR, IEEE ICASSP 2019.\n[Mataev] Mataev et al., DeepRED: Deep Image Prior Powered by RED, ICCVW 2019\nSo, for fair comparisons, it should be compared with these instead of vanilla DIP.\n\nComments:\n- How is the sparsity guaranteed during testing?\n- It is recommended to show images for visual comparisons.", "summary_of_the_paper": "This work proposes a sparsity enforcing structure for GANs by splitting layers into the generator for the sparse vector and for the final image and by training with sparsity enforcing regularizer in the latent space between them. The proposed method yielded improved performance over conventional methods in FID, IS for generators and PSNR for denoising with DIP.", "summary_of_the_review": "The idea of this work is to enforce sparsity in CSC domain in the latent space of GANs and this work showed the effectiveness of it. However, this idea is very close to the work of Mahdizadehaghdam and there are other works to promote sparsity in latent space such as [Zhou]. Moreover, it was tested with very small images, so it is not easy to see this work working well and the experiment with DIP seems to contain some potential issues in terms of iteration number and fairness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tSiQaUMLM1X", "reviewer": "Reviewer_tsoA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is mostly well readable and understandable, and I enjoyed reading the background section on the sparse convolutional models. Perhaps I would include some more technical detail (sizes of layers and filters, regularization weight, etc) in the appendix. The FID gains also seemed potentially promising, but on closer thought I have some significant reservations about the paper, that prevent me from giving a higher score at this time.\n\nThe first half of the paper is framed around understanding and revealing the meaning of the internal activations of deep generative CNN's. As far as I can see, the findings fall short from this goal. They don't address the intermediate activations beyond the last ones, and for those, the reader doesn't really get an understanding of whether and how the meaning has changed or been elucidated. It is not enough to simply argue that sparse coding finds more meaningful features and filters, and therefore this must also happen when they are combined with CNN's. Did it actually happen and how would we tell? Can we more clearly attribute this or that meaning to different feature channels now?\n\nThe expecatation set up in the first five pages was that it would be revealed that generators have been secretly doing sparse coding all along. However this was not really the case, and it seems that the proposed \"interpretation\" is more like a specific way to decompose the last layers so that a sparsity prior can be inserted. No sparsity seems to be really happening by default (other than maybe via relus zeroing out values, but it's arguable if this is genuine sparsity). From there on the paper mostly focused on showing that adding the regularization improves the metrics.\n\nInserting the point of sparsity specifically at the end of the network seemed a bit arbitrary to me, and even if this were justified from some principle, it may be significantly limiting in practice. For a modern GAN with a reasonably high-resolution dataset (unfortunately the only experiments are on the extremely low-resolution CIFAR, more on which later), the meaning of the last layer is arguably rather minor, at least from the standpoint of sparsity of the local neighborhoods. It no longer affects any of the global structure of the image, and effectively just overlays different versions of the same image that have already been built earlier in the network. The filters themselves have very little opportunity to affect anything beyond the finest sharp detail in the full resolution. For this reason it is somewhat unclear if any \"meaning\" can be imposed on that end of the activations, and what exactly the role of sparsity should be there.\n\nIn the DIP experiment, as far as I can see, sparsity is applied all through the network. Why is this not something one would consider with GANs also? It's also not clear why the sparsity should apply specifically to the encoder and not the decoder -- and if it were to apply to the decoder, why wouldn't it apply to the similar looking generator also? And again, are the internal activations easier to interpret in the proposed DIP compared to the original? The metric improvement is an interesting effect, but the promised understanding of the network internals is forgotten in practice.\n\nFigure 4 seems to indicate that there is a significant amount of sparsity even in the baseline model with no regularization. I assume this is because the relus set part of the values to zero? Given that there is such a strong source of apparent (and in my opinion, questionably meaningful) sparsity in the system already, the paper should clarify the relationship between these things. Also, what would happen with some pre-activation that didn't zero out any values, e.g., leaky relu? Wouldn't clamping the lowest ones just end up mostly cutting away the leaked tails, to little effect? What about tanh that produces both positive and negative values?\n\nThese shortcomings would be more acceptable if the results were striking in the end. It is true that some consistent improvement to the FID has been achieved in some cases, which is potentially promising and could warrant a closer examination. Unfortunately the evaluation is solely centered on the CIFAR dataset, which is a reasonable benchmark to include in a GAN paper, but hardly enough if it is the only one. Findings on such toy datasets do not always generalize to cases of actual interest. In this case, CIFAR has specific properties that may hide some issues in the proposed method. The very low resolution may work in favor of this approach, because now even the final filters have a fairly wide footprint, and they can still make meaningful contributions to the overall content of the image. But will this effect be diluted away with realistically sized images, where the filter footprints are on the order of a hundredth of the image size? (See my more detailed comments on the intermediate vs final layers, above.) Also, some very relevant methods have been excluded from the analysis, in particular the entire StyleGAN line. It would be important to know if the advantage persists there, as these are in widespread use.", "summary_of_the_paper": "The paper proposes viewing CNN's through the lens of sparse coding. In practice it mostly boils down to encouraging sparsity in the last layer activations. To this end the paper proposes three different mechanisms that promote sparsity. This is shown to improve the FID scores somewhat in one dataset with a range of different GAN architectures. The ideas are also applied to another generator-like DIP scenario.", "summary_of_the_review": "There are shortcomings in the overall concept as well as its evaluation. The findings suggest that this might be a promising avenue of research, but it would need to be taken further. At present, the paper boils down too much into simply adding a simple regularizer at the end and observing that it somewhat improves some metrics in a limited number of scenarios. Due to the limitations of the evaluation, it remains unclear whether the proposed improvement carries over to state of the art models and datasets. Similarly, the promised elucidation of the purpose of the feature values never really materializes.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JcF6vrLEqq", "reviewer": "Reviewer_7WVd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) The paper is overall well written. I had no problem following the text, the exposition is clear and I detected minimal amount of grammar or typo issues.\n\n2) The method is relatively simple and it could possibly be widely adopted if the experiments in the paper generalize to a wider range of problems. The paper only tested two scenarios be it with a range of baseline architectures. The paper does not discuss any downsides of the technique which I find a bit unexpected. Optimization with multiple objectives will typically introduce some kind of trade-off. In this case we are getting \"better\" (based on some metrics) images but at what cost? Did the sparsity change the convergence rate? Did the behavior of interpolation in the latent space change? Or is everything just strictly better?\n\n3) The simplicity of the method could potentially be viewed as a downside of the paper since it turns up to be just a regularization scheme applied to standard architectures. E.g. Figure 5 shows standard U-Net where additional losses and/or L0 filters are applied where necessary. Since both L1 and L0 metrics are not new it makes the paper look a bit more like an experimental survey than a completely new technique.\n\n4) I miss any qualitative comparisons of the synthesized images. The authors claim that the ML-CSC scheme leads to a significant improvement but from the tables 1,2 alone it is quite impossible to say if it is really a significant improvement or a minor improvement. Not to mention the statistical implications of the term \"significant\" without statistical analysis.\n\n5) The sole reliance on PSNR as a metric without providing visual examples is even more troublesome in the DIP experiment. PSNR (as much as L2 loss) tends to be biased towards blurry results and it is generally not very well correlated with human perception of image quality. It would be fitting to use a more sophisticated image quality metric such as LPIPS or VDP. However, showing an example result would surely be the easiest option. Judging from the numbers in Table 4, I would expect the differences to be quite small, perhaps even indiscernible. \n\n6) The paper does not provide details on how each of the baseline architecture was modified to incorporate the (ML-)CSC technique. I cannot see a supplemental document that would fill these gaps but the authors promise to provide code for all the experiments later. It is, however, not available at the time of review.\n\n\nMinor editing issues:\n- Abstract: \"leading to state-of-the-art performance\" - performance in what?\n- \"a meaningful such regularization\" -> remove such\n", "summary_of_the_paper": "The paper describes image generation as a sparse coding reconstruction process. The authors formulate the sparse coding as a core block of a convolution image generator and test several different approaches for enforcing sparsity in the trained network. The experiments show that the method can improve FID scores of common GAN networks and it can also increase PSNR of image reconstruction using deep prior when compared to the same network architectures without sparsity objective.\n\n", "summary_of_the_review": "The paper is exploring a simple but elegant idea of formulating the typically black-box process of CNN image generation as a process of sparse dictionary decoding. I like this concept and I like that it very simply improves performance of established GAN methods. The paper is also easy to read which should make this trick accessible to the community. On the downside, the technical contribution does seem boil out to a nicely packaged sparse regularization scheme. The authors could have spent more effort analyzing the learned coding. What dictionary atoms were extracted and how meaningful they are? The experimental section covers the main check-boxes including relevant baselines and ablation of the technique but it falls short in presenting the results. The judgment of the performance is limited to a very narrow scopes of quantitative metrics and it is hard to make any conclusion about the actual magnitude of the improvement that is introduced. Furthermore, the authors also do not discuss any limitations. \n\nOverall, after weighing all the cons and pros I am very slightly leaning towards positive outcome. It is mainly because the idea is clear, elegant and at least to the presented extent working. The evaluation is limited in scope but it appears technically valid. I would also expect the authors to address some of the concerns in the final revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6RfTKpMXMA", "reviewer": "Reviewer_4xX8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1). This paper has pointed out the inefficiency of existing multi-layer convolutional sparse coding models, which is to decompose the image synthesis function into two consecutive yet independent processes. This finding is solid and backed with proper proofs. \n2). Extensive experiments demonstrate that this method outperforms the state of the art baseline methods as suggested in the claims of this paper. \n\nWeakness\n1). There is one key assumption in the sparse coding that the latent code between backbone network and output is sparse when the output image is synthesised at the best performance. However, it may not be necessarily this case. The authors are suggested to prove first that the latent code layer must be a sparse vector (or  tensor)\n2). Usually, PSNR or SSIM are also used for evaluating the image restoration tasks. It would be great to provide the scores of these two traditional metrics in the paper in order to have an all-around evalutions.\n3). The paper does not clearly disclose the detailed parameters of the proposed method, for example, what is the selected threshold  $\\lambda$ value in L0 norm? And the reason to use these threshold ? What are the weighting terms of the three different normalization constraints?", "summary_of_the_paper": "This paper proposes a CNN-based (specifically GAN-based) solution to tackle image synthesis using sparse coding concept. The proposed method utilises the generator from a generative adversarial network to synthesise a sparse representation, i.e. the sparse code, for image synthesis. This method proposes to split original ML-CSC into two consecutive function learning. It claims that the two functions are easier to learn. In addition, this paper also proves and adds image regularisation such as DIP into the framework. Extensive experiments demonstrate that the proposed methods outperforms state of the art image synthesis methods including traditional sparse coding methods and deep learning based methods. \n\n", "summary_of_the_review": "This paper generally proposes to improve the existing multi-layer Convolutional sparse coding frameworks by splitting the tasks into independent and consecutive components. The idea is novel and is supported by proper experiments. However, the authors are suggested to address the concerns in the weaknesses. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces sparse modeling-inspired regularizations to improve deep neural network-based image generators. Experimental results on both (low-resolution) image synthesis and deep image prior-based inverse problems are used to validate the proposed method.
The majority of the reviewers were against the acceptance of the paper. As summarized by Reviewer tsoA: "There are shortcomings in the overall concept as well as its evaluation. The findings suggest that this might be a promising avenue of research, but it would need to be taken further. At present, the paper boils down too much into simply adding a simple regularizer at the end and observing that it somewhat improves some metrics in a limited number of scenarios. Due to the limitations of the evaluation, it remains unclear whether the proposed improvement carries over to state of the art models and datasets. Similarly, the promised elucidation of the purpose of the feature values never really materializes." The AC agrees with that summarization and recommends rejection.
Decision: Reject