Closed-loop Transcription via Convolutional Sparse Coding
Authors: Xili Dai
Ke Chen
Shengbang Tong
Jingyuan Zhang
Xingjian Gao
Yuexiang Zhai
Mingyang Li
Xiaojun Yuan
Heung-Yeung Shum
Lionel Ni
Yi Ma
Venue: Submitted to ICLR 2023 (2023)
Keywords: Convolutional Sparse Coding
Inverse Models
Rate Reduction

Abstract: Autoencoding has been a popular and effective framework for learning generative models for images, with much empirical success. Autoencoders often use generic deep networks as the encoder and decoder, which are difficult to interpret, and the learned representations lack clear structure. In this work, we replace the encoder and decoder with standard convolutional sparse coding and decoding layers, obtained from unrolling an optimization algorithm for solving a (convexified) sparse coding program. Furthermore, to avoid computational difficulties in minimizing distributional distance between the real and generated images, we utilize the recent closed-loop transcription (CTRL) framework that maximizes the rate reduction of the learned sparse representations. We show that such a simple framework demonstrates surprisingly competitive performance on large datasets, such as ImageNet-1K, compared to existing autoencoding and generative methods under fair conditions. Even with simpler networks and less computational resources, our method demonstrates splendid visual quality in regenerated images with striking sample-wise consistency. More  surprisingly, the learned autoencoder generalizes to unseen datasets. Our method enjoys several side benefits, including more structured and interpretable representations, more stable convergence, scalability to large datasets -- indeed, our method is the first sparse coding generative method to scale up to ImageNet -- and trainability with smaller batch sizes.

Under review as a conference paper at ICLR 2023
CLOSED-LOOP TRANSCRIPTION VIA CONVOLUTIONAL SPARSE CODING
Anonymous authors Paper under double-blind review
ABSTRACT
Autoencoding has been a popular and effective framework for learning genera- tive models for images, with much empirical success. Autoencoders often use generic deep networks as the encoder and decoder, which difficult interpret, and the learned representations lack clear structure. In this work, we replace the encoder and decoder with standard convolutional sparse coding and decoding layers, obtained from unrolling an optimization algorithm for solving (convex- ified) sparse coding program. Furthermore, avoid computational difficulties in minimizing distributional distance between the real and generated images, we utilize the recent closed-loop transcription (CTRL) framework that maximizes the rate reduction of the learned sparse representations. We show that such a simple framework demonstrates surprisingly competitive performance on large datasets, such ImageNet- 1K, compared existing autoencoding and generative methods under fair conditions. Even with simpler networks and less computational resources, our method demonstrates high visual quality in regenerated images with good sample-wiss consistency. More surprisingly, the learned autoencoder general- izes to unseen datasets. Our method enjoys several side benefits, including more structured and interpretable representations, more stable convergence, scalability large datasets indeed, our method the first sparse coding generative method to scale up to ImageNet and trainability with smaller batch sizes.
INTRODUCTION
In recent years, deep networks have been widely used to learn generative models for real images, via popular methods such as generative adversarial networks (GAN) (Goodfellow et al. 2020)) variational autoencoders (VAE) (Kingma & Welling 2013), and score-based diffusion models (Hyvärinen) 2005)
Sohl-Dickstein et al. 2015; Ho et al. 2020). Despite tremendous empirical successes and progress, these methods typically use empirically designed, or generie, deep networks for the encoder and decoder (or discriminator in the case of GAN). As result, how each layer generates transforms imagery data not interpretable, and the internal structures of the learned representations remain largely unrevealed. Further, the true layer-by- by-layer interactions between the encoder and decoder remain largely unknown. These problems often make the network design for such methods uncertain, training of such generative models expensive, the resulting representations hidden, and the images difficult to be conditionally generated. The recently proposed closed-loop transcription (CTRL) (Dai al., 2022b) framework aims to learn autoencoding models with more structured representations by maximizing the information gain, say in terms of the coding rate reduction (Ma et al. 2007 Yu al. 2020) of the learned features. Nevertheless, like the aforementioned generative methods, CTRL uses two separate generic encoding and decoding networks which limit the true potential of such a framework, as we will discuss later.
On the other side of the coin, in image processing and computer vision, it has long been believed and advocated that sparse convolution or deconvolution is conceptually simple generative model for natural images (Monga et al. 2021). That is, natural images at different spatial scales can be explicitly modeled as being generated from a sparse superposition of a number of atoms/motifs, known as (convolution) dictionary. There has been long history in image processing of using such models for applications such as image denoising, restoration, and superresolution (Elad Aharon 2006 Elad 2010 Yang al. 2010). Some recent literature has also attempted to sparse convolution as building blocks for designing more intepretable deep networks (Sulam et al. 2018). One conceptual benefit of such model is that the encoding and decoding can be interpreted mutually invertible

---

Under review as a conference paper at ICLR 2023
Encoder: f(x)
In Encoding layer
g/rl = argmin AlIcIl: +2 +2 A cell
X
2
? o() 2011
G+1 -t- (1112 a -2))
convolutional operator
deconvolutional operator
o()): soft-threhold function
Decoder: (z)
An invertible CSC-layer
Decoding layer
Shared convolution dictionarics
2
Closed-Loop Transcription
Convolution Sparse Coding (CSC) Layer
Figure I: Left: A CTRL architecture with convolutional sparse coding layers in which the encoder and decoder share the same convolution dictionaries. Right: the encoder of each convolutional sparse coding layer is simply the unrolled optimization for convolutional sparse coding (e.g. ISTA/FISTA).
(sparse) convolution and deconvolution processes, respectively, as illustrated in Figure I right. At each layer, instead of using two separate convolution networks with independent parameters, the encoding and decoding processes share the same learned convolution dictionary. This has been the case for most aforementioned generative or autoencoding methods. Despite their simplicity and clarity, most sparse convolution based deep models are limited to tasks like image denoising (Mohan eet al. 2019) or image restoration (Lecouat al.) 2020). Their empirical performance generation or autoencoding tasks has not yet been shown as competitive as the above mentioned methods (Aberdam et al., 2020), in terms of either image quality or scalability to large datasets.
image
Hence, in this paper, we try to investigate and resolve the following question: can we use convolutional sparse coding layers build deep autoencoding models whose performance can compete with, or even surpass, that of tried-and-tester deep networks? Although earlier attempts to incorporate such layers within the GAN and VAE frameworks have not resulted in competitive performance, the invertible convolutional sparse coding layers are naturally compatible with the objectives of the recent closed-loop transcription CTRL framework (Dai al. (0226),, see Figure left. CTRL utilizes a self-critiquing sequential maximin game (Pai et al. (002) between the encoder and decoder to optimize the coding rate reduction of the learned internal (sparse) representations. Such self- critiquing mechanism can effectively enforce the learned convolution dictionaries, now shared by the encoder and decoder at each layer, to strike a better tradcoff between coding compactness and generation quality. The closed-loop framework also avoids any computational caveats associated with frameworks such as GAN and VAE for evaluating (or approximating) distribution distances in the pixel space.
As we will show in this paper, by simply using invertible convolutional sparse coding layers] within the CTRL framework, the performance of CTRL can be significantly improved compared to using two separate networks for the encoder and decoder. For instance, as observed in Dai et al. (20226), the autoencoding learned by CTRL successfully aligns the distributions of the real and generated images in the feature space, but fails to achieve good sample-wise autoencoding. In this work, we show that CTRL can now achieve precise sample-wis wise alignment with the convolutional sparse coding layers. In addition, we show that deep networks constructed purely with convolutional sparse coding layers yield superior practical performance for image generation, with fewer model parameters and less computational cost. Our work provides compelling empirical evidence which suggests that a multi-stage sparse (de)convolution has the potential to serve as a backbone operator for image generations.
To be more specific, we will see through extensive experiments that the proposed simple closed-loop transcription framework with transparent and interpretable sparse convolution coding layers enjoy the following benefits:
1. Good performance on large datasets. Compared to previous sparse coding based generative autoencoding methods, our method scales well to large datasets such ImageNet- 1k,
Based on the implementation suggested by Zeiler et al. 2010
2

---

Under review as a conference paper at ICLR 2023
with comparable performance to the common generative or autoencoding methods based on GAN or VAE, under fair experimental comparisons.
2. Better sample-wise alignment and dataset generalizabiliny. The learned autoencoder achieves good sample-wise consistency despite only optimizing alignment between distributions. We also show the generalizability of the CSC based autoencoder - an autoencoder trained on CIFAR-I0 can be applied to reconstruct CIFAR-I00.
3. More structured representations. The learned feature representations for each class of images tend to have sparse low-dimensional linear structure that is amenable for conditional image generation.
4. Higher efficiency and stability. Our method can achieve comparable or better performance than other autoencoding methods with smaller networks, smaller training batch sizes, and faster convergence. The autoencoder learned is more stable to noise than autoencoding models based on generic networks.
2 RELATED WORK
Sparse Dictionary Learning. Inspired by neuroscience studies (Olshausen & Field 1996: 1997).
sparse coding sparse dictionary learning (SDL.) has long history and numerous applications in modeling high-dimensional data, especially images (Argyriou et al. 2008 Elad & Aharon 2006 Wright al., 2008 Yang et al. 2010 Mairal et al. 2014; Wright & Ma 2022). Specifically, given a dataset { 1- SDL considers the problem of learning an dictionary A such that y; has sparse representations y: Ax, Vi e [n]] with æ; sparse. To understand the theoretical tractability of
SDL, several lines of works based on f--morm minimization (Spielman et al. [011 Geng & Wright
2014
Sun et al., 2015 Bai et al.) 2018
Gilboa et al. 2018), (P-norm maximization (for p 2 3) (Zhai
ct al. 2020: 2019 Shen et al. 2020
Qu et al., 2019), and sum-of of--suuresssss methods have been
proposed Barak et al. 2015 : Ma et al. 2016 Schramm & Stcurer 2017). Inspired by the empirical
success of SDL on tasks such as face recogmition and image denoising Wright al. 2008); y; Mairal
et al. 2008) 2009 2012 2014), our convolutional sparse coding-bast based networks seek to learn a sparse representation from input images and use the learned sparse representation for image generation or autoencoding purposes. In particular, we adopt the classic e-minimization framework for learning sparse latent representations, as the e² norm can be viewed as a convex surrogate of the ld norm, which is a direct indicator of sparsity (Wright & Ma (022).
Convolutional Sparse Coding Layer The idea of using sparse coding for network architectures can be traced back to the work of unrolling sparse coding algorithms such as ISTA to Iearn the sparsifying dictionary (Greegor LeCun| 2010, Tolooshams Ba 202T) and some deconvolutional networks ZZiler et al., [2000]201] Papyan et al., 2017). Several recent works have explored deep networks with convolutional sparse coding layers for image denoising, image restoration, and (network normalization for) image classification (Sreter & Giryes] 2018: Mohan et al., 2019 Lecouat et al. 2020; Liu et al. 202T). The validity of these networks has mostly been demonstrated on datasets with small or moderate scales, especially for tasks such image classification or generation. Recently the SD-Net (Dai et al., 2022a) successfully demonstrated that convolutional sparse coding inspired networks can perform well on image classification tasks large image datasets such as ImageNet- K. Encouraged by such successes, our work seeks to further validate the capability of the family of convolutional sparse coding-based networks for the more challenging image generation and autoencoding tasks on large-scale datasets.
Sparse Modeling for Generative Models. We are not the first to consider incorporating sparse modeling to facilitate generative tasks. To our knowledge, most existing approaches focus around using sparsity to improve GANs. For example, exammle, MMahdizudehaghdam et al. (2019) exploits patch-based sparsity and takes in a pre- pre-trained dictionary to assembling generated patches. Ganz& & Elad (202T) explores convolutional sparse coding in generative adversarial networks, arguing that the generator is manifestation of the convolutional sparse coding and its multi-layered version synthesis process. Both methods have shown that using sparsity-inspired networks improves the image quality of GANs. However, these two works either use pretrained dictionary or limit to smaller scales of data, such as the CIFAR-I 10 dataset. Aberdamm et al, (2020) uses sparse representation theory to study the inverse problem in image generation. They developed two-layer inversion pursuit algorithm for training generative models for imagery data. On datasets like MNIST, Aberddm et al. (2020) shows that generative models can be inverted. Nonetheless, most sparse-coding inspired generative frameworks
3

---

Under review as a conference paper at ICLR 2023
have only been shown to work smaller datasets like MNIST and CIFAR-10. In this work, demonstrate that by incorporating convolutional sparse coding into a proper generative framework, namely CTRL, the convolutional sparse coding- based networks demonstrate good performance large datasets, and also have several good benefits (refer to section and 4.3
3
OUR METHOD
Our goal is to learn an autoencoder from large image datasets that can achieve both distribution wise and sample-wise autoencoding with high image quality. Our method will be based classic generative model for natural images: a multi-layer sparse (de)convolution model. The autoencoding will be established through learning the (de)convolution dictionaries at all layers based the recent closed-loop transcription framework, as it naturally optimizes coding rates of the sparse representations sought by the convolutional sparse coding.
Autoencoding and Its Caveats. In classic autoencoding problems, we consider a random vector a IR in a high-dimensional space whose distribution is typically supported on low-dimensional submanifold M. We seek a continuous encoding mapping f(.,0) say parameterized by 0, that maps RR_ to a compact feature vector = = =a()) in a much lower-dimensional space R2. In addition, we also seek (inverse) decoding mapping g(- n)), parameterized by 17, that maps the feature back to the original data space RP:
[(. 2 s(.11) : a E
(1)
in such a way that a: and a = g(f())) are "close," i.e., some distance measure D(x, .x) is small.
In practice, we only have set of n samples X = [art...., x=) of æ. Let Z f(X, 0)
[z ...... 2=] C Rdxn with z f(x', f) IR be the set of corresponding features. Similarly let X = g(Z, 17) be the decoded data from the features. The overall autoencoding process can be illustrated by the following diagram:
X (x,8) Z
g(2.)) X.
(2)
In general, we wish that X close to X based on some distance measure D(X, In particular, we often wish that for each sample x', the distance between x' and æ is small.
However, the nature of the distribution of images is typically unknown. Historically, this has caused two fundamental difficulties associated with obtaining a good autoencoding (or generative model) for imagery data. First, is normally very difficult to find a principled, computable, and well- defined distance measure between the distributions two image datasets, say X and X. This the fundamental reason why in GAN (Goodfellow et al. 2020), a discriminator was introduced replace the conceptual role of such a distance; and in VAE (Kingma & Welling 2013), variational bounds were introduced to approximate such a distance. Second, most methods not start with clear generative model for images and instead adopt generic convolution neural networks for the encoder and decoder (or discriminator). Such networks do not have clear mathematical interpretations; also, is difficult to enforce sample-wise invertibility of the networks (Dai et al. 2022b).
Below, we show how both difficulties can be explicitly and effectively addressed in our approach. Our approach starts from simple and clear model of image generation.
3.1 CONVOLUTIONAL SPARSE CODING
A Generative Model for Images as Sparse Deconvolution. We may consider an image æ, or its representation at any given stage a deep network, as a multi-dimenniooall signal æ E RMMH*W where H, W are spatial dimensions and M is the number of channels. We assume the image is generated by a multi-channel sparse code z E Cx deconvolving with a multi- kernel A e Ig-Mx xCxkxx which is referred to convolution dictionary. Here C is the number of channels for z and the convolution kernel A. To be more precise, we denote z as z - (Ci,..., Sc) where each Ç. R H is 2D array (presumably sparse), and denote the kernel A as
O11
01
Q13
ane
0122
agC
E R MXCXXXX
(3)
0LM3
aMC
4

---

Under review as a conference paper at ICLR 2023
where each aij R is 2D motif of size k x k. Then, for each layer of the generator, also called the decoder, g(z, o(z.)), , its output signal a is generated via the following operator A(-) defined by deconvolving the dictionary A with the sparse code x:
x = A(z) +
Σ (a1e * Sa:..., OMMe So) + E &RMXHXW.
(4)
where n is some small isotropic Gaussian noise. For convenience, we use "*" and "x" to denote the convolution and deconvolution operators, respectively, between any two 2D signals (a, ():
(ax ()i,)) = ΣΣ< -ii ..pppppppppp.. g), 9
(o.+) = ΣE
afp. (5)
P
Hence, the decoder g 7) is concatenation of multiple such sparse deconvolution layers and the parameters 1/ are the collection of convolution dictionaries A's (to be learned). Only batch normalization and ReLU are added between consecutive layers normalize the overall scale and to ensure positive pixel values of the generated images. Details can be found in the Appendix A
An Encoding Layer as Convolutional Sparse Coding. Now, given a multi-dimensional input E IR # sparsely generated from a (learned) convolution dictionary A, the function of "each layer" of the encoder fa. 0) is to find the optimal z, e RCC x xW from solving the inverse problem Under the above sparse generative model, according to (Wright & Ma 2022), we can seek the optimal sparse solution z by solving the following Lasso type optimization problem:
from equation {
argmin A||=|| + - A(=)1B
E ERCxHxW
We refer such an implicit layer defined by equation as a convolutional sparse coding layer. The difference between æ and A(z) is penalized by the é--norm of x A(z) flattened into a vector.
The optimal solution of given will be a close reconstruction of æ. Sparsity controlled by the entry-wise f-norm of in the objective. À controls the level of desired sparsity. In this paper, we adopt the the fast iterative shrinkage thresholding algorithm (FISTA) (Beck & Teboulle, 2009) for the forward propagation. The basic iterative operation is illustrated in Figure | A natural benefit of the FISTA algorithm is that it leads to a network architecture that is constructed from an unrolled optimization algorithm, for which backward propagation
be carried out by auto-differenniation.
Hence, the encoder f(x 8) is concatenation of such convolutional sparse coding layers (again, with batch normalization and ReLU). Recently, the work of (Dai et al.) (202a)) has shown that such a convolution sparse coding network demonstrates competitive performance against popular deep networks such the ResNet in large-scale image classification tasks. Note that in the generative setting, the operators of each layer of the encoder f are determined by the same collection of convolution dictionaries A's 's the decoder g. Thus, in the autoencoding diagram [ the parameters of the encoder f(x 0) and 1 of the decoder g(æ, 11) are determined by the same dictionaries. As we will see, this coupling benefits the learned autoencoder, even besides interpretability.
3.2 CLOSED-LOOP TRANSCRIPTION (CTRL)
The above explicit generative model has resolved the issue regarding the structure of the the encoder and decoder for the autoencoding? X f(x.0), Z glz.n), X. It does not yet address another difficulty mentioned above about autoencoding: how should we measure the difference between X and the regenerated X g( 0), ?)? As we discussed earlier, it is difficult to identify the correct distance between (distributions of) images. Nevertheless, if we believe the images are sparsely generated and the sparse codes can be correctly identified through the above mappings, then it is natural to measure the distance in the learned (sparse) feature space.
The recently proposed closed-loop transcription (CTRL) framework proposed by Dai al. (202bb) designed for this purpose. The difference between X and X can be measured through the distance between their corresponding features Z and 2 f(X, 0) mapped through the same encoder:
(x Z g(..n) x
1(x,0) 2.
X
Although the effectiveness of the choice remains to he verified.
5

---

Under review as a conference paper at ICLR 2023
Their distance be measured by the so-called rate reduction (Ma et al. 2007; Yu et al. 2020):
namely the difference between the rate distortion of the union of Z and Z and the sum of their individual rate distortions:
AR Z 2) = u 2) - (
(8)
where R(-) represents the rate distortion function of distribution. In the case of Z being Gaussian distribution and for any given allowable distortion e > 0, R(Z) can be closedly approximated by ZZ Such a A R gives principled distance between subspace-like Gaussian
log det (I ne?
ensembles, with the property that A Z Z)
0 iff Cov
Cov (2) (a al. 2007
As shown in (Dai et al. 2022b Pai et al. 2022). one can provably learn a good autoencoding by allowing the encoder and decoder to play sequential game: the encoder plays the role of
discriminator to separate Z and 2 and g plays as a generator to minimize the difference. This results in the following maxmin program:
max min AR (0), ()).
"
The program in equation 9 somewhat limited because it only aims to align the dataset X and the regenerated X at the distribution level. There is no guarantee that for each sample æ² would be close to the regenerated é' = g(f(2', f). 17). For example, Dai et al. (2022b) shows that an input image of car can be decoded into horse; the so obtained autoencoding not sample-wise consistent.
likely reason for this to happen is because two separate networks are used for the encoder and decoder and the rate reduetion objective function only minimizes error between distributions, not individual samples. Now notice that for the new convolutional sparse coding layers, parameters the encoder / and decoder y are determined by the same convolution dictionaries A. Hence the above rate reduction objective in equation 9 becomes a function of A:
A R (R(Z(((AA) (A)), ((0(A), (A)))
(10)
We can this as a cost function to guide to learn dictionaries A which are discriminative for the inputs and able to represent them faithfully through closed-loop transcription. To this end, for each batch of new data samples, step promotes a discriminative sparse encoder using only the encoder gradient, and the second, minimizing, step promotes a consistent autoencoding by using the gradients of the entire closed loop.
take one ascent step and then one descent step. The first, maximizing,
OAR
00
maxa(A) AR step:
Ast1 = A + Amax
(11)
00
0A
AA
OAR
On
OAR
00
min AR step:
Ast2 = A Amin
+
(12)
On
0A
00
0A
A
Empirically, we find that in the step to minimize AR, taking the gradient as the total derivative with respect to the dictionary A, i.e., using the gradients through both @ and 1), converges to better results than just using the gradient through 1j- see the ablation studies of Appendix B As we will see, by sharing convolution dictionaries in the encoder and decoder, the learned autoencoder can achieve good sample-wise consistency even though the rate reduction objective in equation 10 is meant to promote only distributional alignment.
4 EXPERIMENTS
We now evaluate the effectiveness of the proposed method. The main message we want to convey is that the convolutional sparse coding-based deep models can indeed scale up to large-scale datasets and regenerate high-quality images. Note that the purpose of our experiments is not to claim we can achieve state-of-the-ur performance compared all existing generative or autoencoding methods. including those that may have much larger model complexities and require arbitrary amounts of data and computational resources We compare our method with several representative categories of generative or autoencoding models, under fai fair experimentall conditions: for instance, since our
2022bb for more general cases such mixture of Gaussians or subspaces. We refer (Dai et al.
4 For example, we will not compare with methods that require very large models such as Big-GAN Brrokk
ct al.
2018) or NSCN+ (Song et al. 2020).
less fair ours.

---

Under review as a conference paper at ICLR 2023
CIFAR-10
STL-10
ImageNet
Method
IS
FIDL
IST
FID
1St
FIDJ
GAN based methods
DCGAN RRafford et al.
2015
6.6
35.3
7.8
SNGAN (Miyato et al.
2018
7.4
29.3
9.1
40.1
7.3
48.7
VAE based methods
VAE (Kingma & Welling
2013
5.2
55.9
NVAE (Vahdat & Kautz 2020
50.8
Flow based methods
GLOW (Kingma & Dhariwal
2018
46.9
Residual Flow (Chen al.
2019)
50.8
CTRL based methods
CTRL (Dai 2022b
8.1
19.6
8.4
38.6
7.7
46.9
CSC-CTRL (ours)
8.9
28.9
48.1
12.5
34.5
Table 1: Comparison on CIFAR- 10, STL-10, and ImageNet- IK. The network architectures used in CSC-CTRL are 4-layers for CIFAR-10, 10, 5-layers for STL-10 10 and ImageNet respectively which are much smaller than other compared methods. method uses only two simple networks, mainly compare with methods using two networks e.g.. one for encoder (or generator) and one for decoder (or discriminator).
Datasets and Experiment Setting, We test the performance of our method on CIFAR-10 Krizhevsky ct al. 2009), STL-10 (Coates et al. (2011) and ImageNet- 1k (Deng et al. (009) datasets. Implemen- tation details for CIFAR- I0, STL- 10 and ImageNet-Ii 1k are in Appendix A.1
4.1 PERFORMANCE ON IMAGE AUTOENCODING
We adopt the standard FID (Heusel al. 2017) and Inception Score (IS) (Salimans et al. 2016) to evaluate autoencoding quality. We compare our method to the most representative methods from the following categories: GAN, VAE, flow-based, and CTRL, under the same experimental conditions except that our method typically uses simpler and smaller models.
On medium-size datasets such CIFAR-10, observe in Table thht, in terms of these metrics, method achieves comparable or better performance compared to typical GAN, flowbbaee and VAE methods, and better IS than CTRL and VAE-based methods, which conceptually are the closest to our method. Comparing to CTRL, Fig 2 showcases the different reconstructed image between CTRL and CSC-CTRL. is clear that CSC- ESC-CTRL not only enjoys better visual quality, but also achieves much better sample-wise alignment. Visually, Figure 3 further shows sample-wise alignment between input X and reconstructed X despite our method not enforcing sample-wise constraints or loss functions!
On larger-scale datasets such ImageNet- 1k, Table | shows that we outperform many existing methods in Inception Score. Figure 3 shows that the decoded X looks almost identical to the original X. even in tiny details. All of the images displayed randomly chosen without checry-picking Due to page limitations, we place more results on ImageNet and STL-10 in Appendix
(a) X
(b) CSC-CTRL X
(c) CTRL X
Figure 2: Visualizing the auto-encoding property of the learned CSC-CTRL (x g s(X)) comparing to CTRL CIFAR-I0. (Images randomly chosen.)
'Hence, we will not compare with methods that require multiple networks for additional discriminators such the VAE-GAN (Parmar al. (202T) and the Style-GAN (Karras al. 2020
7

---

Under review as a conference paper at ICLR 2023
(a) CIFAR-IO
(b) CIFAR-10 X
(c) ImageNet X
(d) ImageNet X
Figure 3: Visualizing the auto-encoding property of the learned CSC-CTRL (X go f(X)) on CIFAR-10
and ImageNet. (Images are randomly chosen.)
4.2 STRUCTURES OF LEARNED REPRESENTATIONS
To evaluate the structural properties of the learned feature space, we visualize the reconstructed samples along different principal components in the feature space of learned classes. follow the procedure done in (Dai et al. 2022b), calculating the principal components of the representations in each learned class, and then reconstructing the samples with representation closest to these principal components. Each shows one principal component within each class. It clearly demonstrates that we may express the image diversity within each class by simply computing the principal components of the class. Even though our method does not use class label information, the model preserves statistical diversities between classes and within each class. We provide additional reconstructed images and feature space interpolation in Appendix D
in Figure displays objects of one class; each block of images
Figure 4:
Five reconstructed samples g(z) from closest to subspace spanned by top 4 principal components of learned features for ImageNet classes (class "rajidae", "goldfish", "chicken", "bird", "shark").
4.3 GENERALIZABBLLITY TO AUTOENCODING UNSERN DATASETS
To evaluate the generalizability of the learned model, we reconstruct samples of CIFAR-100 using CSC-CTRL model which is only trained on CIFAR-1O. Figure 5 shows a randomly reconstructed sample without cherry-picking. We observe that lot of classes for example, "lion", "wolf", and "snake" " which never appeared in CIFAR-10 can still be reconstructed, with high image quality. Moreover, if visualize the samples along different principal components within the class, such as "bees" in Figure 6, we see that even the variance in the out-of-domain data samples may be captured by computing the principal components. It demonstrates that model not only generalizes image reconstruction well to out-of-domain data, but also encodes a meaningful representation that preserves diversity between and within out-of-domain classes.
Figure 5: Visualization of randomly chosen reconstructed samples X of CIFAR- 100. The autoencoding model only trained on the CIFAR-10 10 dataset. dataset..
8

---

Under review as a conference paper at ICLR 2023
Figure 6: Five reconstructed samples Φ = g(x) from z's closest to the subspace spanned by the top 20 principal components of learned features for CIFAR-100 class "bee". The model is only trained CIFAR-10.
4.4 STABILITY OF CSC-CTRL
To test the stability of our method to input perturbation, we add Gaussian noise with mean of O to the original CIFAR-10 dataset. We use a to control the standard deviation of the Gaussian noise, i.e. the level of perturbation. Fundamentally, the difference between convolutional sparse coding layer and a simple convolutional layer is that the convolutional sparse coding layer assumes that the input features can be approximated by superposition of a few atoms of dictionary. This property of the convolutional sparse coding layer makes possible a stable recovery of the sparse signals with respect input noise and, therefore, enables denoising (Flad) 2010 Wrightt & Ma (022). Hence, CSC-CTRL's autoencoding also functions as denoising of noisy data. We conducted experiments on CIFAR-10, with a 0.5, and STL-10, with we use larger a so we can visualize the noise more clearly. From Figure 7 we see that CSC-CTRL outputs better-denoised image. When noise level are larger, CSC-CTRL has a obvious advantage over CTRL, which uses traditional convolutional layers. We also present more quantitative analysis of denoising Appendix E.I
0.1. Because CIFAR-I0 has smaller resolution,
CIFAR-10
STL-10
Original
Original
Original naise
Original
CTRL rooosttucted
CTRL reconstmnted
CSC-CTRI reconstrieeed
CSC-CTRL reconstnccol
Figure 7: Denoising using CTRL and CSC-CTRL CIFAR- 10 with 0.5 and STL-10 with 0.1.
The CSC-CTRL model also demonstrates much better stability in training than the generic CTRL model. The coupling between the encoder and decoder makes the training more stable. For instance, the IS score of the CSC-CTRL model typically gradually increases and converges during training, whereas the CTRL model's s IS score continuously drops after convergence. In addition, CSC-CTRL can converge with a wide range of batch sizes, from as small as to as large as 2048, whereas CTRL only converges with batch size larger than 512. These two properties are highly important from the perspective of engineering models within the CTRL framework. More details are in Appendix
5 CONCLUSION
In this work, we have shown with convincing evidence that within a closed-loop transcription framework, the classic and basic convolution sparse coding models sufficient to construct good autoencoders for large sets of natural images. This leads to a simplifying and interpretable framework for learning and understanding the statisties of natural images. This new framework naturally integrates intermediate goals of seeking compact sparse representations with an end goal of obtaining an information-rich and structured representation, measured by the coding rate reduction. The learned models demonstrate generalizability and stability. We believe this gives a new powerful family of generative models that can better support a wide range of applications that require more interpretable and controllable image generation, reconstruction, and understanding.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
All authors agree and will adhere to the conference's Code of Ethics. We do not anticipate any potential ethics issues regarding the research conducted in this work.
REPRODUCIBILITY STATEMENT
Settings and implementation details of network architectures, optimization methods, and some common hyper-paramette are described in the Appendix AA.1 We will also make our source code available upon request by the reviewers the area chairs.
REFERENCES
Aviad Aberdam, Dror Simon, and Michael Elad. When and how can deep generative models be inverted? arXiv preprint arXiv:2006. 75555, 2020.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learning. Machine Learning, 73(3):243-22 2222222222222222 2008.
Yu Bai, Qijia Jiang, and Ju Sun. Subgradient descent learns orthogonal dictionaries. arXiv preprint arXiv: 1810.10002, 2018
Boaz Barak, Jonathan Kelner, and David Steurer. Dictionary learning and tensor decomposition via the sum-of-squares method. In STOC, 2015.
Amir Beck and Mare Teboulle. A fast iterative thinklagethresholdinng algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183-20) 20-8832-222222 2009.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 11096, 2018.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 115-223. JMLR Workshop and Conference Proceedings, 2011.
Xili Dai, Li Mingyang, Tong Shengbang, Zhai Pengyuan, Gao Xingjian, Huang Shaolun, Zhu Zhihui, You Chong, and Ma Yi. Revisiting sparse convolutional model for visual recognition. In Advances in Neural Information Processing Systems, volume 34, pp. 1-12. Curran Associates, Inc., 2022a.
Xili Dai, Shenghang Tong, Mingyang Li, Ziyang Wu, Michael Psenka, Kwan Ho Ryan Chan, Pengyuan Zhai, Yaodong Yu, Xiaojun Yuan, Beeugg-Yung Shum, et al. Ctrl: Closed-loop transcription to ldr via minimaxing rate reduction. Entropy, 24(4):456, 2022b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Michael Elad. Sparse and redundant representations: from theory applications in signal and image processing. Springer Science & Business Media, 2010.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing, 15(1))3376-3375 2006.
Roy Ganz and Michael Elad. Improved image generation via sparsity. (XXi:2/04.00066: 2021.
arXiv preprint
Quan Geng and John Wright. On the local correctness of ~1-minimization for dictionary learning. In 2014 IEEE International Symposium on Information Theory, pp. 3180-3184. IEEE, 2014.
Dar Gilboa, Sam Buchanan, and John Wright. Efficient dictionary learning with gradient descent. arXiv preprint arXiv: 1809. 2018
10

---

Under review as a conference paper at ICLR 2023
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-144, 2020.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. Proceedings of the 27th international conference on international conference machine learning, pp. 399-406, 2010.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-66666 06866-667, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-779) 709, 2005. URL http: http:::/ppmr.//////// pttp:///t...ppppp//////aaaa/////
hyvarinenO5a. mtml
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8110-8119, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141vv:1412.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Citeseer, 2009.
Bruno Lecouat, Jean Ponce, and Julien Mairal. Fully trainable and interpretable non-local sparse models for image restoration. In European Conference on Computer Vision, pp. 23..3882222 238-254. Springer, 2020.
Sheng Liu, Xiao Li, Yuexiang Zhai, Chong You, Zhihui Zhu, Carlos Somnandz-GGrnnde and Qing Qu. Convolutional normalization: Improving deep convolutional network robustness and training. Advances in Neural Information Processing Systems, 14:28919-28922 2021.
Tengyu Ma, Jonathan Shi, and David Steurer. Polynomial-time time tensor decompositions with sum-of- squares. 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp- 438 446. IEEE, 2016.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE Trans. PAMI, 2007.
Shahin Mahdizadehaghdam. Ashkan Panahi, and Hamid Krim. Sparse generative adversarial network. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Discriminative learned dictionaries for local image analysis. Technical report, MINNESOTA UNIV MINNEAPO- LIS INST FOR MATHEMATICS AND ITS APPLICATIONS, 2008.
Julien Mairal, Jean Ponce, Guillermo Sapiro. Andrew Zisserman, and Francis R Bach. Supervised dictionary learning. In Advances in neural information processing systems, pp. 1033-1040, 2009.
Julien Mairal, Francis Bach, and Jean Ponce. Task-driven dictionary learning. IEEE transactions on pattern analysis and machine intelligence, 34(4),791-804, 2012.
11

---

Under review as a conference paper at ICLR 2023
Julien Mairal, Francis Bach, and Jean Ponce. Foundations and Trends in Computer Graphics and Vision, 8(2-3)885 283, 2014. ISSN 1572-2740. doi: (0.1561/06000000558 URLbttp: /wwp org/ 10 1
Sparse modeling for image and vision processing.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:/802.05993 2018.
2018.
Sreyas Mohan, Zahra Kadkhodaie, Eero P Simoncelli, and Carlos Srrandez-Grrnda Robust and interpretable blind image denoising via bias-free convolutional neural networks. arXiv preprint arXiv:1906.05478 2019.
Vishal Monga, Yuelong Li, and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2):18-44, 2021.
Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, (6588))6007-60) 1996.
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311 (311::33113325, 1997.
Druv Pai, Michael Psenka, Chih- Chiu, Manxi Wu, Edgar Dobriban, and Yi Ma. Pursuit of discriminative representation for multiple subspaces via sequential games. 2022.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research, 18(1):2887-2938, 2017.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 823 832, 2021.
Qing Qu, Yuexiang Zhai, Xiao Li, Yuqian Zhang, and Zhihui Zhu. Geometric analysis of nonconvex optimization landscapes for overcomplete learning. In International Conference on Learning Representations, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511 1.06434, 2015.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Tselil Schramm and David Steurer. Fast and robust tensor decomposition with applications to dictionary learning. arXiv preprint arXiv: 08672, 2017.
Yifei Shen, Ye Xue, Jun Zhang, Khaled B Letaief, and Vincent Lau. Complete dictionary learning via ell p-norm maximization. arXiv preprint 20000 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, PMLR, 2015.
pp.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Daniel A Spielman, Huan Wang, and John Wright. Exact recovery of sparsely-used dictionaries. In Conference on Learning Theory, pp. 37-1, 2012.
Hillel Sreter and Raja Giryes. Learned convolutional sparse coding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2191-2195. IEEF, 2018.
Jeremias Sulam, Vardan Papyan, Yaniv Romano, and Michael Elad. Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Transactions on Signal Processing. 66(15): 4090-4104, 2018.
12

---

Under review as a conference paper at ICLR 2023
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery the sphere. arXiv preprint arXiv:1504.06785, 2015.
Bahareh Tolooshams and Demba Ba. Stable and interpretable unrolled dictionary learning. arXiv e-prints, pp. arXiv-2106, 2021.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. Advances Neural Information Processing Systems, 33:19667-19679, 2020.
John Wright and Yi Ma. High-dinennsional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022.
John Wright, Allen Y Yang, Arvind Ganesh, Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE transactions on paltern analysis and machine intelligence, 31): 210-227, 2008.
Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE transactions on image processing, 19(1 11:266161228,, 2010.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-944 900444434444 2020.
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition, pp. 3528-2535. IEEE, 2010.
Matthew D Zeiler, Graham W Taylor, and Rob Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In 2011 international conference on computer vision, pp. 2018-2025. IEEE, 2011.
Yuexiang Zhai, Hermish Mehta, Zhengyuan Zhou, and Yi Ma. Understanding 14-based dictionary learning: Interpretation, stability, and robustness. In International Conference on Learning Representations, 2019.
Yuexiang Zhai, Zitong Yang. Zhenyu Liao, John Wright, and Yi Ma. Complete dictionary learning via 14-norm maximization over the orthogonal group. J. Mach. Learn. Res., 21(165):1-66 2020.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 EXPERIMENT SETTINGS AND IMPLEMENTATION DETAILS
Network backbones. For CIFAR-10, 10, we follow the 4layers architecture which is used for MNIST in (Dai al. 2022b), replacing all the standard convolutional layers with drop-in convolutional sparse coding layers in Table and 3 without extra modifications. Similarly, we adopt the S-layers
architecture for STI-110 (see Table and 5 and ImageNet- Ik (see Table 4 aad
E RIxIx512
RGB image E 32332223
4 x 4, stride=1, pad=0 CnC-iny BN 256 ReLU 4 x 4, stride=2, pad=1 CSC-inv BN 128 ReLU 4 x 4, stride=2, pad=1 CSC-inv BN 64 ReLU 4 x 4, stride=2, pad=1 CSC-inv Tanh
4, stride=2, pad=1 CSC 64 IReLU
4 4, stride=2, pad= CSC BN 128 IReLU
4 x 4, stride=2, pad=1 CSC BN 256 IReLU
x 4, stride=1, pad=0 CSC 512
Table Decoder for CIFAR-10.
Table 3: Encoder for CIFAR-10.
E R 1XTXT024
RGB image E R146643
4 4, stride=1, pad=0 CSC-iny BN 512 ReLU
4, stride=2, pad=1 CSC 64 IReLU
4 4, stride=2, pad=0 CSC-inv BN 256 ReLU
4 x 4, stride=2, pad=1 CSC BN 128 IReLU
4 x 4, stride=2, pad=1 CSC-inv BN 128 ReLU
4 x 4, stride=2, pad=1 CSC BN 256 IReLU
4 x 4, stride=2, pad=1 CSC-inv BN 64 ReLU
4, stride=2, pad=0 CSC 512
4 x 4, stride=2, pad=1 CSC-inv 1 Tanh
4 4, stride=1, pad=0 CSC 1024
Table 4: Decoder for STL-10 and ImageNet-Ik.
Table 5: Encoder for 10 and ImageNet- kk.
A.2 OPTIMIZATION AND TRAINING DETAILS.
General Settings. Adam (ingmma & Ba 2014) adopted as the optimizer for all of our experiments. The hyper-parameters of Adam and the learning rate for each dataset will be discussed later in their own section. We choose e2 0.5 for the maximin program (10) in all experiments, and the À inside the convolutional sparse coding layer is set to be 0.01 by default. For alternating minimizing and maximizing the objectives, we use the simple gradient descent-ascent algorithm. Most experiments conducted on RTX 3090 GPUs.
CIFAR-10. For CIFAR-10 10, the learning rate set to be x 10 with no decay, and we choose 31 = 0, 82 = 0.9 for Adam optimizer. Besides, we run 1000 epochs with mini-batch size 2000 for each experiment. In most cases, the model converges after about 300 epochs, with consistent visual quality and stable Inception Score.
STL-10. For STL-10, images are firstly resized to 64x64 using bilinear interpolation, and we run 1000 epochs with mini-batch size 1024, learning rate x 10-4 with decay, and hyper-parameters 3 = 0.5, 32 = 0.9 for Adam optimizer. The model converges after about 300 epochs, with consistent visual quality and stable Inception Score.
ImageNet-Ik -k. For ImageNet-Il Ik, images are firstly center- to 224x224 and then resized to 64x64 using bilinear interpolation during training. We run 10000 iterations with mini-batch size 128, learning rate 1 x 10-4 and cyper-parameters 81 0.5, B2 = 0.9 for the Adam optimizer.
B ABLATION STUDY ON OPTIMIZATION STRATEGIES
In this section, we justify our choice of optimization strategy to optimize equation T0 We set the following optimization strategy as "Strategy 1", which was adopted in the original CTRL:
ØAR
00
m) AR step:
Agt1 = A + Amax
(13)
00
0A
A.'
aAR
Qn)
(i) AR step: A Act1
Amin
(14)
On
0.A
A
14

---

Under review as a conference paper at ICLR 2023
(a) STL-10 X
(b) STL-10 X
(c) STL-10 X
(d) STL-10 X
Figure 8: Visualizing the auto-encoding property of the learned CSC-CTRL (X ((((())),) on STL-10. (Images are randomly chosen.)
We set the following optimization strategy "Strategy 2", which is used in our work to optimize equation 10
OAR
00
(A) AR step: A&+1 = A₁ + Amax
(15)
00
OAR On
OAR
00
min A AR step: Ast2 Ak+1
Amin
(16)
On
0A
00
dA
Az+1
We run ablation study CIFAR-10 with hyper-parameters all the same from Appendix, A.1 except the training strategy. The results are shown in Table 6 Empirically, we found that Strategy optimizes much better than Strategy 1.
Also in T we compare the performance of both strategies on the original CTRL model, where the encoder and decoder do not share weights. We observe that while Strategy 2 is clearly better than Strategy for CSC-CTRL, it has less stellar performance on the original CTRL model, doing comparably well Strategy 1. This indicates that Strategy 2 is least as good as Strategy 1, roughly speaking, but only achieves significantly better performance in the case where the encoder and decoder share weights.
CSC-CTRL
CTRL
IS (t)
FID (L)
IS (t)
FID (L)
Strategy 1
3.2
197.1
8.1
19.6
Strategy 2
8.9
28.9
8.5
24.7
Table 6: Ablation study of CSC-CTRL on different optimization strategies through reconstructed image quality (IS/FID). means the higher the better. means the lower the better.
C MORE VISUALIZATION OF CSC-CTRL GENERATED IMAGES
Due to limited space in the main body, we show the generated images of STL-10 (see Figure g) and some extra images of ImageNet (Figure Dh. In this section. Fig g shows the auto-encoding properties of our learned framework on STL-10. 10. Fig shows larger version of the reconstruction on ImageNet. We observe that even fine details in the image have been faithfully reconstructed, showcasing the power of our convolutional sparse coding network. Lastly, we include more generated images on ImageNet in Fig T0 demonstrating the image quality of our network.
D LINEAR INTERPOLATION IN THE LEARNED STRUCTURED FEATURE SPACE
Fig 11 shows reconstructed images whose features are linearly interpolated between pairs of images sampled from the ImageNet dataset. Formally, for two images æ1, 22, the interpolated x is given by
Pinterp = g(a (m)) + - a) (2r))
(17)
where E [0, 1] varies in Fig from 0 (on the left side) to (on the right side).
The generated images show a continuous deformation from one sample to another. This verifies that feature space is linearized and discriminative.
15

---

Under review as a conference paper at ICLR 2023
ImageNet X
(b) ImageNet X
Figure 9: Visualizing the auto-encoding property of the learned CSC-CTRL (X = ((((X,)),))
on ImageNet. (Images are randomly chosen.)
E MORE ANALYSIS OF DENOISING
E.1 QUANTITATIVE MEASURE OF IMAGE DENOISING QUALITY
Due space limitations in the main body, we present quantitative analysis of denoising in this sec- tion. We use PSNR (Peak Signal-to-Noiss Ratio), MSE (Mean Squared Error) and SSIM (Structural Similarity Index Measure) to measure the quality of denoising via CTRL and CSC-CTRL. Shown
Noise level 0.5)
PSNR (1)
MSE (L)
SSIM (t)
CTRL
13.3961
0.1914
0.1556
CSC-CTRL
17.0938
0.0837
0.3671
Table 7: Comparison of denoising via CTRL and CSC-CTRL with standard metrics. means the higher the better. & means the lower the better.
in Table 7 CSC-CTRL performs significantly better than CTRL trained with the usual convolu- tional layers. It quantitatively verifies the effectiveness of the convolutional sparse coding layer for denoising.
E.2 BETTER DENOISING THROUGH ADJUSTING
In fact, we can get better denoising effect by simply adjusting the À in the convolutional sparse coding layer in equation 6 without any additional training. Our default À is set to be 0.01 due to the scale between two objectives during the training stage. In the inference stage, we can further increase À to promote sparsity, which naturally leads better denoising. From Table g we that À increases, CSC-CTRL generally improves at denoising.
E.3 COMPARISON WITH OTHER DENOISING METHODS
To show how the CSC-CTRL model performs denoising tasks, compare methods with three denoising methods on the CIFAR- 10 dataset with different noise levels in Table We observe that model achieves better performance than other methods which are specifically trained for denoising.
F STABILITY
In this section, we further verify the training stability of CSC-CTRL from two perspectives: mode collapse during training, and choice of batch size.
Thee compared methods and their implementation can be found in com anushkayadav/Deeni sin__cifar1
16

---

Under review as a conference paper at ICLR 2023
Figure 10: Visualizing randomly chosen reconstructed images of CSC-CTRL (X (l((X.0),))
on ImageNet.
Figure 11: Images generated by features which were linearly interpolated in the learned feature space.
Training Stability, Experimentally. many previous methods such as CTRL and various GANs suffer from training instability. As shown in Figure 12] CTRL shows clear training instability after 600 epochs. In contrast, CSC-CTRL training is much more stable, as the IS score barely drops. We conclude that CSC-CTRL. suffers less from mode collapse.
17

---

Under review as a conference paper at ICLR 2023
Noise level (o 0.5)
PSNR (1)
MSE (C)
SSIM (1)
A 0.01 (default)
17.0938
0.0837
0.3671
0.1
17.5774
0.0733
0.3955
À 0.2
17.9926
0.0655
0.4222
0.3
18.3500
0.0602
0.4479
À 0.4
18.6068
0.0572
0.4658
0.5
18.6155
0.0567
0.4676
0.6
18.4205
0.0601
0,4593
0.7
18.0563
0.0660
0.4364
Table Comparison of denoising using different À with standard metrics. means the higher the
better. 4. means the lower the better.
PSNR (1)
MSE (L)
SSIM (t)
Noise level 0.5)
CSC-CTRL
17.094
0.084
0.367
AE
13.141
0.198
0.219
AE w Sym
15.784
0.117
0,318
DnCNNs
16.142
0.101
0.337
Noise level (a = 0.1)
CSC-CTRL
31.679
0.989
AE
24.830
0.868
AE w Sym
28.254
0.938
DnCNNs
28.992
0.947
Table 9: Comparison three different denoising models CIFARIO with noise levels 0.1 and 0.5.
Choice of Batch Size. One notable flaw of the original CTRL. (Dai et al., 2022b) is its reliance on a large batch size, normally greater than 512. This large batch size greatly increases the model's computational cost and limits its scalability. In Table 10 we compare whether each method converges under different batch sizes, from as small as 10 to as large as 2048. From the table, we see that CSC-CTRL successfully converge wider range of batch sizes, reduces the required computation power and enables easier training on more complicated datasets such ImageNet.
low 10. This greatly
CSC.TN.
300
200
800
000
1000
Figure 12: Training stability comparison of CTRL and CSC-CTRL with IS score on CIFAR-10.
G SENSITIVITY TO CHOICE OF RANDOM SEED
We report in Table I1 the IS/FID of CSC-CTRL with different random seeds. The experiments conducted on CIFAR-10. 10. As we can see, the choice random seed has very little effect on the performance of CSC-CTRI.
18

---

Under review as a conference paper at ICLR 2023
Batch Size
10
64
128
236
512
1024
1600
CSC-CTRL
CTRL
x
Table 10: Comparison of CTRL and CSC-CTRL trained with different batch sizes. v means the method has successfully converged, means the method fails to converge.
Random Seed
5
10
15
100
IS
8.8
8.7
8.9
8.8
8.9
FID
28.9
27.9
28.5
28.1
28.6
Table 11: Ablation study on varying random seeds.
H VISUALIZING THE LEARNED DICTIONARIES
In Figure we provide a visualization of the learned dictionaries of all layers of CSC-CTRL trained on ImageNet. The kernel size for all layers is 4 x 4. The dimension of the dictionary in the first layer of CSC-CTRL is x x x 4. We visualize this dictionary as -4 192 patches of size arranged into a 14 x 14 grid and displayed in grayscale. For the layers after the first layer, there are more than 14: 14 = 196 kernels; we visualize the first 196 kernels for readability.
(a) First layer
(b) Second layer
(c) Third layer
(d) Fourth layer
(d) Fifth layer
Figure 13: Visualization of the learned dictionary of each layer of CSC-CTRL trained on ImageNet.
I ABLATION STUDY ON CSC LAYER BASED AUTOENCODERS
In this section we demonstrate how the CSC layer may benefit autoencoding, especially with respect to generalizing to unseen datasets. use the architecture from Table and Table to get a CSC- layer based autoencoder, which train in several ways. First, directly mean square error loss on the imput æ the encoder and output = g(f(x) (x)) of the decoder to train the model; we call this "CSC-AE". Then, add the MCR loss to "CSC-AE" get the "CSC-sAE" method. Figure 14 shows the reconstructed performance of these methods and their generalizability to CIFAR-100.
On the other hand, to fairly evaluate the influence of the CSC layer reconstruction performance, replace it with a convolutional layer. More precisely, we use the architecture from Table 2 and Table but replace the CSC layers the encoder with convolutional layers, and also replace all CSC-inv layers of the decoder with Cony'Tie Transpose2D layers to get a CNN-based autoencoder. With the generic mean square error loss, we get the "Conv-AE" model: with VAE loss, we get the "Cony-VAE" model. Finally, CCovv-CTRL" is the same as the original CTRL method. Figure [5] shows the reconstruction performance of these methods and their generalizability C10AR-100.
19

---

Under review as a conference paper at ICLR 2023
(a) Input
(b) CSC-AE
(c) CSC-sAE
(d) CSC-CTRL
Figure 14: Image reconstruction results of CSC layer based autoencoder trained on CIFAR-I0 with different loss function such (b) MSE, (c) MSE+MCR", and (d) closed loop framework. The first shows the reconstructed results CIFAR- 10 test set. The second raw shows the reconstructed results on CIFAR- 100 test set, which also reflects the generalizability.
(a) Input
(b) Conv-AE
(c) Conv-VAE
(d) Conv-CTRL
Figure 15: Image reconstruction results of Convolution layer based autoencoder trained on CIFAR-10 with different loss function such (b) MSE, (c) MS,+KL, and (d) closed loop framework. The first raw shows the reconstructed results CIFAR- 10 test set. The second raw shows the reconstructed results on CIFAR-100 test set, which also reflects the generalizability.
20

---

Under review as a conference paper at ICLR 2023
CIFAR-10
STL-10
ImageNet
Method
Model Size
Train Time
1St
FID|
ISt
FIDJ
1St
FID
GAN based methods
DCGAN Radford et al.
2015
6.6
35.3
7.8
SNGAN Miyato et al.
2018
7.4
29.3
9.1
40.1
7.3
48.7
VAΣ based methods
VAE Kingma& & Welling
2013
5.2
55.9
NVAE Vahdat & Kautz
2020
10M
>S5 h
50.8
NVAE (Recon)
10M
>55 h
2.67
DCVAE PParmar et al. 2021
4M
>24h
8.2
17.9
8.1
41.9
DCVAE (Recon)
4M
>24h
7.9
21.4
8.4
43.6
Flow based methods
GLOW Kingma& & Dhariwal
2018)
46.9
Residual Flow (Chen et al.
2019
50.8
CTRL based methods
CTRL Dai et al.
2022b
1.0M
15 h
8.1
19.6
8.4
38.6
7.7
46.9
CSC-CTRL (ours)
0.5M
8 h
8.9
28.9
9.1
48.1
12.5
34.5
Table 12: Comparison on CIFAR- 10, STL-10, and ImageNet-IK. The network architectures used in CSC-CTRL are 4-layers for CIFAR-10, 10, 5 5-layers for STL-10 and ImageNet respectively which are much smaller than other compared VVAEeeco)) means the results of reconstruction, the column "Train Time" means the hours the model used for training.
21


Reviews:
[{"review_id": "yanBEKuy3k", "reviewer": "Reviewer_Ti8A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an autoencoder architecture whose encoder and decoder are constructed based on the convolutional sparse coding generative model. The encoder is a convolutional sparse coding layer solving lasso (sparse coding problem) by unrolling FISTA (fast iterative thresholding algorithm). The decoder is dictated by the sparse coding/dictionary learning model. In place of the usual reconstruction loss, the authors use closed-loop transcription (CTRL) to train the network, i.e., to maximize the rate reduction of the learned sparse codes. To learn the dictionary, they use a max-min optimization problem for the encoder to discriminate between image encoding and reconstruction encoding and for the decoder (generator) to minimize the difference.\n\nHere are their contributions from their perspective: The paper argues that they are the first to scale up the sparse coding model and train it with ImageNet. They argue that one of the benefits of their network architecture is having interpretable and structured representation. This arises due to their sparse coding encoder architecture and their training approach of maximizing the information gain. With the usage of the CTRL approach to train the framework, they offer sample-wise alignment. They emphasize that the construction of a network-based of sparse coding model offers efficient model and training. They focus on the image generation quality of the framework and how it generalizes.\n\nHere is their contribution from the reviewer's perspective: The paper combines two existing ideas of unrolled networks and CTRL training. They show the reconstruction/generation performance of the method.", "summary_of_the_review": "The paper lacks a clear novelty. It combines two ideas and the majority of their findings are already known in each of those ideas (unrolling optimization method to create autoencoder, and CTRL to train for image generating autoencoder). Aside from novelty, the method although argued to be scaled up of CSC, still does not choose SOTA as a baseline. Literature review missing key papers. See my review above.\n\n\n--------------\n\nMy original score was 5. I increased it to 6 after a long discussion.", "strength_and_weaknesses": "strengths:\n\nThe paper was easy to read. It is organized, and I could follow the arguments. The experiments seem reproducible (enough details are provided in the appendix). There are reasonable visualizations and characterizations of their method. However, given my concerns explained below, I do not recommend an acceptance.\n\nweaknesses:\n\n- Although this paper combines two interesting ideas, it lacks novelty. To elaborate, the paper does not offer new knowledge. The ideas are: constructing autoencoders based on an optimization model (in this case the deep generative sparse coding model), and using CTRL to train the network to boost up image generation quality of the network.\n\n- The literature review is not thorough. For example, unrolling optimization algorithms are used abundantly in the literature. [1] is a review paper about this approach and its wide usage in image and signal processing applications. For networks-based convolutional sparse coding, [2,3], among the many, is missing; the works show how to construct a recurrent autoencoder based on a shallow sparse coding model and train it for image denoising. Both of these papers have already shown the computational and trainable parameter efficiency of unrolling approach and have shown the competitive performance against SOTA at the time of their publications.\n\n- Compared to other image generation methods, it is not clear what this paper is offering. A thorough discussion and comparison with SOTA are missing. Or a discussion on why SOTA comparison is not needed. This is especially needed as the proposed method is a combination of two already existing approaches. Moreover, this is particularly important, as the authors argue that they are the first to scale up CSC.\n\n- Lack of baselines: Although I enjoyed looking into the reconstructed images, baselines are lacking for several visualizations. The principal approach for image generation can also be applied to other generating architecture. How is their performance (e.g., Figure 4,6)? For example, in Section 4.3, the authors argue that their framework generalizes to autoencoding unseen datasets. Aren't other autoencoders able to do this? If yes/no, please include such discussion and comparison (e.g., for Figure 5).\n\n- Interpretability: the authors emphasize that their framework has the benefit of interpretable representation. However, they do not define interpretability. Indeed, there are many definitions of interpretability [4], and elaboration is needed here. Regarding the interpretability of deep networks constructed based on the generic sparse coding model, [5] is missing in their literature review. Certain questions arise about the interpretability, the kernel size in the trained architecture is too small. Hence, in terms of learned features, their kernel seems rather small to learn human interpretable features. One question is how the interpretability of their framework is useful.\n\n- The paper confuses two concepts of \"image generation\" and \"image reconstruction\". In many of the examples, for example, Figure 2, the authors are visualizing the network reconstruction (pass the image into the encoders, and then decoder). This does not image generation; the code is created by passing an image into the encoder.\n\nSome questions and recommendations.\n\n- In Section 4.2, the authors say that \"... reconstructing the samples with representation closest to these principal components\". Where are the samples coming from? From my understanding, there seems to be a set of encodings from a set of images. The method is using those encodings that are closest to the principal components of the representations to reconstruct an image. Again, there is no sampling as in GANs. This is not an image generation, and looking into PC of a set of representations can be applied to other autoencoders. Please elaborate.\n\n- What is the significance of large datasets that the author emphasizes if the goal is not classification, but image reconstruction? At the end of the day, all networks are being trained by gradient descent (hence not all data is going to be used at once for parameter updates). I do find the contribution of the fact that they are the first to train on imagenet-1k for image reconstruction (not classification) to be a minor contribution.\n\n- I recommend adding SOTA to Table 1, to give the reader a perspective on how far the performance is from SOTA. Perhaps to help you with your arguments on scaling up csc but not comparing to SOTA, I suggest including an extra column to report on network parameters and computational efficiency.\n\n- The paper frequently makes statements that are not precise and need elaboration. For example, prior to Section 3, it states \"... have several good benefits unknown to any of the previous generative methods\". Please be precise and explain the benefit and specify prior works.\n\n- Please report the number of parameters to support the claim on efficiency.\n\nMinor\n\n1. Page 2, bullet 1, citation is missing for previous sparse coding works.\n2. The authors emphasize much on convolutional and miss the generic sparse coding literature (at the end of the day, convolution is a linear operator with structure). In the unrolling literature, there are many theoretical works on unrolled sparse coding that may worth to be cited. Here are two [6,7].\n3. replace \".\" with \",\" on (4).\n4. It would be nice to provide an additional explanation on CTRL. For example, their method from the perspective of fixed points.\n5. Suggest changing \\lambda_max and \\lambda_min in (11) and (12) to avoid confusion with \\lambda in the lasso.\n6. please cite lasso [8] before (6).\n7. Please report the noise \\sigma of the image in the scale of a/255. What is a? (Section 4.4)\n8. There are certain wordings not supported in the paper. Please remove or support. E.g., in conclusion, the usage of \"convincing\" and \"unprecedented\".\n9. Figure 8, if (a) and (c) are similar, please combine. same for (b) and (d).\n10. I liked the stability discussion from the perspective of the connectivity of the encoder and decoder. If space allows, I recommend moving the losses from the appendix into the main paper.\n\n[1] Monga V, Li Y, Eldar YC. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine. 2021.\n[2] Simon D, Elad M. Rethinking the CSC model for natural images. Advances in Neural Information Processing Systems. 2019.\n[3] Tolooshams B, Song A, Temereanca S, Ba D. Convolutional dictionary learning based auto-encoders for natural exponential-family distributions. In International Conference on Machine Learning 2020.\n[4] Doshi-Velez F, Kim B. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. 2017.\n[5] Tolooshams B, Ba D. Stable and interpretable unrolled dictionary learning. Transaction in Machine Learning Research. 2022.\n[6] Chen X, Liu J, Wang Z, Yin W. Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. Advances in Neural Information Processing Systems. 2018.\n[7] Ablin P, Moreau T, Massias M, Gramfort A. Learning step sizes for unfolded sparse coding. Advances in Neural Information Processing Systems. 2019.\n[8] Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological). 1996.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_vKhOCgC8r", "reviewer": "Reviewer_b6Q7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to replace the encoder and decoder with standard convolutional sparse coding and decoding layers in an autoencoder. The proposed approach can be trained on high-resolution images and can be used to reconstruct images and capture interpretable representations. ", "summary_of_the_review": "Given that the paper's clarity is limited and the experiments are also not very convincing, the paper needs to be improved for acceptance.", "strength_and_weaknesses": "**Strength:**\nThis paper has many experiments.\n\n**Weakness:**\n1. Some of the arguments are overclaimed. For instance, in Introduction: \"The learned autoencoder achieves striking sample-wise consistency\". Not sure if the performance is really that impressive.\n\n2. \"our method scales well to large datasets\": it is unclear what \"scales well\" mean. Does it mean the model is able to be trained on large datasets, or achieve good performance?\n\n3. As mentioned in Section 4, \"The main message we want to convey is that the convolutional sparse coding-based deep models can indeed scale up to large-scale datasets and regenerate high-quality\".  Not sure if the contribution is sufficient. At the same time, the reconstructed images do not seem to be very impressive compared to many other generative models like diffusion models, and normalizing flows.\n\n4. The experiment settings are poorly explained. The model's performance is also very limited.  \n\n5. In section 4.1, besides showing the reconstruction qualitatively in figure 2 and 3, it would be more convincing to evaluate the reconstruction qualitatively and compare it with the baselines.\n\n6. Table 1 is very confusing. Are the IS and FID scores evaluated on samples or reconstructions? If they are evaluated on samples, then there should be more details on sampling from the proposed model. Currently, I do not see the sampling algorithm in the paper. If they are evaluated on reconstructions, then I am not sure if the comparison with the GAN, VAE, flow baselines are fair (since they are evaluated on synthetic samples).\n\n7. The figures and their captions are confusing. For instance, for figure 4-6, it would be much better to label the x, y-axis and each block to clarify what each image is.\n\n8. It is unclear why the performance in figure 6 is impressive: many autoencoders will also generalize to a different datasets due to some inductive bias. Especially since CIFAR100 is relatively similar to CIFAR-10.\n\n9. The denoising experiment in figure 7 does not seem to be working.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RQU26ITeIAP", "reviewer": "Reviewer_uge7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a model for image generation which combines convolutional sparse coding (CSC) with the closed-loop transcription (CTRL) framework by Dai et al (2022). The encoder and decoder of this CSC-CTRL model share dictionaries of convolutional kernels at each sparse coding layer. The encoder produces a sparse code at each CSC-layer by unrolling the FISTA algorithm. The decoder applies deconvolution to the final output of the encoder and each successive CSC-layer output to reconstruct the original input. The system is trained with a rate reduction objective which aims to minimize the distance between the distributions of codes produced from the original inputs and those produced from the reconstructed inputs. The proposed model is trained on CIFAR-10, STL-10, and ImageNet-1K, and achieves comparable or favorable performance when evaluated against GAN, VAE, Flow, and CTRL methods on metrics such as Inception Score and FID. Additional analysis suggests that the feature space of the proposed model preserves class information even though it is trained with an unsupervised objective. Further experimental results suggest that the CSC-CTRL model generalizes to data from classes not seen during training, is robust to noise, and is more stable than the CTRL model when trained with different batch sizes. \n", "summary_of_the_review": "Overall, this work presents an interesting idea of combining convolutional sparse coding with closed loop transcription for image generation. The model is compared to generative models of similar or higher capacity and performs favorably in many cases. The qualitative results on the structure of the feature space, generalization, as well as robustness to noise are insightful. I pose some recommendations on ways to strengthen the statements in the paper by including clarifications and more analysis.", "strength_and_weaknesses": "**Strengths**\n\n- The proposed architecture combining convolutional sparse coding with closed loop transcription is novel.\n\n- The proposed model is compared to and in many cases performs better than existing generative models of similar capacity on popular evaluation metrics.\n\n- Qualitative analysis shows that images generated by the CSC-CTRL model have high visual fidelity, both sample-wise and distribution-wise.\n\n- The paper is generally written clearly.\n\n**Room for improvement**\n\n- *ImageNet scale*: I believe the statement in the abstract that the CSC-CTRL model “scales up to ImageNet” might be misleading since Appendix A.2. states that the ImageNet images are resized from 224x224 to 64x64. Would the authors please clarify what ImageNet image size is used during training? \n\n- *Statistical significance*: It would strengthen the paper’s argument to include confidence intervals in Table 1.\n\n- *Generalization to Unseen Datasets* and *Stability of CSC-CTRL*: It would complement the qualitative analysis in figures 5 and 7 to provide quantitative analysis such as average PSNR the model achieves on: CIFAR-10 (which it is trained on) versus PSNR on CIFAR-100 in Fig 5; the average PSNR of reconstructions of the original versus noisy inputs for both the CTRL and CSC-CTRL models trained on CIFAR-10 and STL-10 (evaluated on a held-out test set). \n\n- *Clarification on equations (9) and (10)*: In my understanding $\\theta$ and $\\eta$ (the encoder and decoder’s parameters) are the same since they are determined by the weights of the shared convolutional dictionaries. Is my understanding correct?\n\n- *Related literature*: I believe it would be helpful to include reference to either [1, 2] in the main text.\n\n- *Visualization of convolutional filters*: It would be informative to display the learned convolutional filters which is a commonly adopted practice in the related literature. \n\n- *Reproducibility*: It would be helpful if the authors release their implementation for reproducibility.\n\n[1] Zeiler, M.D., Krishnan, D., Taylor, G.W. and Fergus, R., 2010, June. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition (pp. 2528-2535). IEEE.\n\n[2] Zeiler, M.D., Taylor, G.W. and Fergus, R., 2011, November. Adaptive deconvolutional networks for mid and high level feature learning. In 2011 international conference on computer vision (pp. 2018-2025). IEEE.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Odxprz1nenr", "reviewer": "Reviewer_qbVZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interesting approach to design autoencoders based on convolutional sparse coding. In this framework, each layer-wise operation is replaced by an approximate solution to a Lasso problem, with a convolutional dictionary. The reconstruction ('decoding') from the so-obtained sparse codes is given simply by a linear model. The authors train the resulting architecture with a recently proposed method, or loss function, termed Closed Loop Transcription (CTRL), which optimizes the rate reduction of encoder and decoder in a minimax formulation. The paper presents empirical results on CIFAR 10, CIFAR 100 and Imagenet 1K.\n", "summary_of_the_review": "Interesting paper with potentially valuable contribution, and with some elements that need further clarification.", "strength_and_weaknesses": "This paper is interesting and represents a valuable contribution. Indeed, it is nice to see models based on CSC providing such nice results. The paper is also clear and easy to read.\n\nThis reviewer does however have a few comments and concerns:\n\nMain comments:\n\n1. This main paper has very few details about the implementation or on how these networks are implemented, and all of these are deferred to the appendix. While this is mostly fine, it would be nice to have some clarifications. Chiefly: from the description of the decoder, it seems like every decoding process of each layer is completely linear. Is this correct? If this is the case, then is the entire decoder (for all layers) completely linear? If this is the case, can the authors explain why not to replace the product of all these linear layers with simply a (single) linear layer, as they can indeed be made equivalent?\n\n2. An important question refers to the way the authors optimize the problem in Eq. (9). The authors parameterize two players (encoder and decoder), and pose this as a max-min problem. Both networks are parameterized by the dictionary parameters, $A$. Now, instead of optimizing this problem in this way, in the line immediately following Eq.(12), the authors comment that, empirically, they find it beneficial to modify one of the first order updates in some pretty ad-hoc way. They also comment in passing that they include an ablation study on this choice in Appendix B. Appendix B indeed shows the comparison of their performance with their 'original' motivated method (strategy 1) versus the 'ad-hoc update' version (strategy 2). My issue with this is that Strategy 2 does not just improve a little their method giving them a small boost -which would be reasonable- but instead fully determines the competitive advantage of their method: Their strategy 2 performs almost an order of magnitude better (3.2 vs 8.9 in IS, and 197 vs 28.9 in FID) for CIFAR-10. Moreover, the \"worst\" performing method that they compare with (VAE) achieve 5.2 (IS) and 56 (FID), which is about 3 times (say, on average) as good as their strategy 1. \nIn an empirical paper like this, it is natural to have some small details of the implementation that can provide a small improvement. Yet, from these numbers, it appears as this little ad-hoc trick is responsible for their method becoming \"state-of-the-art\", and I think that as such it should deserve some closer study.\n\n3. It is also very strange that their method provides sample-wise alignment: indeed, as the authors mention, the entire network is trained on a loss that looks at statistics of the distribution of (real and generated) samples. Thus, it indeed need to provide matched samples, as shown by [Dai et al, 2022]. In this work, the authors seem to use the same loss and training framework, but all of a sudden they can reconstruct matched samples. How come? \n\n4. While the contribution of this paper is interesting and relevant, in my humble opinion and respectfully, the paper is a bit on the verbose side and some comments are a bit too subjective or exaggerated. For instance, in a couple of places, the authors comment that their methods shows \"striking performance on large datasets\", or \"splendid visual quality\". It is true that their IS and FID numbers are better than SNGAN and CTRL in ImageNet, but looking at the generated images (e.g. in Fig 3d and Fig 9b), I would not qualify their quality as \"striking\" or \"splendid\" (plenty of artifacts, both shape-wise and color-wise). \n\n5. To the best of my knowledge, the first work in relating sparse coding networks to deep networks, and even suggesting replacing each convolutional layer by a lasso formulation -as done in this work- is the work by Papyan et al, 2017, in JMLR. I think credit is due to that work for some of the ideas appearing here. \n\nI enjoyed reading this paper, and I look forward to reading the authors' responses to clarify my understanding.\n\n\nSmall comments:\n\n- Why do the authors refer to (de)convolution layers? Looking at their definitions, what they refer to \"(de)convolution\" is just a cross-correlation operator - which is just the adjoint of a convolution operator. \n \n- I'm curious why the approach chose to abbreviate \"closed-loop transcription\" as \"CTRL\". Did the authors perhaps meant \"CLTR\"?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject