Closed-Loop Transcription via Convolutional Sparse Coding

Authors: xili  dai
ke  chen
shengbang  tong
jingyuan  zhang
xingjian  gao
mingyang  li
druv  pai
yuexiang  zhai
xiaojun  yuan
heung-yeung  shum
lionel m ni
yi  ma
Venue: CPAL_cc_2024_Conference
Published: 2023-08-29 06:23:55

Abstract: Autoencoding has achieved great empirical success as a framework for learning generative models for natural images. Autoencoders often use generic deep networks as the encoder or decoder, which are difficult to interpret, and the learned representations lack clear structure. In this work, we make the explicit assumption that the image distribution is generated from a multi-stage sparse deconvolution. The corresponding inverse map, which we use as an encoder, is a multi-stage convolution sparse coding (CSC), with each stage obtained from unrolling an optimization algorithm for solving the corresponding (convexified) sparse coding program. To avoid computational difficulties in minimizing distributional distance between the real and generated images, we utilize the recent closed-loop transcription (CTRL) framework that optimizes the rate reduction of the learned sparse representations. Conceptually, our method has high-level connections to score-matching methods such as diffusion models. Empirically, our framework demonstrates competitive performance on large-scale datasets, such as ImageNet-1K, compared to existing autoencoding and generative methods under fair conditions. Even with simpler networks and fewer computational resources, our method demonstrates high visual quality in regenerated images. More surprisingly, the learned autoencoder performs well on unseen datasets. Our method enjoys several side benefits, including more structured and interpretable representations, more stable convergence, and scalability to large datasets. Our method is arguably the first to demonstrate that a concatenation of multiple convolution sparse coding/decoding layers leads to an interpretable and effective autoencoder for modeling the distribution of large-scale natural image datasets.

Main paper: 
 introduction: 
 In recent years, deep networks have been widely used to learn generative models for real images, via popular methods such as generative adversarial networks (GAN) [20] , variational autoencoders (VAE) [29] , and score-matching based diffusion models [23, 24, 56] . Despite tremendous empirical successes and progress, these methods typically use empirically designed, or generic, deep networks for the encoder and decoder (or generator and discriminator, in the case of GAN). The recently proposed closed-loop transcription (CTRL) [12] framework aims to learn autoencoding models with more structured representations by maximizing the information gain, in terms of the coding rate reduction [35, 68] of the learned features. Nevertheless, like the aforementioned generative methods, CTRL uses two separate generic encoding and decoding networks which limit the potential of such a framework. We seek to remedy this issue in this work.In image processing and computer vision, it has long been believed and advocated that sparse convolution or deconvolution is a conceptually simple and interpretable model for analyzing or synthesizing natural images. That is, natural images at different spatial scales can be explicitly modeled as being generated from a sparse superposition of a number of atoms/motifs, known as a (convolution) dictionary [65] . One conceptual benefit of such a model is that the encoding and decoding can be interpreted as mutually invertible (sparse) convolution and deconvolution processes, respectively, as illustrated in Figure 1 right. At each layer, instead of using two separate convolutional networks with independent parameters (which has been the case for most aforementioned generative or autoencoding methods), the encoding and decoding processes now share the same learned convolution dictionary. Despite their simplicity and clarity, most sparse convolution-based deep models are limited to tasks like image denoising [44] or image restoration [32] . Their empirical performance on image generation tasks has not yet been shown to competitive with the above mentioned methods [1] , in terms of either image quality or scalability to large datasets. Hence, in this paper, we try to investigate and resolve the following outstanding question:In this work, we provide an affirmative answer to this question, using invertible convolutional sparse coding layers within the CTRL framework. We improve the CTRL framework and achieve precise sample-wise alignment with the convolutional sparse coding layers. In addition, we show that deep networks constructed purely with convolutional sparse coding layers yield superior practical performance for image generation, with fewer model parameters, and less computational cost. Our work provides compelling empirical evidence which suggests that a multi-stage sparse (de)convolution has the potential to serve as an interpretable and effective model for natural image analysis and generation. To summarize, the proposed CSC based autoencoders enjoy the following benefits:1. Good performance on large datasets. Compared to previous sparse coding based generative methods, our method scales well to large datasets such as ImageNet-1k, with a comparable performance than the common generative methods based on GAN or VAE, under fair experimental comparisons.Better sample-wise alignment and dataset generalizability. The learned autoencoder achieves striking sample-wise consistency despite only optimizing alignment between distributions. We also show the generalizability of the CSC based autoencoder to unseen datasets -an autoencoder trained on CIFAR-10 can be applied to reconstruct CIFAR-100.3. More structured representations. The learned feature representations for each class of images tend to have sparse low-dimensional linear structure that is amenable for conditional image generation.Higher efficiency and stability. Our method can achieve comparable or better performance compared to other autoencoding methods, with smaller networks, smaller training batch sizes, and faster convergence. The autoencoder learned is more stable to noise than generative models based on generic networks.Our current implementation remains rather basic, and is meant to demonstrate its simplicity. There are many aspects of the implementation which may be further improved and refined for better performance and image quality. 
connections to related work: 
 Sparse Dictionary Learning. Inspired by neuroscience studies [45, 46] , sparse coding or sparse dictionary learning (SDL) has a long history and numerous applications in modeling high-dimensional data, especially images [2, 16, 41, [65] [66] [67] . Specifically, given a dataset {y i } n i=1 , SDL considers the problem of learning an dictionary A such that y i has sparse representations y i ≈ Ax i , ∀i ∈ [n] with x i sparse. To understand the theoretical tractability of SDL, several lines of works based on ℓ 1 -norm minimization [3, 18, 19, 58, 60] , ℓ p -norm maximization (for p ≥ 3) [49, 55, 69, 70] , and sum-of-squares methods have been proposed [4, 34, 54] . Inspired by the empirical success of SDL on tasks such as face recognition and image denoising [15, [38] [39] [40] [41] 66] , our convolutional sparse coding-based networks seek to learn a sparse representation from input images and use the learned sparse representation for image generation or autoencoding purposes.Sparse Modeling for Generative Models. We are not the first to consider incorporating sparse modeling to facilitate generative tasks. To our knowledge, most existing approaches focus around using sparsity to improve GANs. For example, Mahdizadehaghdam et al. [37] exploits patch-based sparsity and takes in a pre-trained dictionary to assemble generated patches. Ganz & Elad [17] explores convolutional sparse coding in generative adversarial networks, arguing that the generator is a manifestation of the convolutional sparse coding and its multi-layered version synthesis process. Both methods have shown that using sparsity-inspired networks improves the image quality of GANs. However, these two works either use a pretrained dictionary or limit to smaller scales of data, such as the CIFAR-10 dataset. Aberdam et al. [1] uses sparse representation theory to study the inverse problem, developing a a two-layer inversion pursuit algorithm with great invertibility on datasets like MNIST. Nonetheless, most sparse-coding inspired generative frameworks have only been shown to work on smaller datasets like MNIST and CIFAR-10. In this work, we demonstrate that by incorporating convolutional sparse coding into a proper generative framework, namely CTRL, the convolutional sparse coding-based networks demonstrate striking performance on large datasets, and also have several benefits unseen by any of the previous generative methods. 
our methods: 
 Our goal is to learn an autoencoder from large image datasets that can achieve both distribution-wise and sample-wise autoencoding with high image quality. Our method will be based on a classic generative model for natural images: a multi-layer sparse (de)convolution model. The autoencoding will be established through learning the (de)convolution dictionaries at all layers. Such dictionaries are learned through the recent closed-loop transcription (CTRL) framework [12, 61, 62] . In particular, the coding rates of the sparse representations sought by the convolutional sparse coding are optimized during the learning process.In classic autoencoding problems, we consider a random vector x ∈ R D in a high-dimensional space whose distribution is typically supported on a lowdimensional submanifold M. We seek a continuous encoding mapping f (•, θ), say parameterized by θ, that maps x ∈ R D to a compact feature vector z = f (x) in a much lower-dimensional space R d . In addition, we also seek an (inverse) decoding mapping g(•, η), parameterized by η, that maps the feature z back to the original data space R D :EQUATIONin such a way that x and x = g(f (x)) are "close," i.e., some distance measure D(x, x) is small.In practice, we only have a set of n samples X = [x 1 , . . . ,x n ] of x. Let Z = f (X, θ) . = [z 1 , . . . , z n ] ⊂R d×n with z i = f (x i , θ) ∈ R d be the set of corresponding features. Similarly let X . = g(Z, η) be the decoded data from the features. The overall autoencoding process can be illustrated by the following diagram:EQUATIONIn general, we wish that X is close to X based on some distance measure D(X, X). In particular, we often wish that for each sample x i , the distance between x i and xi is small.However, the nature of the distribution of images is typically unknown. Historically, this has caused two fundamental difficulties associated with obtaining a good autoencoding (or generative model) for imagery data. First, it is normally very difficult to find a principled, computable, and welldefined distance measure between the distributions of two image datasets, say X and X. This is the fundamental reason why in GAN [20] , a discriminator was introduced to replace the conceptual role of such a distance; and in VAE [29] , variational bounds were introduced to approximate such a distance. Second, most methods do not start with a clear generative model for images and instead adopt generic convolution neural networks for the encoder and decoder (or discriminator). Such networks do not have clear mathematical interpretations; also, it is difficult to enforce sample-wise invertibility of the networks [12] .Below, we show how both difficulties can be explicitly and effectively addressed in our approach.Our approach starts from a simple and clear model of image generation.A Generative Model for Images as Multi-Stage Sparse Deconvolutions. We may consider an image x, or its representation at any given stage of a multi-stage model, as a multi-dimensional signal x ∈ R M ×H×W where H, W are spatial dimensions and M is the number of channels. We assume the image x is generated by a multi-channel sparse code z ∈ R C×H×W deconvolving with a multi-dimensional kernel A ∈ R M ×C×k×k , which is referred to as a convolution dictionary. Here C is the number of channels for z and the convolution kernel A. To be more precise, we denote z asz . = (ζ 1 , . . . , ζ C ) where each ζ c ∈ R H×W is a 2D array (presumably sparse),EQUATIONwhere eachα ij ∈ R k×k is a 2D motif of size k × k.Then, for each layer of the generator, also called the decoder, g(z, η), its output signal x is generated via the following operator A(•) defined by deconvolving the dictionary A with the sparse code z:EQUATION)where n is some small isotropic Gaussian noise (modeling sampling or quantization errors etc.). For convenience, we use " * " and "⋆" to denote the convolution and deconvolution operators, respectively, between any two 2D signals (α, ζ):EQUATIONThe overall decoder g(z, η) is a concatenation of multiple such sparse deconvolution layers and the parameters η are the collection of learned convolution dictionaries A's (to be learned), as illustrated in Figure 1 . Only batch normalization and ReLU are added between consecutive layers, to normalize the overall scale of the features and to ensure positive pixel values of the generated images. Details can be found in Appendix A.An Encoding Layer as Convolutional Sparse Coding. Now, given a multi-dimensional input x ∈ R M ×H×W sparsely generated from a (learned) convolution dictionary A, the function of each layer of the encoder f (x, θ) is to find the optimal z * ∈ R C×H×W from solving the inverse problem from equation 4. Under the above sparse generative model, according to [65] , we can seek the optimal sparse solution z by solving the following LASSO type optimization problem:EQUATIONWe refer to such an implicit layer defined by equation 6 as a convolutional sparse coding layer. The reconstruction difference between x and A(z) is penalized by the ℓ 2 -norm of x -A(z) flattened into a vector.The optimal solution of z given A will be a close reconstruction of x. Sparsity is controlled by the entry-wise ℓ 1 -norm of z in the objective. λ controls the level of desired sparsity. In this paper, we adopt the the fast iterative shrinkage thresholding algorithm (FISTA) [5] for the forward propagation.The basic iterative operation is illustrated in Figure 1 . A natural benefit of the FISTA algorithm is that it leads to a network architecture that is constructed from an unrolled optimization algorithm, for which backward propagation can be carried out by auto-differentiation.Hence, the encoder f (x, θ) is a concatenation of such convolutional sparse coding layers. Recently, the work of [11] has shown that such a convolution sparse coding network demonstrates competitive performance against popular deep networks such as the ResNet in large-scale image classification tasks. Note that in the generative setting, the operators of each layer of the encoder f are determined by the same collection of convolution dictionaries A's as the decoder g. Thus, in the autoencoding diagram in equation 2, the parameters θ of the encoder f (x, θ) and η of the decoder g(x, η) are determined by the same dictionaries. As we will see, this coupling brings tremendous benefits to the learned autoencoder, even besides interpretability.The above explicit generative model has resolved the issue regarding the structure of the the encoder and decoder for the autoencoding:1 X f (x,θ) ----→ Z g(z,η)----→ X. It does not yet address another difficulty mentioned above about autoencoding: how should we measure the difference between X and the regenerated X = g(f (X, θ), η)? As we discussed earlier, it is difficult to identify the correct distance between (distributions of) images. Nevertheless, if we believe the images are sparsely generated and the sparse codes can be correctly identified through the above mappings, then it is natural to measure the distance in the learned (sparse) feature space.The recently proposed closed-loop transcription (CTRL) framework proposed by [12] is designed for this purpose. The difference between X and X can be measured through the distance between their corresponding features Z and Ẑ = f ( X, θ) mapped through the same encoder:EQUATIONTheir distance can be measured by the so-called rate reduction [14, 35, 68] : namely the difference between the rate distortion of the union of Z and Ẑ and the sum of their individual rate distortions:EQUATIONwhere R(•) represents the rate distortion function of a distribution. In the case of Z being a Gaussian distribution and for any given allowable distortion ϵ > 0, R(Z) can be closedly approximated by1 2 log det I + d nϵ 2 ZZ ⊤ .Such a ∆R gives a principled distance between subspace-like Gaussian ensembles, with the property that ∆R(Z, Ẑ) = 0 iff Cov(Z) = Cov( Ẑ) [35] .Ensuring Self-Consistency of Autoencoding via a Sequential Game. As shown in [12, 36, 47] , one can provably learn a good autoencoding by allowing the encoder and decoder to play a sequential game: the encoder f plays the role of discriminator to separate Z and Ẑ and g plays as a generator to minimize the difference. This results in the following maxmin program:EQUATIONThe program in equation 9 is somewhat limited because it only aims to align the dataset X and the regenerated X at the distribution level. There is no guarantee that for each sample x i would be close to the regenerated xi = g(f (x i , θ), η). For example, [12] shows that an input image of a car can be decoded into a horse; the so obtained autoencoding is not sample-wise consistent.A likely reason for this to happen is because two separate networks are used for the encoder and decoder and the rate reduction objective function only minimizes error between distributions, not individual samples. Now notice that for the new convolutional sparse coding layers, parameters of the encoder f and decoder g are determined by the same convolution dictionaries A. Hence the above rate reduction objective in equation 9 becomes a function of A:EQUATIONWe can use this as a cost function to guide us to learn dictionaries A which are discriminative for the inputs and able to represent them faithfully through closed-loop transcription. To this end, for each batch of new data samples, we take one ascent step and then one descent step. The first, maximizing, step promotes a discriminative sparse encoder using only the encoder gradient, and the second, minimizing, step promotes a consistent autoencoding by using the gradients of the entire closed loop.EQUATION)EQUATIONEmpirically, we find that in the step to minimize ∆R, taking the gradient as the total derivative with respect to the dictionary A, i.e., using the gradients through both θ and η, converges to better results than just using the gradient through η -see the ablation studies of Appendix B. As we will see, by sharing convolution dictionaries in the encoder and decoder, the learned autoencoder can achieve striking sample-wise consistency even though the rate reduction objective in equation 10 is meant to promote only distributional alignment.We note that this optimization strategy is different from the usual techniques for maximin games [47] ; this is because the encoder and decoder share the parameter A. Nevertheless our alternating steps have the same conceptual effects as they do in the usual optimization strategy, i.e., alternatively maximizing the encoder's power and the consistency of the autoencoding.To summarize, we explicitly model the distribution of natural images as being generated from a multistage sparse deconvolution model. This implies that the decoder g(•, η) should be a multi-layer sparse deconvolutional network. Thus, its inverse, the encoder f (•, θ), should be a multi-layer convolutional sparse coding network. This means that we can well-approximate the sparse code features z = f (x, θ) by a mixture of Gaussians, so we can efficiently measure the distance between two such distributions by ∆R in closed-form. For the whole process, we rely on our explicit assumptions about the generative model to derive the overall or intermediate objectives, the overall network architecture and layer-wise operators, and the final optimization approach. This contrasts heavily with all extant approaches, e.g. GAN, VAEs, and score-based models, which do not rely on such explicit generative models and instead rely on heuristic constructions for their networks. 
experiments: 
 We now evaluate the effectiveness of the proposed method. The main message we want to convey is that the convolutional sparse coding-based deep models can indeed scale up to large-scale datasets and regenerate high-quality images. Note that the purpose of our experiments is not to claim we can achieve state-of-the-art performance compared to all existing generative methods, including those that may have much larger model complexities and require arbitrary amounts of data and computational resources. 2 We compare our method with several representative categories of generative models, under fair experimental conditions: for instance, since our method uses only two simple networks, we mainly compare with methods using two networks3 , e.g., one for encoder (or generator) and one for decoder (or discriminator). Datasets and Experiment Setting. We test the performance of our method on CIFAR-10 [31], STL-10 [9] and ImageNet-1k [13] datasets. The detailed implementation settings and network parameters can be found in Appendix A.1 for CIFAR-10, STL-10 and ImageNet-1k.We adopt the standard FID [22] and Inception Score (IS) [53] to evaluate the generative quality of learned representations. We compare our method to the most representative methods from the following categories: GAN, VAE, flow-based, and CTRL, under the same experimental conditionsexcept that our method typically uses simpler and smaller models.On medium-size datasets such as CIFAR-10, we observe in Table 1 that, in terms of these metrics, our method achieves comparable or better performance compared to typical GAN, flow-based and VAE methods, and better IS than CTRL and VAE-based methods, which conceptually are the closest to our method. Comparing to CTRL, Figure 2 showcases the different reconstructed image between CTRL and CSC-CTRL. It is clear that CSC-CTRL not only enjoys better visual quality, but also achieves much better sample-wise alignment. Visually, Figure 3 further shows an amazing sample-wise alignment between input X and reconstructed X despite our method not enforcing sample-wise constraints or pixel-level loss functions! On larger-scale datasets such as ImageNet-1k, Table 1 shows that we outperform many existing methods in Inception Score. Figure 3 shows that the decoded X looks almost identical to the original X, even in tiny details. All of the images displayed are randomly chosen without cherry-picking. Due to page limitations, we place more results on ImageNet and STL-10 in Appendix C. To evaluate the structural properties of the learned feature space, we visualize the reconstructed samples along different principal components in the feature space of learned classes. We follow the procedure done in [12] , calculating the principal components of the representations in each learned class, and then reconstructing the samples with representation closest to these principal components. Each row in Figure 4 -4 ) principal components of learned features for ImageNet (class "rajidae", "goldfish", "chicken", "bird", "shark").To evaluate the generalizability of the learned model, we reconstruct samples of CIFAR-100 using a CSC-CTRL model which is only trained on CIFAR-10. Figure 5 shows a randomly reconstructed sample without cherry-picking. We observe that a lot of classes -for example, "lion", "wolf", and "snake" -which never appeared in CIFAR-10 can still be reconstructed, with high image quality. Moreover, if we visualize the samples along different principal components within the class, we see that even the variance in the out-of-domain data samples may be captured by computing the principal components. It demonstrates that our model not only generalizes image reconstruction well to out-of-domain data, but also encodes a meaningful representation that preserves diversity between and within out-of-domain classes. To test the stability of our method to input perturbation, we add Gaussian noise with mean of 0 to the original CIFAR-10 dataset. We use σ to control the standard deviation of the Gaussian noise, i.e., the level of perturbation. The property of the convolutional sparse coding layer makes possible a stable recovery of the sparse signals with respect to input noise and, therefore, enables denoising [15, 65] . Hence, CSC-CTRL's autoencoding also functions as denoising of noisy data. We conducted experiments on CIFAR-10, with σ = 0.5, and STL-10, with a σ = 0.1. Because CIFAR-10 has a smaller resolution, we use a larger σ so we can visualize the noise more clearly. From Figure 6 , we see that CSC-CTRL outputs a better-denoised image. When noise level are larger, CSC-CTRL has an obvious advantage over CTRL, which uses traditional convolutional layers. We also present more quantitative analysis of denoising in Appendix E.1. The CSC-CTRL model also demonstrates better stability in training than CTRL model. The coupling between the encoder and decoder makes the training more stable. For instance, the IS score of the CSC-CTRL model typically gradually increases and converges during training, whereas the CTRL model's IS score continuously drops after convergence. In addition, CSC-CTRL can converge with a wide range of batch sizes, from as small as 10 to as large as 2048, whereas CTRL can only converges with batch size larger than 512. These two properties are highly important from the perspective of engineering models within the CTRL framework. More details can be found in Appendix F. 
conclusion and future work: 
 In this work, we have shown the classic and basic convolution sparse coding models are sufficient to construct strikingly good autoencoders for large sets of natural images. This leads to a simplifying and interpretable framework for learning and understanding the statistics of natural images. This new framework integrates intermediate goals of seeking compact sparse representations with an end goal of obtaining an information-rich and yet compact & structured representation, measured by the coding rate reduction. The learned models have demonstrated unprecedented generalizability and stability. We believe this gives a new powerful family of generative/autoencoding models that can better support a wide range of applications that require more interpretable and controllable image generation and understanding.Notice that our current implementation is extremely basic and simple. There are rich variants to the convolution sparse coding layers that could promote different types of sparsity or low-dimensional structure in images [65] as well as variants to the rate reduction objective for the final feature representations [12] that could incorporate richer class-wise or sample-wise information. These variants have not been considered in this work. Hence there is ample reason to believe that the performance, scalability, and efficiency of this method can be significantly improved in the future with better engineering and implementation. 
 
 Appendix: Our formulation in terms of rate distortion is conceptually similar to score matching [25, 57, 64] . The score function is the gradient of the (un-normalized) log-likelihood, that is, ∇ x log p(x), whereas the expectation of the (negative) log-likelihood can be interpreted as the coding rate of the distribution ([10], Chapter 14). In the case of a distribution with degenerate (low-dimensional) supports, its density or log-likelihood (hence score function) is not well-defined. A natural surrogate is its (lossy) rate distortion R(x) subject to a prescribed quantization error [35] , which is well-defined even in these contexts. Thus, the rate distortion formulation extends the log-likelihood to high-dimensional (image) distributions with degenerate structures 4 . Hence the gradient of the rate distortion can be viewed as a surrogate to the score function. 5 In the case the distribution is (locally) approximated as a mixture of Gaussians, the score function can be efficiently computed in closed-form as the gradient of the Gaussian rate distortions. 6 The above rate reduction gives a closed-form formula for the distance between such two distributions.Each sparse deconvolution layer transforms its sparse input z into a denser representation A(z).One can view the concatenation of these layers, i.e., the decoder g(z, η), as carrying out multiple steps of an incremental deformation from a sparse code z to a dense and higher-dimensional x. This is conceptually analogous to the so-called diffusion process [23, 56, 57] . The main difference is that traditional diffusion models start at the structured high-dimensional image data x and, by incrementally adding isotropic Gaussian noise, diffuse to standard Gaussian noise. In contrast, we start with even more organized and lower-dimensional sparse codes z and, by adding anisotropic Gaussian noise (i.e. isotropic Gaussian noise pushed through the sparse deconvolution at each layer), diffuse it to get x. In this way, one may consider the decoder as conducting a form of "structured diffusion" or "anisotropic diffusion."In the inverse direction, each convolutional sparse coding layer extracts a sparse code z from a dense representation or image via LASSO regression against a learned convolutional dictionary of the distribution of natural images. Dictionary learning is in fact one of the purposes why the score matching was introduced by [25] in the first place. Thus one can consider our encoder, which is a concatenation of such layers, as carrying out multiple steps of an incremental deformation from dense and high-dimensional image data x to a sparse and lower-dimensional encoding z. This is conceptually analogous to the so-called Langevin dynamics process which is used in diffusion models [23, 56, 57] . The main difference between Langevin dynamics and our process is that the Langevin dynamics starts with standard Gaussian noise, and incrementally denoises to transform it to structured and high-dimensional natural images x. In contrast, we start with the structured natural images x and incrementally "denoise" (regress against the learned convolutional structures) to transform it to even more structured output, i.e., the compact codes z. In this way, one may consider the encoder as conducting a form of "structured denoising."To summarize, while diffusion models map to and from unstructured noise, our process maps to and from an explicit structure modeled by a learned sparse (de)convolution as illustrated in Figure 7 .n x z denoising s θ ( x, σ) = x-x σ 2 diffusion x = x + n structured denoising z = arg min z ′ ( 1 2 ∥A(z ′ ) -x∥ 2 2 +λ∥z ′ ∥1) structured diffusion x = A(z) + nFigure 7 : Connections and contrasts between "traditional" diffusion and our structured diffusion/denoising processes. While conventional diffusion and denoising process consider isotropic noise, our process consider generation and denoising against a learned (convolutional) dictionary A.The goal is to obtain a more, compact, structured (e.g. sparse) internal representation z.In this section, we justify our choice of optimization strategy to optimize equation 10. We set the following optimization strategy as "Strategy 1", which was adopted in the original CTRL:EQUATIONEQUATIONWe set the following optimization strategy as "Strategy 2", which is used in our work to optimize equation 10.EQUATION)EQUATION)We run an ablation study on CIFAR-10 with hyper-parameters all the same from Appendix A.1, except the training strategy. The results are shown in Table 6 . Empirically, we found that Strategy 2 optimizes much better than Strategy 1.Strategy 1 3.2 197.1 Strategy 2 8.9 28.9 Table 6 : Ablation study of CSC-CTRL on different optimization strategies through reconstructed image quality (IS/FID). ↑ means the higher the better. ↓ means the lower the better.Due to limited space in the main body, we show the generated images of STL-10 (see Figure 8 ) and some extra images of ImageNet (Figure 9 ) in this section. Figure 8 shows the auto-encoding properties of our learned framework on STL-10. Figure 9 shows a larger version of the reconstruction on ImageNet. We observe that even fine details in the image have been faithfully reconstructed, showcasing the power of our convolutional sparse coding network. Lastly, we include more generated images on ImageNet in Figure 10 , demonstrating the image quality of our network. Linear interpolation. Figure 11 shows reconstructed images whose features are linearly interpolated between pairs of images sampled from the class of "beach wagon" of ImageNet dataset (the class ID: n02814533). Formally, for two images x 1 , x 2 , the interpolated x is given byEQUATIONwhere α ∈ [0, 1] varies in Figure 11 from 0 (on the left side) to 1 (on the right side).The generated images show a continuous deformation from one sample to another. This verifies that our feature space is linearized and discriminative. In fact, we can get better denoising effect by simply adjusting the λ in the convolutional sparse coding layer in equation 6 without any additional training. Our default λ is set to be 0.01 due to the scale between two objectives during the training stage. In the inference stage, we can further increase λ to promote sparsity, which naturally leads to better denoising. From Table 8 , we see that as λ increases, CSC-CTRL generally improves at denoising. 8 : Comparison of denoising using different λ with standard metrics. ↑ means the higher the better. ↓ means the lower the better.In this section, we further verify the training stability of CSC-CTRL from two perspectives: mode collapse during training, and choice of batch size.Training Stability. Experimentally, many previous methods such as CTRL and various GANs suffer from training instability. As shown in Figure 12 , CTRL shows a clear training instability after 600 epochs. In contrast, CSC-CTRL training is much more stable, as the IS score barely drops. We conclude that CSC-CTRL suffers less from mode collapse.Choice of Batch Size. One notable flaw of the original CTRL [12] is its reliance on a large batch size, normally greater than 512. This large batch size greatly increases the model's computational cost and limits its scalability. In Table 9 , we compare whether each method converges under different batch sizes, from as small as 10 to as large as 2048. From the table, we see that CSC-CTRL can successfully converge on a wider range of batch sizes, even as low as 10. This greatly reduces the required computation power and enables easier training on more complicated datasets such as ImageNet. Although the effectiveness of the choice remains to be verified.For example, we will not compare with methods that require very large models such as Big-GAN[6] or NSCN++[57].Hence, we will not compare with methods that require multiple networks for additional discriminators such as the VAE-GAN[48] and the Style-GAN[27].This is crucial if one wants to identify such structures explicitly, which is precisely the purpose of this work. Note that this differs from almost all extant generative models.Geometrically, the score function, and hence the gradient of the coding rate, indicates directions in which the encoder can most effectively compress or expand the (local) volume of the representations, measured in terms of decreasing or increasing the coding rate.Such gradients are the basic layer-wise operators for the ReduNet[7]. 


Decision: The CSC-CTRL framework proposed by the authors is an interesting combination of convolutional sparse coding and a distributional loss function related to compression. The resulting autoencoders are more lightweight than the compared convolutional networks, while yielding perceptually good quality images and outperforming some of the comparable generative techniques proposed earlier. Overall it is intriguing to see that this convolutional framework can perform so well and yield strong sample-level encoding.

For balance, I also want to state some places where the manuscript could be improved. The "interpretability" aspect is in my opinion overstated. I don't think that the sparse feature representations are any more or less interpretable than feature maps in convnets, nor is the related generative process more interpretable than a convnet decoder. Since 2 iterations of FISTA are generally not enough to produce the _sparsest_ code, the feature maps here are _some_ sparse feature maps which are optimized end-to-end for generative performance.

Some of the arguments in responses seem to sidestep important issues—for example, why it is not necessary to compare with SOTA. Naming "engineering tricks" as culprits for performance differences sounds somewhat evasive, especially without a detailed elaboration, and acknowledging that the authors work with tiny images. Batch normalization and ReLUs in between the "principled" (de)convolution layers could similarly be declared engineering tricks (without a deeper discussion, at least). Finally, I doubt that CSC strategies can scale to high resolutions without explicitly accounting for regularity across scales and directions, which necessarily involves some kind of multiscale design. But it would be intriguing to be proven wrong. 

I agree with the reviewers that this paper is marginally above the acceptance threshold and urge the authors to incorporate the promised changes, including improved comparisons with prior art, in the camera ready version.

Reviews: review: title: title: Incorporating convolutional sparse coding in CTRL generative framework
main_review: main_review: **Summary of the proposed work:**

Conventional autoencoders typically use generic network architectures and lack structure or interpretability in the latent code representations. A recently proposed CTRL framework uses rate reduction to build generative models while imposing a Gaussian mixture model like structure in the latent codes.

The submitted work proposes CSC-CTRL in which the generation of natural images is modeled using a stacked concatenation of multiple convolutional sparse coding (CSC) layers and incorporated within the overall CTRL framework. The atoms in the convolutional dictionaries of the encoder and decoder are kept the same which allows for sample-wise alignment in the generated samples—an improvement over existing CTRL approach. 

The paper shows good image generation quality with interpretability in latent codes with respect to image classes in the dataset. There is additionally a higher stability to noise than vanilla CTRL owing to the incorporation of sparsity.

**Pros:**

- The results are nice and convincing with respect to the main ideas in the paper’s body.
- Barring a few minor typos, the paper is generally clearly written.

**Cons:**

I have the following questions/concerns that I would like the authors to clarify.

- The authors emphasize that the key goal is to show the feasibility of high quality image generation using CSC layers. While their results are interesting, there exist a few recent works which also use multi-scale CSC to generate high quality images, for example [1] and [2].  These focus on imaging inverse problems rather than autoencoding, though one could extend them to autoencoding by say tying the weights of their encoders and decoders.

There are indeed some differences between the proposed work and the 2 references, for example, [1] performs direct pixel level MSE loss minimization instead of distribution matching or CTRL. However, given similarities like CSC based high quality generation and interpretability in obtained dictionaries, a comparison (conceptual or experimental) would be helpful to better place the contributions of the submission with respect to literature.

- One of the advantages mentioned is that CSC-CTRL has reduced computational cost over prior models (line 48 in the paper). But wouldn’t the use of FISTA iterations in the encoder’s forward pass add extra overhead making it more expensive than a conventional autoencoder?   


[1] Liu, Tianlin, et al. "Learning multiscale convolutional dictionaries for image reconstruction." IEEE Transactions on Computational Imaging 8 (2022): 425-437

[2] Li, Minghan, et al. "Video rain streak removal by multiscale convolutional sparse coding." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
title: title: Important direction and a well-written paper. The method is principled but may lack practicality.
main_review: main_review: Summary:

This paper proposes an auto-encoding architecture that implements convolutional sparse coding layers with weighting sharing between the encoder and the decoder. Trained with CTRL, the model exhibits decent scalability, structured representation, and superior sample-wise alignment.

Merits:

1. The paper is well written, carefully introducing and discussing technical details.

2. The direction toward more powerful white-box models is relevant and essential in the era of large black-box models.

3. The proposed approach is well-grounded with principles. Each component has a clear interpretation, including the convolutional kernels shared by the encoder, the decoder, and the CTRL objective.

4.  The method exhibits stronger scalability and reconstruction performance than its precursor CTRL.


Disadvantages:

1. I am unsure about the experimental comparison with popular deep-learning generative models like VAEs and GANs. If I understand it correctly, the proposed approach can also perform reconstruction, whereas VAEs and GANs are designed to sample novel samples from the training distribution. Therefore, the comparison in Table 1 seems unfair (or not comparable) to me. Plus, the lack of the capability of sampling novel samples is undesirable in comparison.

2. The encoding process involves iterative algorithms, which may significantly slow the training and induce numerical instability. On this aspect, it would be appreciated if the authors could offer training time reports contrasting the evaluated approaches.
title: title: Well-written paper with convincing experiments
main_review: main_review: This paper introduces a new autoencoder design that uses convolutional sparse coding (CSC) and trains it using the closed-loop transcription (CTRL) method. The resulting generative model performs well in both image reconstruction and generation.

Pros:
1. Clear writing: The paper is easy to understand, and the experiments are well explained and supported.
2. Good literature review: The related works section covers the most relevant research in the field.
3. Strong experiments: The experiments are convincing, with comparisons to other models that make the proposed approach look solid.

Cons:
1. Discuss weaknesses: The paper should discuss its limitations more clearly.
2. Higher resolution experiments: Please test the model on higher-resolution datasets like 128x128 if possible, or explain why not and how it compares to the baselines in this regard.
3. Include timing information: Please clarify how long the training and inference take compared to the baselines.
4. Reconstruction quality metrics: Please use metrics like PSNR or SSIM to measure image quality in Figures 2, 3, and 5.
5. Please show the number of trainable parameters for different models in Table 1. You can also add training and inference times to this table.
6. Please Include the original images in Figure 5 so readers can see how well the model recreates them.
7. Please make Figure 4 clearer by adding more space between blocks and brief captions for each block.

I believe this is overall a decent paper. The proposed framework is promising but should address these concerns to improve its overall quality and clarity. These improvements could lead to a higher evaluation score.