Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes to replace the encoder and decoder with standard convolutional sparse coding and decoding layers in an autoencoder. The proposed approach can be trained on high-resolution images and can be used to reconstruct images and capture interpretable representations. 
strength_and_weaknesses: **Strength:**
This paper has many experiments.

**Weakness:**
1. Some of the arguments are overclaimed. For instance, in Introduction: "The learned autoencoder achieves striking sample-wise consistency". Not sure if the performance is really that impressive.

2. "our method scales well to large datasets": it is unclear what "scales well" mean. Does it mean the model is able to be trained on large datasets, or achieve good performance?

3. As mentioned in Section 4, "The main message we want to convey is that the convolutional sparse coding-based deep models can indeed scale up to large-scale datasets and regenerate high-quality".  Not sure if the contribution is sufficient. At the same time, the reconstructed images do not seem to be very impressive compared to many other generative models like diffusion models, and normalizing flows.

4. The experiment settings are poorly explained. The model's performance is also very limited.  

5. In section 4.1, besides showing the reconstruction qualitatively in figure 2 and 3, it would be more convincing to evaluate the reconstruction qualitatively and compare it with the baselines.

6. Table 1 is very confusing. Are the IS and FID scores evaluated on samples or reconstructions? If they are evaluated on samples, then there should be more details on sampling from the proposed model. Currently, I do not see the sampling algorithm in the paper. If they are evaluated on reconstructions, then I am not sure if the comparison with the GAN, VAE, flow baselines are fair (since they are evaluated on synthetic samples).

7. The figures and their captions are confusing. For instance, for figure 4-6, it would be much better to label the x, y-axis and each block to clarify what each image is.

8. It is unclear why the performance in figure 6 is impressive: many autoencoders will also generalize to a different datasets due to some inductive bias. Especially since CIFAR100 is relatively similar to CIFAR-10.

9. The denoising experiment in figure 7 does not seem to be working.
 
clarity,_quality,_novelty_and_reproducibility: The paper's clarity needs to be improved, The experiment settings are not properly explained, and the motivation is not very clear.
summary_of_the_review: Given that the paper's clarity is limited and the experiments are also not very convincing, the paper needs to be improved for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model for image generation which combines convolutional sparse coding (CSC) with the closed-loop transcription (CTRL) framework by Dai et al (2022). The encoder and decoder of this CSC-CTRL model share dictionaries of convolutional kernels at each sparse coding layer. The encoder produces a sparse code at each CSC-layer by unrolling the FISTA algorithm. The decoder applies deconvolution to the final output of the encoder and each successive CSC-layer output to reconstruct the original input. The system is trained with a rate reduction objective which aims to minimize the distance between the distributions of codes produced from the original inputs and those produced from the reconstructed inputs. The proposed model is trained on CIFAR-10, STL-10, and ImageNet-1K, and achieves comparable or favorable performance when evaluated against GAN, VAE, Flow, and CTRL methods on metrics such as Inception Score and FID. Additional analysis suggests that the feature space of the proposed model preserves class information even though it is trained with an unsupervised objective. Further experimental results suggest that the CSC-CTRL model generalizes to data from classes not seen during training, is robust to noise, and is more stable than the CTRL model when trained with different batch sizes. 

strength_and_weaknesses: **Strengths**

- The proposed architecture combining convolutional sparse coding with closed loop transcription is novel.

- The proposed model is compared to and in many cases performs better than existing generative models of similar capacity on popular evaluation metrics.

- Qualitative analysis shows that images generated by the CSC-CTRL model have high visual fidelity, both sample-wise and distribution-wise.

- The paper is generally written clearly.

**Room for improvement**

- *ImageNet scale*: I believe the statement in the abstract that the CSC-CTRL model “scales up to ImageNet” might be misleading since Appendix A.2. states that the ImageNet images are resized from 224x224 to 64x64. Would the authors please clarify what ImageNet image size is used during training? 

- *Statistical significance*: It would strengthen the paper’s argument to include confidence intervals in Table 1.

- *Generalization to Unseen Datasets* and *Stability of CSC-CTRL*: It would complement the qualitative analysis in figures 5 and 7 to provide quantitative analysis such as average PSNR the model achieves on: CIFAR-10 (which it is trained on) versus PSNR on CIFAR-100 in Fig 5; the average PSNR of reconstructions of the original versus noisy inputs for both the CTRL and CSC-CTRL models trained on CIFAR-10 and STL-10 (evaluated on a held-out test set). 

- *Clarification on equations (9) and (10)*: In my understanding $\theta$ and $\eta$ (the encoder and decoder’s parameters) are the same since they are determined by the weights of the shared convolutional dictionaries. Is my understanding correct?

- *Related literature*: I believe it would be helpful to include reference to either [1, 2] in the main text.

- *Visualization of convolutional filters*: It would be informative to display the learned convolutional filters which is a commonly adopted practice in the related literature. 

- *Reproducibility*: It would be helpful if the authors release their implementation for reproducibility.

[1] Zeiler, M.D., Krishnan, D., Taylor, G.W. and Fergus, R., 2010, June. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition (pp. 2528-2535). IEEE.

[2] Zeiler, M.D., Taylor, G.W. and Fergus, R., 2011, November. Adaptive deconvolutional networks for mid and high level feature learning. In 2011 international conference on computer vision (pp. 2018-2025). IEEE.

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The idea of combining convolutional sparse coding with closed loop transcription is novel to the best of my knowledge. The implementation details are provided in the appendix. Having the implementation code would help reproduce the results presented in the work.

summary_of_the_review: Overall, this work presents an interesting idea of combining convolutional sparse coding with closed loop transcription for image generation. The model is compared to generative models of similar or higher capacity and performs favorably in many cases. The qualitative results on the structure of the feature space, generalization, as well as robustness to noise are insightful. I pose some recommendations on ways to strengthen the statements in the paper by including clarifications and more analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an autoencoder architecture whose encoder and decoder are constructed based on the convolutional sparse coding generative model. The encoder is a convolutional sparse coding layer solving lasso (sparse coding problem) by unrolling FISTA (fast iterative thresholding algorithm). The decoder is dictated by the sparse coding/dictionary learning model. In place of the usual reconstruction loss, the authors use closed-loop transcription (CTRL) to train the network, i.e., to maximize the rate reduction of the learned sparse codes. To learn the dictionary, they use a max-min optimization problem for the encoder to discriminate between image encoding and reconstruction encoding and for the decoder (generator) to minimize the difference.

Here are their contributions from their perspective: The paper argues that they are the first to scale up the sparse coding model and train it with ImageNet. They argue that one of the benefits of their network architecture is having interpretable and structured representation. This arises due to their sparse coding encoder architecture and their training approach of maximizing the information gain. With the usage of the CTRL approach to train the framework, they offer sample-wise alignment. They emphasize that the construction of a network-based of sparse coding model offers efficient model and training. They focus on the image generation quality of the framework and how it generalizes.

Here is their contribution from the reviewer's perspective: The paper combines two existing ideas of unrolled networks and CTRL training. They show the reconstruction/generation performance of the method.
strength_and_weaknesses: strengths:

The paper was easy to read. It is organized, and I could follow the arguments. The experiments seem reproducible (enough details are provided in the appendix). There are reasonable visualizations and characterizations of their method. However, given my concerns explained below, I do not recommend an acceptance.

weaknesses:

- Although this paper combines two interesting ideas, it lacks novelty. To elaborate, the paper does not offer new knowledge. The ideas are: constructing autoencoders based on an optimization model (in this case the deep generative sparse coding model), and using CTRL to train the network to boost up image generation quality of the network.

- The literature review is not thorough. For example, unrolling optimization algorithms are used abundantly in the literature. [1] is a review paper about this approach and its wide usage in image and signal processing applications. For networks-based convolutional sparse coding, [2,3], among the many, is missing; the works show how to construct a recurrent autoencoder based on a shallow sparse coding model and train it for image denoising. Both of these papers have already shown the computational and trainable parameter efficiency of unrolling approach and have shown the competitive performance against SOTA at the time of their publications.

- Compared to other image generation methods, it is not clear what this paper is offering. A thorough discussion and comparison with SOTA are missing. Or a discussion on why SOTA comparison is not needed. This is especially needed as the proposed method is a combination of two already existing approaches. Moreover, this is particularly important, as the authors argue that they are the first to scale up CSC.

- Lack of baselines: Although I enjoyed looking into the reconstructed images, baselines are lacking for several visualizations. The principal approach for image generation can also be applied to other generating architecture. How is their performance (e.g., Figure 4,6)? For example, in Section 4.3, the authors argue that their framework generalizes to autoencoding unseen datasets. Aren't other autoencoders able to do this? If yes/no, please include such discussion and comparison (e.g., for Figure 5).

- Interpretability: the authors emphasize that their framework has the benefit of interpretable representation. However, they do not define interpretability. Indeed, there are many definitions of interpretability [4], and elaboration is needed here. Regarding the interpretability of deep networks constructed based on the generic sparse coding model, [5] is missing in their literature review. Certain questions arise about the interpretability, the kernel size in the trained architecture is too small. Hence, in terms of learned features, their kernel seems rather small to learn human interpretable features. One question is how the interpretability of their framework is useful.

- The paper confuses two concepts of "image generation" and "image reconstruction". In many of the examples, for example, Figure 2, the authors are visualizing the network reconstruction (pass the image into the encoders, and then decoder). This does not image generation; the code is created by passing an image into the encoder.

Some questions and recommendations.

- In Section 4.2, the authors say that "... reconstructing the samples with representation closest to these principal components". Where are the samples coming from? From my understanding, there seems to be a set of encodings from a set of images. The method is using those encodings that are closest to the principal components of the representations to reconstruct an image. Again, there is no sampling as in GANs. This is not an image generation, and looking into PC of a set of representations can be applied to other autoencoders. Please elaborate.

- What is the significance of large datasets that the author emphasizes if the goal is not classification, but image reconstruction? At the end of the day, all networks are being trained by gradient descent (hence not all data is going to be used at once for parameter updates). I do find the contribution of the fact that they are the first to train on imagenet-1k for image reconstruction (not classification) to be a minor contribution.

- I recommend adding SOTA to Table 1, to give the reader a perspective on how far the performance is from SOTA. Perhaps to help you with your arguments on scaling up csc but not comparing to SOTA, I suggest including an extra column to report on network parameters and computational efficiency.

- The paper frequently makes statements that are not precise and need elaboration. For example, prior to Section 3, it states "... have several good benefits unknown to any of the previous generative methods". Please be precise and explain the benefit and specify prior works.

- Please report the number of parameters to support the claim on efficiency.

Minor

1. Page 2, bullet 1, citation is missing for previous sparse coding works.
2. The authors emphasize much on convolutional and miss the generic sparse coding literature (at the end of the day, convolution is a linear operator with structure). In the unrolling literature, there are many theoretical works on unrolled sparse coding that may worth to be cited. Here are two [6,7].
3. replace "." with "," on (4).
4. It would be nice to provide an additional explanation on CTRL. For example, their method from the perspective of fixed points.
5. Suggest changing \lambda_max and \lambda_min in (11) and (12) to avoid confusion with \lambda in the lasso.
6. please cite lasso [8] before (6).
7. Please report the noise \sigma of the image in the scale of a/255. What is a? (Section 4.4)
8. There are certain wordings not supported in the paper. Please remove or support. E.g., in conclusion, the usage of "convincing" and "unprecedented".
9. Figure 8, if (a) and (c) are similar, please combine. same for (b) and (d).
10. I liked the stability discussion from the perspective of the connectivity of the encoder and decoder. If space allows, I recommend moving the losses from the appendix into the main paper.

[1] Monga V, Li Y, Eldar YC. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine. 2021.
[2] Simon D, Elad M. Rethinking the CSC model for natural images. Advances in Neural Information Processing Systems. 2019.
[3] Tolooshams B, Song A, Temereanca S, Ba D. Convolutional dictionary learning based auto-encoders for natural exponential-family distributions. In International Conference on Machine Learning 2020.
[4] Doshi-Velez F, Kim B. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. 2017.
[5] Tolooshams B, Ba D. Stable and interpretable unrolled dictionary learning. Transaction in Machine Learning Research. 2022.
[6] Chen X, Liu J, Wang Z, Yin W. Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. Advances in Neural Information Processing Systems. 2018.
[7] Ablin P, Moreau T, Massias M, Gramfort A. Learning step sizes for unfolded sparse coding. Advances in Neural Information Processing Systems. 2019.
[8] Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological). 1996.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. However, there are certain statements that need clarification. See my comments above. Quality is ok. However, the paper is not original. See my comments on novelty above.
summary_of_the_review: The paper lacks a clear novelty. It combines two ideas and the majority of their findings are already known in each of those ideas (unrolling optimization method to create autoencoder, and CTRL to train for image generating autoencoder). Aside from novelty, the method although argued to be scaled up of CSC, still does not choose SOTA as a baseline. Literature review missing key papers. See my review above.


--------------

My original score was 5. I increased it to 6 after a long discussion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an interesting approach to design autoencoders based on convolutional sparse coding. In this framework, each layer-wise operation is replaced by an approximate solution to a Lasso problem, with a convolutional dictionary. The reconstruction ('decoding') from the so-obtained sparse codes is given simply by a linear model. The authors train the resulting architecture with a recently proposed method, or loss function, termed Closed Loop Transcription (CTRL), which optimizes the rate reduction of encoder and decoder in a minimax formulation. The paper presents empirical results on CIFAR 10, CIFAR 100 and Imagenet 1K.

strength_and_weaknesses: This paper is interesting and represents a valuable contribution. Indeed, it is nice to see models based on CSC providing such nice results. The paper is also clear and easy to read.

This reviewer does however have a few comments and concerns:

Main comments:

1. This main paper has very few details about the implementation or on how these networks are implemented, and all of these are deferred to the appendix. While this is mostly fine, it would be nice to have some clarifications. Chiefly: from the description of the decoder, it seems like every decoding process of each layer is completely linear. Is this correct? If this is the case, then is the entire decoder (for all layers) completely linear? If this is the case, can the authors explain why not to replace the product of all these linear layers with simply a (single) linear layer, as they can indeed be made equivalent?

2. An important question refers to the way the authors optimize the problem in Eq. (9). The authors parameterize two players (encoder and decoder), and pose this as a max-min problem. Both networks are parameterized by the dictionary parameters, $A$. Now, instead of optimizing this problem in this way, in the line immediately following Eq.(12), the authors comment that, empirically, they find it beneficial to modify one of the first order updates in some pretty ad-hoc way. They also comment in passing that they include an ablation study on this choice in Appendix B. Appendix B indeed shows the comparison of their performance with their 'original' motivated method (strategy 1) versus the 'ad-hoc update' version (strategy 2). My issue with this is that Strategy 2 does not just improve a little their method giving them a small boost -which would be reasonable- but instead fully determines the competitive advantage of their method: Their strategy 2 performs almost an order of magnitude better (3.2 vs 8.9 in IS, and 197 vs 28.9 in FID) for CIFAR-10. Moreover, the "worst" performing method that they compare with (VAE) achieve 5.2 (IS) and 56 (FID), which is about 3 times (say, on average) as good as their strategy 1. 
In an empirical paper like this, it is natural to have some small details of the implementation that can provide a small improvement. Yet, from these numbers, it appears as this little ad-hoc trick is responsible for their method becoming "state-of-the-art", and I think that as such it should deserve some closer study.

3. It is also very strange that their method provides sample-wise alignment: indeed, as the authors mention, the entire network is trained on a loss that looks at statistics of the distribution of (real and generated) samples. Thus, it indeed need to provide matched samples, as shown by [Dai et al, 2022]. In this work, the authors seem to use the same loss and training framework, but all of a sudden they can reconstruct matched samples. How come? 

4. While the contribution of this paper is interesting and relevant, in my humble opinion and respectfully, the paper is a bit on the verbose side and some comments are a bit too subjective or exaggerated. For instance, in a couple of places, the authors comment that their methods shows "striking performance on large datasets", or "splendid visual quality". It is true that their IS and FID numbers are better than SNGAN and CTRL in ImageNet, but looking at the generated images (e.g. in Fig 3d and Fig 9b), I would not qualify their quality as "striking" or "splendid" (plenty of artifacts, both shape-wise and color-wise). 

5. To the best of my knowledge, the first work in relating sparse coding networks to deep networks, and even suggesting replacing each convolutional layer by a lasso formulation -as done in this work- is the work by Papyan et al, 2017, in JMLR. I think credit is due to that work for some of the ideas appearing here. 

I enjoyed reading this paper, and I look forward to reading the authors' responses to clarify my understanding.


Small comments:

- Why do the authors refer to (de)convolution layers? Looking at their definitions, what they refer to "(de)convolution" is just a cross-correlation operator - which is just the adjoint of a convolution operator. 
 
- I'm curious why the approach chose to abbreviate "closed-loop transcription" as "CTRL". Did the authors perhaps meant "CLTR"?
clarity,_quality,_novelty_and_reproducibility: See my comments above.
summary_of_the_review: Interesting paper with potentially valuable contribution, and with some elements that need further clarification.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a convolutional sparse coding auto-encoding model, where the encoder is a convolutional sparse coding layer solving lasso (sparse coding problem) by unrolling FISTA, and the decoder is dictated by the sparse coding/dictionary learning model. Instead of the usual reconstruction loss, the authors use closed-loop transcription (CTRL) to train the network. They show the reconstruction/generation performance of the method. In summary, the paper combines the idea of convolutional sparse coding and CTRL.

strengths:

The paper was easy to read and performed ablation studies. There are reasonable visualizations and characterizations of the method.

weaknesses:

Reviewers had concerns on the paper over-claiming the advantage in generation, while the benefit was mostly shown for the reconstruction task, and the "sampling" procedure is not the same (or as natural) as compared methods. The reviewers also think the experimental setup is problematic in using very small filters which hamper the interpretability of the method, and that the mixed results on denoising and out-of-domain reconstruction. The paper combines two existing approaches (with CTRL being more recent): the sparse coding part does involve a L2 reconstruction loss, while CTRL tries to avoid measuring reconstruction error in image space.  Thus there seems to be a contradiction between the principles of the two parts, making the combination not very natural at first look. The authors may need strong theoretical motivations to reconcile them. 
justification_for_why_not_higher_score: The combination of two existing approaches needs stronger intuition/justification. Showing mostly reconstruction quality is not very interesting. Reviewers had concerns on the experimental setups.
Decision: Reject