Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work visualizes image generation as sequential low-rank approximations to the weight matrices in a FWP(Fast Weight Programming) setup. The motivation is to not generate high fidelity images but to rather enable human interpretable visualizations of the weight updates in a Neural Network (NN). The authors have shown human comprehensible weight update sequences in six popular datasets. 


strength_and_weaknesses: *Novelty and Originality*

The work is interesting in it being one of the few works that draw similarities between images and weight matrices that are generated by another neural net. The recent of this type being the class of Implicit Neural Representations (INR) [1 ]. The work also is a a slightly novel extension to the existing literature on FWP to image generation. 


*Methodology*

The authors claims that image generation is not the best of the strengths of FPA, leaving the visualization of weight updates as the central purpose of the work. Given the poor generation quality of FPA, my question is if the  performance vs interpretability/explainability tradeoff presented in the work is worth a shot. Can the method be adapted to exercise control over generations by manipulating the low-rank updates, ref [4] or restrict the low-rank updates to find discriminating components, ref [5], instead of a rank based updation approach. 

Why shouldn't  one compare FPA with recurrent painting techniques like Draw [2].

The authors posit that FPA does not include any explicit inductive bias as noted in page 2, "*we show that our
generic models can generate images of respectable visual quality without any explicit inductive
bias for image processing (e.g., no convolution is used).*" But the training involves discriminator of LightGAN [3] that uses convolutional layers. 

From Table 2, the number of train steps are directly proportional to the generation quality. Is it due to having a softmax over the key as discussed in Section 4.2 that forces slow update rules. In my understanding, and the authors' note, "*The first thing we observe is that for many steps, the “key” is almost one-hot (which is encouraged by the softmax), i.e., the part of the image is generated almost column-wise.*" the key is crucial in determining the attention or the generative patterns for the next iteration. Have the authors explored on other activations for key matrices. 

Could the authors explain why there is more drastic effect of trains steps on the AFHQ dataset when compared to MetFaces dataset (as shown in Table 2). Have they explored the possible effects of the rank of an image on the generation process. 



References:

1. Skorokhodov, Ivan, Savva Ignatyev, and Mohamed Elhoseiny. "Adversarial generation of continuous images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
2. Gregor, Karol, et al. "Draw: A recurrent neural network for image generation." International conference on machine learning. PMLR, 2015.
3. Liu, Bingchen, et al. "Towards faster and stabilized gan training for high-fidelity few-shot image synthesis." International Conference on Learning Representations. 2020.
4. Zhu, Jiapeng, et al. "Low-rank subspaces in gans." Advances in Neural Information Processing Systems 34 (2021): 16648-16658.


clarity,_quality,_novelty_and_reproducibility: *Clarity*

The paper is mostly clear to read. But would appreciate a more vivid view of the FPA generator architecture and the role of a third NN to map z to initial hidden states of RNN layers of the slow NN.


*Reproducibility*

The work builds upon the existing architecture and has given links to the versions of the implementations used in the work. The work should be fairly reproducible given the code for FPA generator is released

*Novelty and Originality* (repeated from above)

The work is interesting in it being one of the few works that draw similarities between images and weight matrices that are generated by another neural net. The recent of this type being the class of Implicit Neural Representations (INR) [1 ]. The work also is a a slightly novel extension to the existing literature on FWP to image generation. 


 
summary_of_the_review: Overall, I appreciate the novelty in equating weight updation to image generation in FWP framework. The paper is clearly written and mostly easy to follow. My main concerns are with the limitations of the proposed approach in terms of its scalability w.r.t image size and to its applicability beyond the visualization of weight updates as discussed under the first point in the main review under methodology.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Fast weight programming sequentially generates a weight matrix of a neural network from another neural network, which matrix is generated as the summation of outer products of self-invented keys and values. 
This paper employs the idea of fast weight programming in the image generation task,  which results in fast weight painters (FPAs).
FPAs learn to generate images following a sequence of delta learning rules and adding rank-one components to existing images at each step. 
The authors train FPAs in the GAN framework in 64x64 resolution and evaluate it on various image datasets. 
While the performance largely lags behind the StyleGAN2,  authors claim this method allows for visualizing the iterative procedure of complex connections patterns in synaptic learning rules. 
strength_and_weaknesses: **Strength**:
1. I like the core idea of this paper and feel it is quite interesting. Images are generated as the summation of a sequence of rank-one matrices following the delta learning rules, whose matrices are outer products of key/value invented from the neural network. 
The self-invented value/key has subtle connections to transformers, and the iterative updating mechanism is intuitively related to diffusion models,  where adding new ranks is a kind of "denoising." I feel this paper may have the potential to inspire new image-generation methods.
2. The writing is good, and the paper is easy to follow. 

**Weakness**
Although the core idea is cool, my concern is: the current paper doesn't provide enough intriguing results/contributions, which are listed below. 
1. **Using image generation task to understand the weight generation process.** One motivation for this paper is to "visually illustrate the behavior of an NN that learns to execute sequences of learning rules." Visualizing weight generation sequence is hard to understand, so authors visualize image generation. 
I am not convinced by this statement since image generation and weight generation are quite different, and it is unclear to me how much visualization/observation obtained from visualizing image generation could be transferred/generalized to weight generation. 
Moreover, observations from visualization(Sec 4.2, Appendix A) are quite specific for each dataset, which doesn't reveal any genetic observations/ideas. 
Some experiments, like summarising the genetic properties of the generation process and validating it on different domains, would support this claim. Otherwise, I personally think it is specific to image generation tasks.
2. I understand "Clearly, our goal is not to achieve the best possible image generator (for that, much better convolutional
architectures exist). Instead, we use natural images to visually illustrate the behavior of
an NN ..." But I feel more solid experiments/design exploration might improve this paper. 
E.g., the current image generation results have a big gap to stylegan 2 (which is fine for me), and are limited to 64x64 resolution. I wonder, does it still work in 128x128 or even higher resolution? 
What if inventing several key/value pairs at each step and calculating multi-rank matrices?
What if updating is conducted on each image patch instead of the whole image? 
Why UNet's induct bias is so helpful?  
It is not necessarily a weakness but adding more design exploration/analysis would be helpful to illustrate the power of the proposed method as an image generation method. 


In summary, as a paper visualizing the generation process, weakness 1 would be my main concern. From the image generation perspective, I am worried about the current performance and experiments. 
clarity,_quality,_novelty_and_reproducibility: The writing is clear,  and overall reproducibility and novelty should be fine. 
More training details and network architecture would be appreciated. 
summary_of_the_review: This paper is interesting, at least for me.
My major concern is: the observation from the image generation is dataset specific and not helpful in understanding the generic process.
As an image generation method, it is not powerful and well-designed enough.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an image generation model called fast weight painters (FPA), applying the technique of fast weight programmer. The proposed method treats the generated image as the generated weight matrix.

1. This work seems to be one of the first attempts to apply fast weight programmer method to image generation task, which is novel.
2. The iterative optimizations of image generation also provide a good interpretability.
strength_and_weaknesses: Pros:
1. The paper is well-written.
2. The proposed method is technically sound.
3. It is novel to introduce fast weight programmer method into image generation task.

Cons:
1. The generation results is not comparable to SOTA image generation method, although I acknowledge this work is an attempt to introduce fast weight programmer into image generation task.

2. It is good to see the decomposition of image generation process step-by-step, however, the author didn't provide enough illustration or explanation for the visualization results. 

3. Lack of high-resolution image generation results.
clarity,_quality,_novelty_and_reproducibility: 1. The author has provided source code.
2. The proposed method is novel as it introduced fast weight programmer method into image generation task.
3. The paper is easy to follow.
summary_of_the_review: Overall, the proposed method looks novel to me, although not flawless as mention above. I would recommend to accept this paper.

UPDATE: After discussing with other reviewers and AC, I decide to lower the score. We agree the proposed method is novel and interesting. But as I mentioned the the review, the paper needs more explanation for the visualization results. And the author should also demonstrate why it is important to the community. We believe the paper could be more solid if the author could address these issues.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores extending the idea of fast weight with outer-product based to human-interpretable domains beyond network weight space, and examines the idea with the example of natural image generation. The slow-weight network, now termed Fast Weight Painter now progressively refines the pixel values of a 3D image tensor with one rank-1 updating at a time step. Fair generation results can be observed without imposing any domain-specific inductive bias, and further performance improvements can be achieved by U-net-based one-step denoising. 
strength_and_weaknesses: **Strength**

This paper demonstrates a very strong case for extending the fast weight idea in parameters space to other modalities like natural images. 
While it is expected to see a lower performance compared to domain-specific methods, the authors show how to further close this gap by exploiting more inductive bias in the image domain. 


**Weaknesses**

1. Figure 2 can be further improved by simply annotating everything in Eq 4-7 in the figure. 
Now I'm very confused by the arrow pointing from 'Current image' to 'Value' and the one pointing from 'Key' to 'Current image'. What are they referring to?

2. While I understand this paper is just trying to illustrate a case of extending the idea of fast weight to more domains, more experiments certainly help to show how generic the proposed method is. Currently, there are only results with *image* + *GAN training*, while a quick experiment with other data domains like audio or training methods like VAE or diffusion can be a plus. 



Minor points:

1. 
In the Sequence Processor paragraph, *we did not manage to train any models successfully when the standard Transformer was used instead.* Does it mean transformers completely fail in this case? I don't see a clear reason why transformers cannot work at all here. 

2. Based on the 2nd point in Weaknesses, I am very interested to see a synergy between the proposed method and diffusion, as both are iterative.  


clarity,_quality,_novelty_and_reproducibility: While some figures can be further improved, this paper is presented in a very clear way supported by strong empirical results. 
It is a novel idea on representing data in a domain-agnostic way. 
I do not notice any reproducibility issues. 
summary_of_the_review: This paper presents an interesting idea with strong but insufficient empirical results. 
Please refer to Strength And Weaknesses for detailed comments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an application of fast weight programmers to image generation tasks, termed fast weight painters (FPAs). FPAs are trained in a GAN framework on a variety of image generation datasets. Although FPAs do not achieve state-of-the-art performance on image generation, the key result of the paper is to demonstrate that this generic learning rule can produce reasonable results on this task.

Initially this paper received borderline reviews. Reviewers understood that the goal of the paper was not to achieve state-of-the-art image generation, and generally found the paper interesting, and a novel approach to image generation. Reviewers raised concerns with many details of the paper which were mostly resolved by the author’s responses, and after the rebuttal period the reviewer scores were generally positive.

The AC met with Reviewers oERk and 81gc to discuss the paper in more detail. Both reviewers found the approach novel and interesting, but had some borderline feelings about the paper. Overall, they felt unclear about the positioning of the paper, following the discussion that Reviewer oERk had with the authors during the discussion phase. The paper demonstrates that FPA can be used for image generation, but it does not outperform existing image generation methods. Per the author’s discussion with Reviewer wMfX, the proposed algorithm is also not intended to trade off image generation performance vs interpretability. Overall, the reviewers could not see any clear advantage of the proposed algorithm over existing methods for image generation, apart from using a different architecture. It is also not clear whether the results of the paper offer insight into fast weight algorithms that could be applied to other tasks; indeed in discussion with Reviewer oERk the authors clarify that they “do not intend to draw any generic conclusion” and “can not tell whether this observation is helpful for future studies.”

The AC agrees with the sentiments raised by Reviewers oERk and 81gc. The paper would be much more convincing if the authors could either (1) give some clear reason why sequential learning might be preferred to existing image generation algorithms; or (2) use this as a case study to bring some new insights into sequential learning rules generally. For (1), FPA need not outperform existing methods, or generate high-resolution images; but there should be some argument or reason why this algorithm is interesting for image generation, other than its novelty.

However due to the novel and interesting approach to image generation, the AC believes that the submission would be appreciated by the ICLR community. The authors are encouraged to revise their paper for the camera-ready version, incorporating feedback from the reviewers to clarify and improve the positioning of the paper.
justification_for_why_not_higher_score: The paper essentially shows that fast weight programmers can be used for image generation, but does not give any clear reasons why it should be preferred over existing algorithms, or show that it has unique properties with regards to other approaches. It also does not give any clear insight into fast weight programmers more generally, or give insights that could be used when applying this approach on other problems.
justification_for_why_not_lower_score: A majority of reviewers felt the paper should be accepted.
Decision: Accept: poster