Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules
Authors: Kazuki Irie
Jürgen Schmidhuber
Venue: ICLR 2023 poster (2023)
Keywords: learning rules
Fast Weight Programmers
linear Transformers
image generation
GANs

Abstract: Work on fast weight programmers has demonstrated the effectiveness of key/value outer product-based learning rules for sequentially generating a weight matrix (WM) of a neural net (NN) by another NN or itself. However, the weight generation steps are typically not visually interpretable by humans, because the contents stored in the WM of an NN are not. Here we apply the same principle to generate natural images. The resulting fast weight painters (FPAs) learn to execute sequences of delta learning rules to sequentially generate images as sums of outer products of self-invented keys and values, one rank at a time, as if each image was a WM of an NN. We train our FPAs in the generative adversarial networks framework, and evaluate on various image datasets. We show how these generic learning rules can generate images with respectable visual quality without any explicit inductive bias for images. While the performance largely lags behind the one of specialised state-of-the-art image generators, our approach allows for visualising how synaptic learning rules iteratively produce complex connection patterns, yielding human-interpretable meaningful images. Finally, we also show that an additional convolutional U-Net (now popular in diffusion models) at the output of an FPA can learn one-step "denoising" of FPA-generated images to enhance their quality.
Our code is public.

Published as a conference paper at ICLR 2023
IMAGES AS WEIGHT MATRICES: SEQUENTIAL IMAGE GENERATION THROUGH SYNAPTIC LEARNING RULeS
Kazuki Irie' Jürgen Schmidhuber'
'The Swiss AI Lab, IDSIA, USI & SUPSI, Lugano, Switzerland 2 AI Initiative, KAUST, Thuwal, Saudi Arabia
(kazuki, juergen Gidsia.
ABSTRACT
Work fast weight programmers has demonstrated the effectiveness of key/value outer product-based learning rules for sequentially generating a weight matrix (WM) of neural (NN) by another NN or itself. However, the weight generation steps are typically not visually interpretable by humans, because the contents stored in the WM of NN are not. Here we apply the same principle generate natural images. The resulting fast weight painters (FPAs) learn to execute sequences of delta learning rules sequentially generate images sums of outer products of self- invented keys and values, one rank at time, as if each image was a WM of an NN. We train our FPAs the generative adversarial networks framework, and evaluate various image datasets. We show how these generic learning rules can generate images with respectable visual quality without any explicit inductive bias for images. While the performance largely lags behind the one of specialised state-of- the-art image generators, our approach allows for visualising how synaptic learning rules iteratively produce complex connection pattems, yielding human-interpree interpretablle meaningful images. Finally, we also show that an additional convolutional U-Net (now popular in diffusion models) the output of FPA learn one-step "denoising" of FPA- PPA-generated images to enhance their quality. Our code is public
INTRODUCTION
Fast Weight Programmer (Schmidhuber 1991a 1992p a neural network (NN) that can learn to continually generate and rapidly modify the weight matrix (i.e., the program) of another NN in response to a stream of observations to solve the task at hand (reviewed in Sec. 2.I). At the heart of the weight generation process lies an expressive yet scalable parameterisation of update rules (or learning rules, or programming instructions) that iteratively modify the weight matrix to obtain any arbitrary weight patterns/programs suitable for solving the given task. Several recent works (Schlag et al.] 220 Irie et al.] [021: 2022c b) have demonstrated outer products with the delta rule (Widrow & Hoff 1960 Schlag al. 202
effective mechanism for weight generation. In particular, this has been shown to outperform the purely additive Hebbian update rule (Hebb) 1949) used in the Linear Transformers (Katharopoulos et al., 2020 Choromanski et al. 2021) in various settings including language modelling (Schlag et al. 202 time series prediction Irie et al. 2022b), and reinforcement learning for playing video games (Irie al. 202T). However, despite its intuitive equations gquations-tteeting treating the fast weight matrix as a key/value associative memory the effective "actions" these learning rules on the "contents" stored the weight matrix still remain opaque, because in general the values stored in weight matrix are not easily interpretable by humans.
Now what if we a fast weight programmer generate a "weight matrix" that corresponds to some human-interrpetabbe data? While outer product-based pattern generation may have a good inductive bias for generating a weight matrix linear layer"] it can also be seen as a generic mechanism for iteratively generating any high dimensional data. So let us apply the same principle to generate and
https:/ gi thub. com/ A/ ffaafannee
"I linear layer, the weight matrix is multiplied with an input vector to produce an output vector. Conse- quently, assuming that the output vector is used to compute some scalar loss function, the gradient of the loss w.r.t weights is expressed outer product (between the input and the gradient of the loss w.r.t. the output).

---

Published as a conference paper at ICLR 2023
incrementally refine natural images. We treat colour image three weight matrices representing synaptic connection weights of a fictive NN, and generate them iteratively through sequences of delta learning rules whose key/value patterns and learning rates are produced by actual NN that train. The resulting Fast Weight Painters (FPAs) learn to sequentially generate images, as sums outer products, rank time, through sequential applications of delta learning rules. Intuitively, the delta rule allows painter to look into the currently generated image in a computationally efficient way, and apply a change the image at each painting step. We empirically observe that the delta rules largely improve the quality of the generated images compared to the purely additive outer product rules.
We train our FPAs in the framework of Generative Adversarial Networks (GAN; GANN:eoolllloww et al. (2014); Niemitalo (2010); Schmidhuber 1990); reviewed Sec. 2.2). We evaluate our model on six standard image generation datasets (CelebA, LSUN-Churee Metfaces, AFHQ-Cat/Doo Caoo//Ca//D//Wdd; all the resolution 64x64), and report both qualitative image quality as well as the commonly used Fréchet Inception Distance (FID) evaluation metric (Heusel et al. 2017). Performance is compared to the one of the state-of-the-art StyleGAN2 (Karras al. 2020b a) and the speed-optimised "light- weight" GAN (LightGAN; Liu al. (202I)).
While the performance still largely lags behind the one of StyleGAN2, we show that our generic models generate images of respectable visual quality without any explicit inductive bias for image processing (e.g., no convolution is used in the generator). This confirms and illustrates that generic learning rules effectively produce complex weight patterns that, in case, yield natural images in various domains, Importantly, we can visualise each step of such weight generation in the hummn-interpretable image domain. This is unique feature our work since learning rules are typically not visually meaningful to humans in the standard weight generation scenario-sed the example shown in Figure (weight generation for few-shot image classification).
Clearly, our goal is not to achieve the best possible image generator (for that, much better convo- lutional architectures exist). Instead, use natural images to visually illustrate the behaviour of an NN that learns to execute sequences of learning rules. Nevertheless, it is also interesting to see how convolutional NN can further improve the quality of FPA-generated images. For this purpose, conduct an additional study where we add to the FPA's output,
popular convolutional U-Net (Ronneberger et al.) 2015; Salimans et al. 2017) used as the standard architecture (Ho et al. 2020 Song al. 2021] Dhariwal & Nichol 2021) h for denoising diffusion models Sicksteik et al. 2013). The image-to-iman transforming U-Net learns (in this case) one-step "denoising" of FPA -eneerered images and effectively improves their quality.
Head 12
Layer
Layer 1
Head 16
Inputs
4
3
2 3
4
4
3
4
0
4
1
0
3
Figure 1: An illustration of the hardly human-interpretable standard weight generation process through sequences of delta rules in FWP (DeltaNet) trained for 5-way 5-shot image classification on Mini-ImageNe
(Vinyals et al.) 2016; Ravi & Larochelle| 2017). The model is trained with the public code of Irie al. (2022c) and achieves (shown at the bottom) is sequence of images with their label (except for the last one to be predicted) processed from left to right, one image/label pair per step. The model has four layers with 16 heads each and hidden layer size of 256. Each head generates 16x 1 d-ddmensional fast weight matrix. Here we visualise weight generation of two heads: head '16' in layer and head '12' layer 4 as examples. In each case, the top row shows the rank-one update term (last term in Eq. 2 and the bottom row shows their cumulative sum, i.c., the fast weight matrix W, of Eq. 2 generated at the corresponding step.
accuracy of 62.5%. The input to the model
2

---

Published as a conference paper at ICLR 2023
2
BACKGROUND
Here we review the two background concepts that are essential to describe our approach in Sec. Fast Weight Programmers (Sec. 2.1 and Generative Adversarial Networks (Sec. 2.2).
2.1 FAST WEIGHT PROGRAMMERS (FWPS)
A Fast Weight Programmer (FWP: Schmidhuber 1991a 1992)) is general-purpose auto-regressive
NN originally proposed as an alternative to the standard recurrent NN (RNN). The system consists of two networks: the slow net, or the programmer that learns (typically by gradient descent) to generate and rapidly modify a weight matrix (i.e., the program) of another net, the fast net, based on the input available at each time step. The context-sensitive fast weight matrix serves as short-term memory of this sequence processor. This concept has seen a recent revival due to its formal connection (Katharopoulos al., 2020; |Schlag et al. [2211a) to Transformers (Vaswani ct al. 2017). In fact, Transformers with linearised attention (Katharopoulos et al. (020)) have a dual form (Aizerman al., 1964 Irie et al., (022a) that is an FWP with an outer product-based weight generation mechanism (Schmidhuber 1991a]1999
Let t, dkey. din. and domt denote positive integers. notable example of such a model is the DeltaNet (Schlag et al.) (021a) that, at each time step f, transforms an input Ø2 E Rd into an output Ye E IR while updating the fast weight matrix W R Xday (starting from W,- = 0) follows:
(9:h kr, Vt, 3e] = W 2o
W. = Woo(g:)
- @ (kks)
(2)
where the slow net (Eq. with a learnable weight matrix W slow E (2 A + (xxde) generates query qe E RRdy, key ki R.day, value vr &R vectors as well as a scalar 3r E IR (to which we apply the sigmoid function r). & denotes an element-wise activation function whose output elements are positive and sum up to one (we use softmax) that is crucial for stability (Schlag et al., 2021a). Eq. corresponds to the rank-one update of the fast weight matrix, from Wi-1 to Wr, through the delta learning rule (Widrow & Hoff| 1960) where the slow net-generated patterns, 2t, (k;,) and (Be), play the role of target, input, and learning rate of the learning rule respectively. We note that if we replace this Eq. [ with a purely additive Hebbian learning rule and set the learning rate to i.e., W, W 0z o(kz), we fall back to the standard Linear Transformer (Katharopoulos ct al. 2020 In fact, some previous works consider other learning rules in the context of linear TransformersFWWP c.g.. a gated update rule (Peng et al. 2021) and the Oja learning rule (Oja 1982 Irie et al. (022b)). Our main focus is on the delta rule above, but we" 'lll also support this choice with an ablation study comparing it to the purely additive rule.
An example evolution of the fast weight matrix following delta rules of Eq. is shown in Figure As stated above, the visualisation itself does not provide useful information. The core idea of this work is to apply Eq. to image generation, and conduct similar visualisation.
2.2 GENERATIVE ADVERSARIAL NETWORKS
The framework of Generative Adversarial Networks (GANs; Goodfellow et al. (2014); Niemitalo (010)) trains two neural networks G (as in generator) and D (as in discriminator), given some dataset R, by making them compete against each other. The GAN framework itself special instantiation (Schmidhuber) 2020) of the more general min-max game concept of Adversarial Artificial Curiosity (Schmidhuber) 1990 1991b), and has applications across various modalities, e.g., to speech (Binkowski al. (2200) and with limited success also to text (de Masson d Autume al.) 2019). but here we focus on the standard image generation setting (Goodfellow et al.) 2014).
what follows, let h, w, and d denote positive integers. Given an input vector E IRd whose elements randomly sampled from the zero-mean unit-variance Gaussian distribution A (0, 1), G generates an image-like data G(z) Rexhxw with height of h, a width of w, and channels (typically e 3; a non-transparent colour image). D takes an image-like input X E Rexxxx and outputs a scalar
'Sometimes we refer to the slow net as the "fast weight programmer" by generally referring to it as an NN whose output is weight matrix, e.g., see the first sentence of the introduction. However, without the forward computation of the fast net, the slow net alone not general purpose sequence processing system.
3

---

Published as a conference paper at ICLR 2023
D(X) E R between 0 and 1. The input X is either sample from the real image dataset R it is a "fake" sample generated by G. The training objective D is to correctly classify these inputs as real (label 'I') or fake (label '0'), while G is trained to fool D, i.c., the error of D is the gain of G The two networks are trained simultaneously with alternating parameter updates. The goal of this process is to obtain G capable of generating "fake" images indistinguishable from the real ones. In practice, there are several ways of specifying the exact form of the objective function. We refer to the corresponding description (Sec. 3.3 and the experimental section for further details.
Over the past decade, many papers have improved various aspects of GANs including the architecture (e.g.. Karras et al. (2019) 2021)). loss function (e.g.. Lim & Ye (2177)) data augmentation strategies (e.g.. Zhao et al. (2020); Karras et al. (2020a), and even the evaluation metric (e.g.. Heusel et al. (2017)) )) to achieve higher quality images at higher resolutions. In this work, our baselines are the state-of-the-art StyleGAN2 (Karras et al., 2020b a) and speed-optimised "light-weight" GAN (LightGAN: Liu al. (021
3 FAST WEIGHT "PAINTERS" (FPAS)
Fast Weight Painter (FPA) is a generative model of images based on the weight generation process
of outer product-based based Fast Weight Programmers (FWP; Sec. 2.1 Eq. (2) Conceptually, such model can be trained as a generator in the GAN framework (Sec. 2.2 or as a decoder in a Variational Auto-Encoder (VAE; Kingma & Welling (2014)) Here, we train it in the former setting that offers a rich set of accessible baselines. In the orginal FWPs, the slow net or the programmer's 's goal is to generate useful programs/weights for the fast net to solve given problem. In the proposed FPAs under the GAN framework, the "slow net" or the painter's objective is to generate images that maximise the "fast /ddiisrriiinattr/ritiiiss prediction error.
Here we describe the main architecture of our FPA (Sec. 3.1 (Sec. 3.2 and other GAN specifications (Sec. 3.3
its extension through the U-Net
3.1
MAIN ARCHITECTURE
Like typical generator in the GAN framework, an FPA is NN that transforms randomly-sampled latent noise vector into an image. Its general idea is to use a sequence processing NN to decode the input vector for a fixed number of steps, where in each step, we generate a key/value vector pair and a learning rate that are used in synaptic learning rule to generate a rank-one update to the currently generated image (starting from 0). The final image thus corresponds the sum of these update terms. The number of decoding (or painting) steps is a hyper-parameter of the model or the training setup.
In what follows, let T, c, drey. dralues duaeen din. duidden: denote positive integers. Here we first provide an abstract overview of the building blocks of the FPA, followed by specific descriptions of each block. Given a random input vector z E Rddeee, and a number painting steps T, an FPA generates an image W E Rexdotn Xduy with e channels, a height of dyalee and width of daey. through the following sequence of operations:
(I,, FT)
InputGencrator(s)
(h1,... hr)
Sequenceeroccessr((1)) 2T)
(5)
W.
UpdateNee((Wz-11. h1) for E (I, T}
(6)
W WT
where fort [I...,?') T}, E Rds, h, E Rdusso, and W, E Rexdimx ddey with Wo = 0. InputGenerator, Sequencel EequenceProcessor, rocessor, and NptateNet denote the abstract blocks of operations described as follows:
Input Generator. As its name indicates, the role of Input is to transform the latent vector dha to sequence of input vectors (#:,..., with 2; Rdda for E (1,., T}. for In practice, we consider two variants.
the subsequent sequence processing NN, SoquenecProeessee
DDeppte the "adversarial" nature of this description, when G is trained by gradient descent, it nothing but the gradient feedback of D) through (GG(=) that continually improves G. In this view, D is "collaborative".
4

---

Published as a conference paper at ICLR 2023
Current image
New Image
Delta Rule
Rankk-1 Update
Value
Key
Learning
Image Update Net
Input
Sequence Processor
Figure 2: An illustration of the Fast Weight Painter architecture. The example taken from the last painting step of our best FPA trained on the AFHQ Wild dataset.
In the first variant (v1), the input generator does not do anything: it feeds the same latent vector to Sequencel rocessor at every step, i.c., D, = for all with dnn- = dhatent- In the second variant (v2), Input is an NN that maps from input vector of dimension diatnt to output vector of size T * dm- The latter is split into T vectors æ; (f f1...., T) of size dm each. An obvious advantage of the first approach is that scales independently of T, and it may also be rolled out for an arbitrary number of steps. In most datasets, however, we found the second variant to perform better, yielding more stable GAN training.
Sequence Processor. SequenceProcessn further processes the input vector sequence produced by InpurGenerator. our preliminary experiments, we found that auto-regressive processing and in particular RNNs good choice for this component (e.g., did not manage to train any models successfully when the standard Transformer was used instead). We use a multi-layer long short-term memory (LSTM: Hochreiter & Schmidhuber (1997)) RNN in all of our models. We also allocate separate NN that maps the latent vector z to the initial hidden states each RNN layer (omitted for clarity Figure 2.
Image Update Network. The actual painting finally emerges in the image update network, UpdateNet. The role of UpdateNet is to apply modifications to the current image W (starting from Wo 0) and obtain new image W., through an application of the delta learning rule. At each time step I, the input ht E Rduss is projected to key k, E R Resduyy value Vr E I Resdato and learning rate Bg E RS vectors by using a learnable weight matrix W Wslow E Rea duase + +1) xixdia
[kt Dt, 3.]
W Wlowhh
Similarly multi-head attention in Transformers Vaswani et al.) 2017). the generated vectors are split into sub-vectors, one for each channel, i.e., for each painting step & {1, .... T), we have [k] [k...... k:]] k for keys, [b] .... ed] v, for values, and [,8(..., ..., 3r = 3, for learning rates, where kf E Rd of Bidat and 3 E IR for i {1, e) denoting the channel index.
Finally, for each step L (1.... T} and for each channel index i E (1, .... c), the corresponding image channel Wi E Rddat Xdey is updated through the delta update rule (same as in Eq. 2h:
wi -.- W: + wi
@ o(k))
Here each channel of the image is effectively treated as "weight matrix," and "painting" is based on synaptic learning rules. The output image is finally obtained from the final step = as W [W] W E xexd (to which apply tanh). We'll visualise the iterative generation process of Eq. 9 in the experimental section.
An overview of this architecture is depicted in Figure
5

---

Published as a conference paper at ICLR 2023
3.2 OPTIONAL FINAL U-NET REFINEMENT STEP
Experimentally, we find the process described above alone can generate reasonably fine looking images without any explicit inductive bias for images. However, the resulting evaluation scores are worse than those of the baseline methods based on convolutional architectures. This motivates us to further investigate how the quality of images generated by an FPA can be improved by an additional convolutional component. For this purpose, we use the image-to-img architecture (Ronneherger et al.) 2015 Salimans et al., 2017). the core architecture (Ho al. 2020 Song et al.] |001]]22]]]] | Dhariwal & Nichol 2021) of the now popular denoising diffusion models (Sohl
transforming U-Net
Dickstein et al., 2015). We apply the U-Net to the output of the FPA, i.e., after Eq. it transforms the image W WT E Rexd duy into another image of the same size W' E Rexdoat Xduy;
W = UNet(Wrr
(10)
From the U-Net's perspective, the output of the FPA is a "noisy" image. Its operation can be viewed as one-step "denoising." This process is depicted in Figure 3 In practice, we append this U-Net to the output of a pre-trained FPA, and train the resulting model the GAN framework, while only updating parameters of the U-Net. We also discuss end-to-end training in Appendix C.1 However, in the main setting, all our U-Net models are trained with pre-trained frozen FPA. In the experimental section, we'll I1l show that such a U-Net can effectively improve the quality of FPA-generated images.
Fast Weight Painter
Figure 3: An illustration of the extra U-Net-based refinement. The example is generated by our best FPA/U-Net trained on the AFHQ-Wild dataset.
3.3 DISCRIMINATOR ARCHITECTURE AND OTHER SPECIFICATIONS
We adopt the training configurations and the discriminator architecture of the LightGAN (Liu et al., 2021), i.e., we replace its generator by our FPA. In our preliminary study, we also tried the StyleGAN2 setting, however, we did not observe any obvious benefit. In the LightGAN framework, in addition to the main objective function of the hinge loss (Lim & Ye 2017] Tran et al. 2017). the discriminator is regularised with an auxiliary image reconstruction loss at two different resolutions (for that two "simple" image decoders are added to the discriminator), For details, we refer to Appendix B.3
4 EXPERIMENTS
4.1
BENCHMARKING ON THE STANDARD DATASETS
We start with evaluating how well the proposed FPA performs as an image generator without explicit inductive bias for images. For this, train model on six datasets: CelebA (Liu al., [2013). LSUN Church Yu et al. (015)), Animal Faces HQ (AFHQ) Cat/Dog/Wild (Choi et al. 2020). and MetFaces (Karras al. 2020aa) all at the resolution of 64x64. No data augmentation used want to keep the comparison as simple as possible. For details of the datasets, we refer to Appendix B.1 As first set of experiments, we set the number of generation steps for FPAs to
64
(we Il vary this number in Sec. 4.3); such that the output images can be of full rank. We provide all syper-parameters Appendix B.2 and discuss training/generation speed in Appendix C.2 Following the standard practice, we compute the FID using 50 K sampled images and all real images (for further discussion on the FID computation, we refer to Appendix [B.4). Table I shows the FID scores. We first observe that the tfateooftheeaae StyleGAN2 outperforms our FPA by a large margin. The performance gap is the smallest in the case of the small dataset, MetFaces, while for larger datasets, CelebA and LSUN-Church, the gap is large. At the same time, the reasonable FID values show that the FPA quite successful. Qualitatively, we observe that the FPA can produce images with a respectable quality, even though also observe that the StyleGAN2 tends to generate fine-looking

---

Published as a conference paper at ICLR 2023
Table 1: FII scores. The resolution is 64x64 for all datasets. No data augmentation is used. The StylcGAN2 models are trained using the official public implementation.
AFHQ
LSUN
Model
CelebA
MetFaces
Cat
Dog
Wild
Church
StyleGAN2
1.7
17.2
7.5
11.7
5.2
2.8
LightGAN
3.4
26.4
7.9
16.8
10.2
5.1
FPA
18.3
36.3
17.1
45.3
20.2
42.8
+ U-Net
3.7
24.5
6.8
19.9
9.4
5.2
images more consistently, and with a higher diversity. Figure displays the curated output images generated by various models. By looking at these examples, it is hard to guess that they are generated as sums of outer products through sequences of synaptic learning rules. The visualisation confirms the FPA's respectable performance.
In Table [ we also observe that the extra convolutional U-Net (Sec. 3.2) largely improves the quality of the images generated by the FPA. Its performance is comparable to that of the LightGAN baseline across all datasets. For further discussion of the UNet's ' effect, we refer to Appendix C.1
Fast Weight Painter
Fast Weight Painter U-Net
StyleGAN2
Figure 4: Curated image samples generated by different models specified above the images. Four images are shown for each dataset, from left right: CelebA, MetFaces, AFHQ Cat, Dog, Wild, and LSUN Church. All at the resolution of 64x64.
4.2 VISUALISING THE ITERATIVE GENERATION PROCESS
How are images like those shown in Figure generated iteratively as sums of outer products by the FPA? Here we visualise this iterative generation process. Generally speaking, we found almost all examples interesting to visualise. We show first set of examples in Figure As we noted above, all FPAs in Table Figure use T = 64 painting steps. The first thing we observe is that for many steps, the "key" is almost one-hot (which is encouraged by the softmax), i.e., the part the image generated almost column- wise. For other parts, such generation/refinemeen of the background (e.g., steps 57-64 AFHQ Wild or steps 1-16 in MetFaces), rank-1 update covers a large region of the image. Generally (e.g., adding the extra colour the background in steps 59-64 LSUN Church, or drawing draft of the animal face in steps 7-34 in AFHQ-Wild). We discuss many more examples in Appendix A
recognise the "intended action" of each learning rule step the image
7

---

Published as a conference paper at ICLR 2023
AFHO-Wild
LSUN-Church
64
MetFaces
16
24
32
40
48
56
Figure 5: Example painting steps for the FPAs of Table I All images have the resolution of 64x64. In each example, the generation steps are shown from left to right. The numbers below images indicate the step. Each has two mini-rows showing rank-1 1 update generated at the corresponding step. while the bottom mini-row shows the cumulative sum thereof, i.c., the currently generated image (Eq. 19. For visualisation, the image each step normalised by the norm of the final image, and apply tanh. We also note that the colour scaling specific to each plot (in consequence, we observe some cases like the one we see in step 31 of LSUN-Church above where the effect of the rank-1 update is not visible once added to the image).
sequences of images. The top mini-row shows the
4.3 ABLATION STUDIES
Varying Number of Painting Steps. In all examples above, we train FPAs with number of painting steps T = 64 such that the 64x64 output can be of full rank. Here we study FPAs with reduced numbers of steps. The task should re- main feasible, as natural images typically keep looking good (at least to human eyes, to some extent) under low-rank approximations (Andrews & Patterson 1976). We select the model configuration that achieves the best FID with 7 64, and train the same model with fewer steps {8. 16. 32). Table 2 shows the FIDs. Globally, we find that fewer steps exception is MetFaces (and AFHQ some extent) where the
Table 2: FIID scores of FPAs for various numbers paint- ing steps. The resolution is 64x64 for all datasets. The numbers for = 64 are copied from Table
AFHQ
Steps T
MetFaces
Cat
Dog
Wild
64
36.3
17.1
45.3
20.2
32
43.6
25.9
69.5
31.9
16
53.8
86.6
165.8
220.5
8
80.1
164.4
220.1
215.2
tend to greatly hurt performance. degradation is much smaller. Figure(JJshows examples of 16-step generation of 64x64-resolttion im- ages these two datasets, where we observe that FPAs find and exploit symmetries (for AFHQ-Cat) and other regularities (for MetFaces) as shortcuts for low rank/complexity image generation.
Choice of Learning Rules. As mentioned in Sec. 2.1 the delta rule is not the only way of parameterising the weight update (Eq. 9.. However, we experimentally observe that both the purely additive nule and the Oja rule underperform the delta rule. This is in line with previous works
8

---

Published as a conference paper at ICLR 2023
FWPs (see introduction). The best FID obtain the CelebA using the purely additive rules is above 80, much worse than 18.3 obtained by the delta rule (Table DT. With the Oja rule, we did not manage train any reasonable model on CelebA in our settings.
AFHO-cat
MetFaces
16
8
16
16
Figure 6: Examples of 16-step generation of 64x64 images by the FPA.
DISCUSSION
Limitations. We have shown how the concept of FWPs can be applied to image generation, to visualise sequences of NN-controlled learning rules that produce natural images. While this contributes to the study of learning rules in general, we note that our visualisations are tailored to the image generation task: mapping random vector sequence of images, where only the final image (their sum) is "evaluated." In many FWP use cases, however, the generated WMs are queried/used at every time step to solve some sequence processing task. What the learning rules actually do in such scenarios remains opaque. Also, from the perspective of image generation methods, there are many aspects that we do not investigate here. For example, we directly use the convolutional LightGAN discriminator. However, since the FPA is non-comvolutional, there may be alternative architectures with better feedback/gradient properties for training FPAs. Also, our experiments are limited to images with resolution of 64x64. Increasing the resolution is typically not straightforward (Karras) ct al. 2018) but outside the scope of this work.
Diffusion models. We use the GAN framework (Sec. 2.2 tive of using VAEs, At first glance. it also seems attractive to use rank-l noise as an efficient alternative the expensive U-Net of diffuusinn models (Sohi-Dickstein al. 2015). Unfortunately, unlike in the standard Gaussian case (Feller 1949). if the forward process is based on rank-1 noises (e.g., obtained as outer products of two random Gaussian noise vectors), we have no guarantee that the reversal pro- cess can be parameterised in the same way using an outer product. Nevertheless, generally speaking, an exchange of ideas between the fields of image and NN weight cpneratioo/processiii both research domains. An example is the use of discrete cosine transform (DCT) to parameterise WM of an NN (c.g., Koutník et al. (2010a b): van Steenkiste et al. (2016); Irie Schmidhuber (2021)).
train FPA, and mention the alterna-
may stimulate
Other Perspectives. We that the same generic computational mechanism can be used for both fast weight programming and image generation (painting). From a cognitive science perspective, may be interesting compare painting and programming as sequential processes.
CONCLUSION
We apply the NN weight generation principles Fast Weight Programmers (FWPs) to the task of image generation. The resulting Fast Weight Painters (FPAs) effectively learn to generate weight matrices looking like natural images, through the execution of sequences of NN-controlled learning rules applied to self-invented learning patterns. This allows us to visualise the iterative FWP process in six different image domains interpretable by humans. While this method certainly not the best approach for image generation, results clearly semmonstrat/illlsttttevvviss can learn to control sequences of synaptic weight learning rules in a goal-directed way to generate complex and meaningful weight patterns.
how NN

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Róbert Csordás for helpful suggestions on Figure 2
This research was partially funded by ERC Advanced grant no: 742870, project AlgoRNN, and by Swiss National Science Foundation grant no: _19356,, project NEUSYM. We are thankful for hardware donations from NVIDIA and IBM. The resources used for this work were partially provided by Swiss National Supercomputing Centre (CSCS) project s 145 and s1154.
REFERENCES
Mark A. Aizerman, Emmanuil M. Braverman, and Lev I. Rozonoer. Theoretical foundations of potential function method in pattern recognition. Automation and Remote Control, 25(6)99 1964.
Harry C. Andrews and Claude L. Patterson. Singular value decompositions and digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing, 24(1):26-553 1976.
Mikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In Int. Conf. on Learning Representations (ICLR), Virtual only, April 2020.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wo Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 8185-8194, Virtual only, June 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021.
Cyprien de Masson d' Autume, Shakir Mohamed, Mihaela Rosca, and Jack Rae. Training language GANs from scratch. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 4302-4313, Vancouver, Canada, December 2019.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 8780-8794, Virtual only, December 2021.
William Feller. On the theory stochastic processes, with particular reference to applications. In Proc. Berkeley Symposium on Mathematical Statistics and Probability, volume pp. 403 403-433, 1949.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 2672-26880 Montréal, Canada, December 2014.
Donald Olding Hebb. The organization of behavior; a neuropsycholocigal theory. Wiley Book in Clinical Psychology, 62:78, 1949.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 6626-6637, Long Beach, CA, USA, December 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Kazuki Irie and Jürgen Schmidhuber. Training and generating neural networks in compressed weight space. In ICLR Neural Compression Workshop, Virtual only, May 2021.
10

---

Published as a conference paper at ICLR 2023
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transform- ers with recurrent fast weight programmers. In Proc. Advances Neural Information Processing Systems (NeurIPS), Virtual only, December 2021.
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In Proc. Int. Conf. on Machine Learning (ICML), Baltimore, MD, USA, July 2022a.
Kazuki Irie, Francesco Faccio, and Jürgen Schmidhuber. Neural differential equations for learning to program neural nets through continuous learning rules, In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, December 2022b.
Kazuki Irie, Imanol Schlag. Róbert Csordás, and Jürgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In Proc. Int. Conf. on Machine Learning (ICML), pp. 9660-9677 9677, Baltimore, MA, USA, July 2022c.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Int. Conf. Learning Representations (ICLR), Vancouver, Canada, April 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 4401-4410, Long Beach, CA, USA, June 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 2020b.
Tero Karras, Mika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 852-863, Virtual only, December 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers RNNs: Fast autoregressive transformers with linear attention. Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2020.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Int. Conf. on Learning Representations (ICLR), Banff, Canada, April 2014.
Jan Koutník, Faustino Gomez, and Jürgen Schmidhuber. Evolving neural networks in compressed weight space. In Proc. Conference on Genetic and Evolutionary Computation (GECCO), pp. 619-626, 2010a.
Jan Koutník, Faustino Gomez, and Jürgen Schmidhuber. Searching for minimal neural networks in fourier space. Proc. Conf. on Artificial General Intelligence, Lugano, Switzerland, March 2010b.
Jae Hyun Lim and Jong Chul Ye. Geometric GAN. Preprint 1rXivv1705.000 02894.. 2017.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized GAN training for sigh-iidelity few-shot image synthesis. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. IEEE Int. Conf. on Computer Vision (ICCV), pp. 3730-3738, Santiago, Chile, December 2015.
Olli Niemitalo. A method for training artificial neural networks to generate missing data within variable context. Internet Archive, 2010.

---

Published as a conference paper at ICLR 2023
Erkki Oja. Simplified neuron model principal component analyzer. Journal of mathematical biology, 15(3):267-2755 1982.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in GAN evaluation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 11410-11420, New Orleans, LA, USA, June 2022.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. Int. Conf. Medical Image Computing and Computer-Assisted Intervention MICCAI, pp. 234-244 Munich, Germany, October 2015.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixeICNN++: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear Transformers secretly fast weight programmers. In Proc. Inf. Conf. on Machine Learning (ICML), Virtual only, July 2021a.
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast weight memory. In Int. Conf. Learning Representations (ICLR), Virtual only, May 2021b.
Jürgen Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. Institut für Informatik, Technische Universität München. Technical Report FKI-126, 90, 1990.
Jürgen Schmidhuber.
Learning to control weight memories: An alternative to recurrent nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, March
1991a.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Pruc. Int. Conf. on simulation of adaptive behavior: From animals to animals, pp. 222-227, 1991b.
Jürgen Schmidhuber. Learning to control fast weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
Jürgen Schmidbuber, Generative adversarial networks are special cases of artificial curiosity (1990) and also closely related predictability minimization (1991). Neural Networks, 127:58-66 66, 2020.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. Int. Conf. on Machine Learning (ICML), pp. 220622266222 2265, Lille, France, July 2015.
Yang Song. Jascha Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.
Dustin Tran, Rajesh Ranganath, and David M. Blei. Hierarchical implicit models and likelihood-free variational inference. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 5523-5533, Long Beach, CA, USA, December 2017.
Sjoerd van Steenkiste, Jan Koutník, Kurt Driessens, and Jürgen Schmidhuber. A wavelet-based en- coding for neuroevolution. In Proc. Genetic and Evolutionary Computation Conference (GECCO), pp. 517-524, Denver, CO, USA, July 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Adyances in Neural Information Processing Systems (NIPS), pp. 5998 -008, Long Beach, CA, USA, December 2017.
12

---

Published as a conference paper at ICLR 2023
Oriol Vinyals, Charles Blundell, Tim Lillierap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 3630-3638, Barcelona, Spain, December 2016.
Bernard Widrow and Marcian Hoff. Adaptive switching circuits. In Proc. IRE WESCON Convention Record, pp. 96-104, Los Angeles, CA, USA, August 1960.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. Preprint arXiv:1506.03365, 2015.
Shengyu Zhao, Zhijian Liu, Ji Lin, Yan Zhu, and Song Han. Differentiable augmentation for data- efficient GAN training. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
13

---

Published as a conference paper at ICLR 2023
A MORE VISUALISATION EXAMPLES
As continuation of Sec. 4.2 provide further visualisations to illustrate the image generation process of FPAs.
Figure shows some interesting effects we observe on CelebA. In Example (top), from step to 16, the image is column-wise generated from the left. Then from step 18, the generation of the right part of the face starts from the right. Around step 40 and 46, we see that the separately generated left and right parts of the face do not fit each other. But then the FPA fixes this in three steps from step 46 48 that "harmonise" the image. Example 2 (bottom) shows another similar example.
Figure 8 iilustrates another typical behaviour we observe on both CelebA and AFHQ-Dog. In the CelebA example, we can see that the face is already generated around step 48. The rest of the steps are used to add extra hair. Similarly in the AFHQ-Dog example, step-48 image is already fine-looking dog. An extra body painted (rather unsuccessfully) from the right the remaining steps.
example
64
ColebA, example2
24
32
33
40
48
64
Figure 7: Example painting steps of the FPAs of Table trained on CelebA at the resolution of 64x64. For details about the visualisation process, we refer the caption of Figure
ColebA
24
32
56
AFHQ-Dog
24
55
Figure 8: Example painting steps of the FPAs of Table I trained on CelebA and AFHQ-Dog the
resolution 64x64. For details about the visualisation process, we refer to the caption of Figure 5 in particular, in step 25 of CelebA observe effect that is similar to step 31 of LSN Church in Figure(5)
14

---

Published as a conference paper at ICLR 2023
B EXPERIMENTAL DETAILS
B. DATASETS
We use six standard benchmark datasets for image generation: CelebA Liu et al. 2015), LSUN Church Yu et al. 2013 Animal Faces HQ (AFHQ) Cat/Dog/Wild (Choi 2020). and MetFaces (Karras et al.) 2020a). The number of images are 203 K for CelebA, 126 for LSUN Church, 5 K each subsets of AFHQ, and 1336 for the smallest MetFaces. For further information about the datasets, we refer to the original papers.
B.2 HYPER-PARAMENE PARAMETERS
We conduct FPA tuning in terms of both hyper-parameters and model variations. Regarding the latter, we use three specifications: InputGenerator version v1 vs. v2 (see their descriptions in Sec. 3.1p. input generator with or without tanh, and with without a linear layer that maps the latent vector to the RNN initial states (otherwise the initial states are zeros). We also train models with/wittout applying tanh to the generated image at the FPA output. Using the notations introduced in Sec. 3.1 the hyper-parameter search is conducted across the following ranges: dhatent E {128, 256, 512]. dhitden (1024, 2048, 4096), and the number of RNN layers
{1,2]. When Inpul add an extra linear layer between the input
v2, have d {8, 16, 32} and furthermore, generator and the RNN with a dimension d/ E {128, 256, 512]. The batch size and learning rate are fixed to 20 and 2e respectively. We compute the FID score every 5 training steps monitor the performance. We take the hyper-parameters achieving the best FID as the best configuration. Table 3 summarises the corresponding results. For MetFaces, more parameter-efficient v2 variant (with 20 M parameters) achieves a FID of 40.5 that is close to 36.3 (Table achieved by the best model. However, the parameter count does not matter for our study, we strictly take the best model.
Table 3: Hyper-parameters of FPAs used for different datasets.
AFHQ
LSUN
Hyper-Paramett
CelebA
MetFaccs
Cat
Dog
Wild
Church
InputGeenrraoor
v2
vl
v2
v2
v2
v2
Input Gen. tanh
No
No
No
No
Yes
Latent to RNN init
No
Yes
Yes
No
No
Yes
dhaent din d'm dhidden
128
512
256
256
512
512
8
8
8
16
128
128
128
128
256
1024
4096
1024
1024
1024
1024
L
2
2
Gen. Param. Count (M)
14
220
6
14
17
Total Param Count (M)
26
232
18
26
19
29
B.3 GAN DETAILS & IMPLEMENTATION
As stated in Sec. 3.3 our GAN setting based on that of the LightGAN by Liu (021) For architectural details of the discriminator, we refer to the original paper. To be more specific. our code is based on the unofficial public LightGAN implementation https: / thub com/ Aucidraing ns/ //1 www.ggiggtttggee.e
which is slightly different from the official implementation. In particular, the auxiliary reconstruction losses for the discriminator require 8x8 and 16x16 resolution images. In the original implementation, these are obtained directly from the intermediate layers of the generator, while in this unofficial implementation, they are obtained by scaling down the final output image of the generator by interpolation. The latter is the only compatible approach for FPAs since an FPA does not produce such intermediate images with small resolutions unlike the standard convolution based generators. For any further details, refer our public code. For the StyleGAN2
15

---

Published as a conference paper at ICLR 2023
baseline, use the official implementation of StyleGAN3 KKrras al. 2021 that also supports StyleGAN2: httpp:///itttiiee com http///aabbbsttyyaaaaaaa
B.4 FID COMPUTATION
While the Fréchet Inception Distance (eesell et al. (017)) is a widely used metric for evaluating machine-generated images, it is sensitive to many details (Parmar et al. 2022). We also observe that it is crucial to use the same setting for models consistently. We use the following procedure for all datasets and models (including the StyleGAN2 bascline). We store both the generated and resized real images in JPEG format, and use the pytorch- fid implementation of hhttps //github com/ m tr wtt.e/eerptrr.h.coocoooo f fid to compute the FID. While the usage of PNG is generally recommended by Parmar et al. 2022), since consistency is all we need for the purpose of our study, and JPEG more convenient for frequently monitoring FID scores during training (as JPEG images are faster to store, and take less disk space), we opt for it here.
Nevertheless, in Table 4 we also report FID scores computed using the generated and resized real images saved in PNG format. These are computed using the models from Table [ i.c., the best model checkpoint found based on JPEG-based FID scores during training. All numbers grow beyond those in Table I including those of the baselines, but the general trend does not change. The FID scores reported in all other tables are computed using the JPEG format.
Table 4: FID scores using images in PNG format. The resolution 64x64 for all datasets. No data augmentation is used.
AFHQ
LSUN
Model
CelebA
MetFaces
Cat
Dog
Wild
Church
StyleGAN2 LightGAN
2.7
24.8
11.7
17.4
7.0
9.8
8.8
47.2
13.0
27.9
14.0
9.6
FPA
33.6
73.7
23.4
71.6
28.6
64.7
+ U-Net
6.9
43.1
9.8
30.5
12.6
8.5
C EXTRA RESULTS & DISCUSSION
C.I DISCUSSION OF THE U-NET EXTENSION
Visualising the U-Net's Effect. In Sec. 4.1 we report the U-Net's quality improvements of FPA- generated images in terms of the FID metric. Here further look into its effect the actual images. Figure D displays some example images generated by FPA/U-Net models before and after the U-Net application. We observe that in general, the U-Net is successful improving the quality of the FPA-generater images without completely ignoring the original images. However, we also observe examples where this is not the case, i.e., the U-Net generates almost completely new images as illustrated by the last row of MetFaces and AFHQ Dog examples.
End-to-End Training. As described in Sec. 3.2 we train the FPA/U-Net model using pre-trained FPA parameters. We also tried end-to-end training from scratch, but observed that the U-Net starts learning before the FPA can generate fine-looking images. As result, the resulting U-Net ignores the FPA output, hence the FPA does not receive any useful gradients for learning generate meaningful images, and the FPA remains a noise generator. We also tried to train it by providing both the FPA and U-Net outputs the discriminator (and by stopping the gradient flow from the U-Net to the FPA). This alleviates the problem of the FPA remaining noise generator, but is still not good enough to make the U-Net properly act as denoising/refinement component of the FPA.
Choice of the Pre-Trained FPA. In general, take the best performing standalone FPA the pre-trained model to train a FPA/U-Net model (Sec. 3.2).. In some cases, however, we find that the best standalone FPA model does not yield the best FPA/U-Net model. This is the case for AFHQ-Cat
16

---

Published as a conference paper at ICLR 2023
and Dog illustrated in Table The FPA/U-Net model based on the best pre-trained FPA with input generator of type v1 (Sec. 3.1 yields a slightly better FID than the one based on the best v2 variant, even though the trend is the opposite for the standalone FPA models. In Table of the main text, we report the overall best FIDs.
Our U-Net implementation is based the diffusion model implementation of /ftttss://gtthti htt///ucidraa i ns/ /eesoinoiiss dnn www.dfffsooo---oorrttttt We use three down/up-sampling ResNet blocks with a total of M parameters.
AFHQ Wild
MetFaces
AFHQ Dog
Figure 9: Example images generated by the FPA/U-Net before (left) and after (right) the U-Net.
Table 5: FID scores for the FPA/U-Net with various pre-trained FPA models. The resolution is 64x64 for all datasets. No data augmentation is used. v1/v2 indicates the type of the input generator described in Sec. 3.1
AFHQ-Cat
AFHQ-Dog
Model
v1
v2
v1
v2
FPA
30.8
17.1
49.3
45.3
U-Net
6.8
7.8
19.9
22.9
C.2 TRAINING AND GENERATION SPEED
While it is difficult to compare speed across different implementations, the generation speed of StyleGAN2 and the FPA are similar: about 55 images are generated every second for a batch size of V100 GPU, while the LightGAN
generate 160 images per second in the same setting. The training/convergence speed of the FPA is similar to or better than that of the LightGAN on the relatively large datasets. For example, our FPA converges after about 150 K training steps on CelebA (vs. 285 K steps for the LightGAN), and 70 steps on LSUN Church (similarly to the LightGAN) with batch size of 20. The FPA/U-Net variant converges more slowly. For example on LSUN, continues improving until 380 K steps in the same setting. On the small datasets, e.g., AFHQ-Car the LightGAN converges faster (about 30 steps) than the FPA (which continues improving until about 280 K steps) in the same setting. Any training run can be completed within one to three days on a single V100 GPU.
17


Reviews:
[{"review_id": "xis9yVUOrYl", "reviewer": "Reviewer_wMfX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work visualizes image generation as sequential low-rank approximations to the weight matrices in a FWP(Fast Weight Programming) setup. The motivation is to not generate high fidelity images but to rather enable human interpretable visualizations of the weight updates in a Neural Network (NN). The authors have shown human comprehensible weight update sequences in six popular datasets. \n\n", "summary_of_the_review": "Overall, I appreciate the novelty in equating weight updation to image generation in FWP framework. The paper is clearly written and mostly easy to follow. My main concerns are with the limitations of the proposed approach in terms of its scalability w.r.t image size and to its applicability beyond the visualization of weight updates as discussed under the first point in the main review under methodology. ", "strength_and_weaknesses": "*Novelty and Originality*\n\nThe work is interesting in it being one of the few works that draw similarities between images and weight matrices that are generated by another neural net. The recent of this type being the class of Implicit Neural Representations (INR) [1 ]. The work also is a a slightly novel extension to the existing literature on FWP to image generation. \n\n\n*Methodology*\n\nThe authors claims that image generation is not the best of the strengths of FPA, leaving the visualization of weight updates as the central purpose of the work. Given the poor generation quality of FPA, my question is if the  performance vs interpretability/explainability tradeoff presented in the work is worth a shot. Can the method be adapted to exercise control over generations by manipulating the low-rank updates, ref [4] or restrict the low-rank updates to find discriminating components, ref [5], instead of a rank based updation approach. \n\nWhy shouldn't  one compare FPA with recurrent painting techniques like Draw [2].\n\nThe authors posit that FPA does not include any explicit inductive bias as noted in page 2, \"*we show that our\ngeneric models can generate images of respectable visual quality without any explicit inductive\nbias for image processing (e.g., no convolution is used).*\" But the training involves discriminator of LightGAN [3] that uses convolutional layers. \n\nFrom Table 2, the number of train steps are directly proportional to the generation quality. Is it due to having a softmax over the key as discussed in Section 4.2 that forces slow update rules. In my understanding, and the authors' note, \"*The first thing we observe is that for many steps, the “key” is almost one-hot (which is encouraged by the softmax), i.e., the part of the image is generated almost column-wise.*\" the key is crucial in determining the attention or the generative patterns for the next iteration. Have the authors explored on other activations for key matrices. \n\nCould the authors explain why there is more drastic effect of trains steps on the AFHQ dataset when compared to MetFaces dataset (as shown in Table 2). Have they explored the possible effects of the rank of an image on the generation process. \n\n\n\nReferences:\n\n1. Skorokhodov, Ivan, Savva Ignatyev, and Mohamed Elhoseiny. \"Adversarial generation of continuous images.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n2. Gregor, Karol, et al. \"Draw: A recurrent neural network for image generation.\" International conference on machine learning. PMLR, 2015.\n3. Liu, Bingchen, et al. \"Towards faster and stabilized gan training for high-fidelity few-shot image synthesis.\" International Conference on Learning Representations. 2020.\n4. Zhu, Jiapeng, et al. \"Low-rank subspaces in gans.\" Advances in Neural Information Processing Systems 34 (2021): 16648-16658.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UuEUwcUYkH", "reviewer": "Reviewer_zop8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores extending the idea of fast weight with outer-product based to human-interpretable domains beyond network weight space, and examines the idea with the example of natural image generation. The slow-weight network, now termed Fast Weight Painter now progressively refines the pixel values of a 3D image tensor with one rank-1 updating at a time step. Fair generation results can be observed without imposing any domain-specific inductive bias, and further performance improvements can be achieved by U-net-based one-step denoising. ", "summary_of_the_review": "This paper presents an interesting idea with strong but insufficient empirical results. \nPlease refer to Strength And Weaknesses for detailed comments. \n", "strength_and_weaknesses": "**Strength**\n\nThis paper demonstrates a very strong case for extending the fast weight idea in parameters space to other modalities like natural images. \nWhile it is expected to see a lower performance compared to domain-specific methods, the authors show how to further close this gap by exploiting more inductive bias in the image domain. \n\n\n**Weaknesses**\n\n1. Figure 2 can be further improved by simply annotating everything in Eq 4-7 in the figure. \nNow I'm very confused by the arrow pointing from 'Current image' to 'Value' and the one pointing from 'Key' to 'Current image'. What are they referring to?\n\n2. While I understand this paper is just trying to illustrate a case of extending the idea of fast weight to more domains, more experiments certainly help to show how generic the proposed method is. Currently, there are only results with *image* + *GAN training*, while a quick experiment with other data domains like audio or training methods like VAE or diffusion can be a plus. \n\n\n\nMinor points:\n\n1. \nIn the Sequence Processor paragraph, *we did not manage to train any models successfully when the standard Transformer was used instead.* Does it mean transformers completely fail in this case? I don't see a clear reason why transformers cannot work at all here. \n\n2. Based on the 2nd point in Weaknesses, I am very interested to see a synergy between the proposed method and diffusion, as both are iterative.  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q1NpXIyV5C", "reviewer": "Reviewer_oERk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Fast weight programming sequentially generates a weight matrix of a neural network from another neural network, which matrix is generated as the summation of outer products of self-invented keys and values. \nThis paper employs the idea of fast weight programming in the image generation task,  which results in fast weight painters (FPAs).\nFPAs learn to generate images following a sequence of delta learning rules and adding rank-one components to existing images at each step. \nThe authors train FPAs in the GAN framework in 64x64 resolution and evaluate it on various image datasets. \nWhile the performance largely lags behind the StyleGAN2,  authors claim this method allows for visualizing the iterative procedure of complex connections patterns in synaptic learning rules. ", "summary_of_the_review": "This paper is interesting, at least for me.\nMy major concern is: the observation from the image generation is dataset specific and not helpful in understanding the generic process.\nAs an image generation method, it is not powerful and well-designed enough. ", "strength_and_weaknesses": "**Strength**:\n1. I like the core idea of this paper and feel it is quite interesting. Images are generated as the summation of a sequence of rank-one matrices following the delta learning rules, whose matrices are outer products of key/value invented from the neural network. \nThe self-invented value/key has subtle connections to transformers, and the iterative updating mechanism is intuitively related to diffusion models,  where adding new ranks is a kind of \"denoising.\" I feel this paper may have the potential to inspire new image-generation methods.\n2. The writing is good, and the paper is easy to follow. \n\n**Weakness**\nAlthough the core idea is cool, my concern is: the current paper doesn't provide enough intriguing results/contributions, which are listed below. \n1. **Using image generation task to understand the weight generation process.** One motivation for this paper is to \"visually illustrate the behavior of an NN that learns to execute sequences of learning rules.\" Visualizing weight generation sequence is hard to understand, so authors visualize image generation. \nI am not convinced by this statement since image generation and weight generation are quite different, and it is unclear to me how much visualization/observation obtained from visualizing image generation could be transferred/generalized to weight generation. \nMoreover, observations from visualization(Sec 4.2, Appendix A) are quite specific for each dataset, which doesn't reveal any genetic observations/ideas. \nSome experiments, like summarising the genetic properties of the generation process and validating it on different domains, would support this claim. Otherwise, I personally think it is specific to image generation tasks.\n2. I understand \"Clearly, our goal is not to achieve the best possible image generator (for that, much better convolutional\narchitectures exist). Instead, we use natural images to visually illustrate the behavior of\nan NN ...\" But I feel more solid experiments/design exploration might improve this paper. \nE.g., the current image generation results have a big gap to stylegan 2 (which is fine for me), and are limited to 64x64 resolution. I wonder, does it still work in 128x128 or even higher resolution? \nWhat if inventing several key/value pairs at each step and calculating multi-rank matrices?\nWhat if updating is conducted on each image patch instead of the whole image? \nWhy UNet's induct bias is so helpful?  \nIt is not necessarily a weakness but adding more design exploration/analysis would be helpful to illustrate the power of the proposed method as an image generation method. \n\n\nIn summary, as a paper visualizing the generation process, weakness 1 would be my main concern. From the image generation perspective, I am worried about the current performance and experiments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "LJAKwY8xaau", "reviewer": "Reviewer_81gc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an image generation model called fast weight painters (FPA), applying the technique of fast weight programmer. The proposed method treats the generated image as the generated weight matrix.\n\n1. This work seems to be one of the first attempts to apply fast weight programmer method to image generation task, which is novel.\n2. The iterative optimizations of image generation also provide a good interpretability.", "summary_of_the_review": "Overall, the proposed method looks novel to me, although not flawless as mention above. I would recommend to accept this paper.\n\nUPDATE: After discussing with other reviewers and AC, I decide to lower the score. We agree the proposed method is novel and interesting. But as I mentioned the the review, the paper needs more explanation for the visualization results. And the author should also demonstrate why it is important to the community. We believe the paper could be more solid if the author could address these issues.", "strength_and_weaknesses": "Pros:\n1. The paper is well-written.\n2. The proposed method is technically sound.\n3. It is novel to introduce fast weight programmer method into image generation task.\n\nCons:\n1. The generation results is not comparable to SOTA image generation method, although I acknowledge this work is an attempt to introduce fast weight programmer into image generation task.\n\n2. It is good to see the decomposition of image generation process step-by-step, however, the author didn't provide enough illustration or explanation for the visualization results. \n\n3. Lack of high-resolution image generation results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster