Idempotent Generative Network
Authors: Assaf Shocher
Amil V Dravid
Yossi Gandelsman
Inbar Mosseri
Michael Rubinstein
Alexei A Efros
Venue: ICLR 2024 poster (2024)
Keywords: Generative model
idempotent
energy based models
Primary area: generative models
TL;DR: A new generative model, based on projection onto the data manifold, s.t. f(f(z))=f(z).

Abstract: We propose a new approach for generative modeling based on training a neural network to be idempotent. An idempotent operator is one that can be applied sequentially without changing the result beyond the initial application, namely $f(f(z))=f(z)$. The proposed model $f$ is trained to map a source distribution (e.g, Gaussian noise) to a target distribution (e.g. realistic images) using the following objectives: 
(1) Instances from the target distribution should map to themselves, namely $f(x)=x$. We define the target manifold as the set of all instances that $f$ maps to themselves.
(2) Instances that form the source distribution should map onto the defined target manifold. This is achieved by optimizing the idempotence term, $f(f(z))=f(z)$ which encourages the range of $f(z)$ to be on the target manifold. Under ideal assumptions such a process provably converges to the target distribution. This strategy results in a model capable of generating an output in one step, maintaining a consistent latent space, while also allowing sequential applications for refinement. Additionally, we find that by processing inputs from both target and source distributions, the model adeptly projects corrupted or modified data back to the target manifold. This work is a first step towards a ``global projector'' that enables projecting any input into a target data distribution.

Published as a conference paper at ICLR 2024
IDEMPOTENT GENERATIVE NETWORK
Assaf Shocher! Amil Dravid' Yossi Gandelsman' UC Berkeley
Inbar Mosserì Michael Rubinstein" Alexei A. Efros
Googge Research
ABSTRACT
We propose new approach for generative modeling based on training a neural network to be idempotent. An idempotent operator is one that can be applied sequentially without changing the result beyond the initial application, namely f(f(=))= = f()). The proposed model f is trained to map source distribution (e.g, Gaussian noise) to target distribution (e.g. realistic images) using the fol- lowing objectives: (1) Instances from the target distribution should map to them- selves, namely f(z) = z. We define the target manifold as the set of all instances that / maps themselves. (2) Instances that form the source distribution should map onto the defined target manifold. This is achieved by optimizing the idem- potence term, f([(=)) f=) which encourages the range of (=) be the target manifold. Under ideal assumptions such a process provably converges to the target distribution. This strategy results in model capable of generating an output in one step, maintaining a consistent latent space, while also allowing sequential applications for refinement. Additionally, we find that by processing inputs from both target and source distributions, the model adeptly projects corrupted or mod- ified data back to the target manifold. This work first step towards a "global projector" that enables projecting any input into target data distribution.
INTRODUCTION
GEORGE: You't gonna "overdry"
JERRY:
You, you can't "overdry." Why not?
GEORGE:
JERRY: Same you can't 't "overwet." You once something is wet, it's wet. Same thing with dead: like once you die you 're dead, right? Let's say you drop dead and shoot you: you not gonna die again, you'l ve already dead. You can't 't" "ovendie," you can't ' "overdry."
"Seinfeld", Season
Episode NBC 1989
Generative models aim to create synthetic samples by drawing from a distribution underlying the given data. There various approaches such GANs (Goodfellow et al., 2014), VAE (Kingma & Welling, 2022), diffusion models -Sohl-Dickstein et al., 2015; Ho et al., 2020), pixel autoregressive methods (van den Oord al., 2017; 201 2016b;a) and some recent like consistency models (Song al., 2023) and Bayesian flow networks (Graves et al., 2023). Inputs to these models could range from samples of random noise to specific input images in conditional setups, which are then mapped to outputs aligned with given target distribution, typically the manifold of natural images. However, each model is specifically trained to expect particular type of input. What model to be able to take any type of input, be it corrupted instances (e.g., degraded images), an alternative distribution (e.g., sketches), or just noise, and project them onto the real image manifold in one step, kind of "Make It Real" button? As first step toward this ambitious goal, this work investigates a new generative model based on generalization of projection Idempotence.
wanted a single
An idempotent operator is one that can be applied sequentially multiple times without changing the result beyond the initial application, namely f(f(2) = f(z). Some real-life actions can also be considered idempotent, as humorously pointed out by Jerry Seinfeld (1). One mathematical example is the function mapping 2 to |2]; applying it repeatedly yields ||>|| = |2), leaving the result unchanged. In the realm of linear operators, idempotence equates to orthogonal projection. Over R", these are matrices A that satisfy A² = A, with eigenvalues that are either or 1; they can be interpreted geometrically preserving certain components while nullifying others. Lastly, the

---

Published as a conference paper at ICLR 2024
identity function naturally exhibits idempotent behavior, applying it multiple times leaves the input unchanged.
We propose Idempotent Generative Net- works (IGN), a model based on the idea of projection. Given dataset of examples { Our goal to "project" our input onto the target distribution Pz from which 2;'s are drawn. Fig. illustrates the basic objectives. We assume that distributions Pz and P= lie in the same space. Given that, valid to apply f given ex- ample r ~ Pa. What should the outcome of doing so be then? The natural answer to that is "nothing". Considering the intu- ition of projection, an instance that already lies the target manifold should just re- main the same- "You can't overdry". The first objective is then perfectly fulfilled when estimated manifold of the data as the sub-set of all instances that f maps close to themselves.
Target distribution
Source distribution
[e.g. noise)
Figure 1:
The basic idea behind IGN: real examples (a) other inputs (z)
are invariant to the model f: f(z) projected onto the manifold of instances that maps themselves by optimizing for f(f(2) f()..
(xx x We leverage this notion, and define the
Next, want to map instances from different distribution onto that estimated manifold. To do so, we want f(2) to be on the manifold for every Pz. which is characterized by being mapped to itself. This defines our second objective, Idempotence: f(f(=) = f(z). While the aforementioned objectives ensure both and f(z) reside on the estimated target manifold, they do not inherently constrain what else populates that manifold. To address this, we introduce third term, tighten the manifold, pushing for f(f(2) + f(2). The intricacy of reconciling opposing loss terms is unraveled in Section 2.1.
While the landscape of generative modeling is rich with diverse methodologies, our Idempotent Generative Network (IGN) features specific advantages that address existing limitations. In contrast to autoregressive methods, which require multiple inference steps. IGN produces robust outputs a single step, akin to one-step inference models like GANs. Yet, also allows for optional sequential refinements, reminiscent of the capabilities in diffusion models. Unlike diffusion models, however, IGN maintains a consistent latent space, facilitating easy manipulations and interpolations. The model shows promise in generalizing beyond trained instances, effectively projecting degraded or modified data back onto the estimated manifold of the target distribution. Moreover, the model's ability accept both latent variables and real-world instances as input simplifies the editing process, eliminating the need for the inversion steps commonly required in other generative approaches. We draw connections to other generative models in Section 5.
METHOD
We start by presenting our generative model, IGN. It is trained generate samples from target distribution P,, given imput samples from a source distribution Pz. Formally, given dataset of examples {fi))ie(....... with each example drawn from Pas we train model f to map Pa Pa- We assume both distributions P. and Pa lie in the same space, i.e., their instances have the same dimensions. This allows applying f to both types of instances z Pa and x ~ P..
Next, we describe the optimization objectives, the training procedure of the model, the architecture, and practical considerations the training.
2.1 OPTIMIZATION OBJECTIVES
The model optimization objectives rely on three main principles. First, each data sample from the target distribution should be mapped by the model to itself. Second, the model should be idempotent applying it consecutively twice should provide the same results applying it once. Third, The subset of instances that are mapped to themselves should be as small as possible. Next we explain the objectives and show how these principles are translated optimization objectives.
2

---

Published as a conference paper at ICLR 2024
Reconstruction objective. Our first objective, motivated in the introduction, is the reconstruc- tion objective, which is perfectly achieved when each sample x ~ Pz is mapped to itself:
f(x
Given distance metric D (e.g., L2), we define the drift measure of some instance as:
daly) D(y, fo(w))
Where are the parameters of a model So. We then seek to minimize the drift measure:
min (z) min fo(x)()))
The fact that real instances mapped themselves motivates to define the ideal estimated target manifold as the subset all possible inputs that are mapped to themselves by our model:
8 = {y f(y) y) = 8(y) = 0}
Idempotent objective. the estimated manifold:
We desire f to map any instance sampled from the source distribution onto
(() E
z Pz
(5)
Taking the definition in Eq. 4 and combining with our goal 5, we can define our second objective, that when perfectly achieved we get:
[((())) = [=))
This implies that f is idempotent over the domain of all possible Pa. This idempotence objective formulated then as follows.
min (f(((())) = min D ( fo(f(=)))
However, we will next see that directly optimizing this formulation of Idempotnce has a caveat that requires us split it into two separate terms.
Tightness objective. In the formulation of the objectives far, there is missing link. The reconstruction objective, if optimized perfectly (eq. 2.1), determines that all given examples are on the estimated manifold. However, it does not imply that other instances are not that manifold. An extreme example is that if is identity ((z)
Vz. it perfectly satisfies both objectives. Furthermore, the idempotent objective as formulated far, is encouraging the manifold to expand.
Fig. 2 and fig. 17 illustrate this problem. There are two distinct pathways of gradients that flow when the idempotent objective in eq. minimized, both which would contribute when optimizing. The first pathway, which is the desired one, is by modifying f such that f(z), the first application of (red fig. 2) is better mapped to the currently defined manifold. The second pathway for given f(z) making (y) closer to (green in fig. 2). Since the estimated manifold is defindd by (y: fly) = y) This second way of optimizing is effectively expanding it.
f(f(z) = fz)
Z
Figure 2:
Two distinct pathways to enforce Idempotence: By updating f so maps f()) into S [y [(y) у} (updating through first instatiation, Aʃ by expanding the S [y f(y = w} that for given f(=), we get f(y) y (updating through second instantiation Af). If we age the red update while discouraging the green one, we simultaneously map into estimated manifold while tightening it around the data examples. See also fig. 17 for illustration of the gradient pathways.
3

---

Published as a conference paper at ICLR 2024
In order to discourage the incentive to expand the manifold, we only optimize w.r.. the first (inner) instantiation of f. while treating the second (outer) instantiation as a frozen copy of the current state of f. We denote by 0' the parameters of the frozen copy of f. They are different entities, in the sense that a gradient taken w.r.t A will not affect 0'.
equal in value to f but they
Lidem (2; 8 0') óg (fo(2)) = D (for (fo(z)), fo(z))
(8)
We denote the expectation of losses as
fisdem (0:0") = R2 Lidem (z; 0, 0')]
Eq. 8 prevents the encouragement expand the manifold, but are interested in tightening the manifold as much as possible. We therefore maximize the distance between fv) and for given y f()). Effectively, this trains f to exclude that generated y = f(z) from the estimated manifold S (v f'v) = y by optimizing only the second (outer) instantiation of treating the first as frozen copy. The term we want to minimze is then
(10) Notice that (z;g 0') Lidem (z; 0', 0). This induces an adversarial fashion for training them together. However, there is no alternating. Gradients are accumulated on 0 in a single step.
Lzighe 0, 0' -0 (z)) - fa
Final optimization objective. to the final loss:
Combining the three optimization terms described above brings us
0),,0) = (ore Chin)
(11)
= -En, [0((z)= + A. (fe(z)) xroo(fer
with (f) Ex (ffo(z)), x)] being the reconstruction term and X; and X being the weights of the idempotent and tightening loss terms respectively. Note that while the losses are assigned with O' = 0. the gradient which is made of partial derivatives is only w.r.t. the original argument @ of the loss Lidem 02) The general update rule therefore:
0' + 0
(12)
f - 0- 7VL(000))
2.2 TRAINING
For a single model f that appears multiple times in the calculation, we want to optimize by taking gradients of distinct losses w.r.t. different instantiations of f. fig. 2, eq. 13 and fig. 3 all share the same color coding. Red indicates the update of ʃ through its first (inner) instantiation, by minimizing 8. Green indicates the update of ff through its second (outer) instantiation, by maximizing 8; We examine the gradient of de (f)().
vao({(z)) (z) df(-) as((=)) di ((=)
(8(((()) 0S((()) 08(f(z)) as((()) 0f(z) 0f(=)
df(-)
+
(13)
d0
Cright Gradient ascent
Lidem Gradient descent
The two terms of the gradient exemplify the different optimization goals for the different appear- ances of f. Optimizing Liight is done by gradient ascent on the first term while optimizing Lidem is done by gradient descent on the second term.
For Lligtt it is trivial to prevent optimization of the first (inner) instantiation of f. As depicted in fig. 3, it be done by stopping the gradients in the backward process between the two instan- tiations of f, treating f(z) as a static input. This method, however, cannot be applied for Lidem- Eq. 13 shows that the gradient w.r.t the wanted first instantiation of (5(f())) is calculated with 0f(=)
af((())) To cope, fig. shows that we employ copy of the model, foopy. It is updated at every iteration to be identical to f. but as a different entity, we can calculate backpropagation through it without accumulating gradients w.r.t. its parameters.
chain rule through the second a5 af(f()))
fig. and source-code. 2.2 show how the training is performed in practice. For efficiency we first calculate f()) that can be shared by both the idempotent loss and the tightening loss. In source- code. 2.2 we provide the basic training PyTorch code for IGN. This is the actual code used for MNIST experiments, once provided with model, optimizer and data-loader.
4

---

Published as a conference paper at ICLR 2024
X
f(x)
Lrec
fcopy
f(f(z)
Lidem
Z
f())
//
(((
-
Lightt
Stop
Grad
Figure 3: A diagram of the proposed method. The top depicts the reconstruction term real data. The bottom depicts the Idempotence and tightness terms. The colored arrows depict the gradients. The colors match the colors in eq. 13 and fig. 2. Dashed arrow indicates back propagation without accumulating gradients on the parameters it passes through. The final loss is the sum of all the losses.
def train E_copy, opt, data Loader, n_epochs):
for epoch in range (n_epochs);
for data_loader:
2 rorchh.rndd_liki (x)
apply to get all needed
f. s_copy.loare staa__stte_diie (E, sts.stteeddee dict ())
f(x)
fz f(2)
f_ f2.detach ()
ff_z = f(f_z)
f_ f_copy (fz)
calculate losses
loss rec = (fx x) .p.pow mean ()
loss idem
iz) .pow (2) mean ()
loss_tight (ff_z f_z) pow mean ()
18
optimize for losses
loss = loss_ loss_ree rec loss idem + loss loss_tight 0.1
opt wew.ero__ss. zero_grad () loss. backward opt.stap
23
Source Code IGN training routine (PyTorch)
2.3 ARCHITECTURE AND OPTIMIZATION
Network architecture. The typical model be used with IGN is built autoencoder. One possibility is using an existing GAN architecture, "flipping" the order such that the encoder the discriminator, with the binary classification head chopped off, and the encoder is the generator.
Tightening loss metric. One undesirable effect caused by Liight is that benefits from applying big modifications to relatively good generated instance. Moreover, optimized increase distance between input and output encourages high gradients and instability. To ameliorate these issues modify the distance metric for Liight and limit its value. We smooth clamp by
5

---

Published as a conference paper at ICLR 2024
hyperbolic tangent with the value dependent on the current reconstruction loss for each iteration
[Lrighe(z) abreal2)
Liight (z) = -tanh
alIrea(2)
(14)
With Lesght the loss as defined before and 2 constant ratio. The rationale is that at any given time latent that is mapped far out of the estimated manifold, we have reason to push it further.
Noise distribution. We found slight improvement occurs when instead standard Gaussian noise sample noise with frequenc--statistin as the real data. We apply Fast Fourier Transform (FFT) to batch of data and take the mean and variance for the real and imaginary parts of each frequency. We then use these statisties sample and apply inverse FFT get the noise. Examples of how this noise looks like are shown in fig. 4.
3 THEORETICAL RESULTS
Under idealized assumptions, our proposed training paradigm leads to a noteworthy theoretical out- come: After convergence, the distribution of instances generated by the model is aligned with the target distribution. Moreover, the Idempotence loss describes at each step the probability a ran- dom input z to map onto the manifold estimated by the other losses.
Theorem 1. Under ideal conditions, IGN converges to the target distribution. We define the generated distribution, represented by Po(y), as the PDF of y when y = fo(z) and Pa. We split the loss into two terms.
L(0;0') (0) + XeLeighe 0') + Liddem (0:0')
(15)
fire
assume a large enough model capacity such that both terms obtain global minimum:
0 = arg min (0; 0*) arg
argmin min Lidem (0; 0*)
(16)
Then, 30" : Pg- Pa and for >e this is the only possible Pg-.
Proof. We first demonstrate that C,t minimizes the drift 8 over the target distribution while maxi- mizing it at every other fz). Next, we show that Lidem maximizes the probability of f to map to minimum drift areas.
We first find the global minimum of frt given the current parameters #":
Lrr(0;0") = E,, [D[o((x - A,E (=)), (=))
(17)
/ 8n (x)) Xe / 50 (=)) (=)dz
(18)
We now change variables. For the left integral, let == x and for the right integral, let y fo- (z).
Crr(0;00) / 6o(u) xs (o(v))o- (v)dy
(19)
/ bo(w) (v)(w) A.Po- (v) dy
(20)
We denote M sup y2 D(v 12), where the supremum is taken over all possible pairs Vı, Note that M can be infinity, Since dg is non-negative, the global minimum for (rr 0*) is obtained when:
óg- (v) M ((P.(w)<A.PP- (+))
Vy
(21)
Next, we characterize the global minimum of Lidem given the current parameters 0*: Lidem (0,0") 0*) = E, [D (fe- fo())))).(()))) = E. (ōg (ff()))
(22)
Plugging in Eq. 21 and substituting 6* with 0 as we examine the minimum of the inner f:
Lideo (0; 0²) M Be
(23)

---

Published as a conference paper at ICLR 2024
To obtain 8", according to our assumption in Eq. 16, take arg ming of Eq. 23:
6* = .M- arg min E (P
(24)
The presence of parameters to be optimized in this formulation is in the notion of the distribution Po(y). If Pg- = Pa and A I 1, the loss value will be O, which is its minimum. If A = 1, 0 Pg. Ps the only minimizer. This is because the total sum of the probability needs to be 1. Any for which Po(y) < Pu) would necessarily imply that By such that Po(u) (v), which would increase the loss.
Qualitatively, the value de (y) can be thought of as energy, minimized where the probability Pv(y) is high and maximized where P=(y) is low. Under the ideal assumptions, is binary, but in practical scenarios, it would be continuous.
Interestingly, Eq. 23 returns O if 60(y) = which is the definition of being on the estimated mani- fold. This indicator essentially describes the event of fo(z) & Sg. Taking the expectation over the indicator yields the probability of the event. The loss is the probability of random to be mapped outside of the manifold. Optimizing idempotence is essentially maximizing the portion of z's that are mapped onto the manifold.
While the theoretical derivation guarantees a single desired optimum for A = 1, the practical optimization of a finite capacity neural network suffers undesirable effects such instability. The fact that is continuous makes the optimal theoretical 0* which produces discontinuous 5g. unobtainable in practice. This means that Lright tends to push toward high values of (y) also for y that is in the estimated manifold. Moreover, in general, it is easier maximize distances than minimize them, just by getting big gradient values.
In practice, we use Ae
4 EXPERIMENTAL RESULTS
Following the training scheme outlined in Sections 2.2 and 2.3, we train IGN on two datasets CelebA and MNIST. We present qualitative results for the datasets, as well out-of-dist projection capabilities and latent space manipulations.
Our generative outcomes, this stage, periments currently operate with smaller models and lower-resolution datasets. In our exploration, primarily focus on a streamlined approach, deferring additional mechanisms maintain the purity of the primary method. It's worth noting that foundational generative modeling techniques, like GANs Goodfellow et al. (2014) and Diffusion Models Sohl-Dickstein al. (2015), took con- siderable time to reach their matured, scaled-up performance. We view this as preliminary step, providing initial evidence of the potential capabilities. Our future work will aim at refining and scaling up the approach.
competitive with sate-of-the-art models. Our
Experimental settings. We evaluate IGN on MNIST (Deng, 2012), a dataset of grayscale hand- written digits, and CelebA (Liu et 2015), dataset of face images. We use image resolutions of 28 x 28 and 64 x 64 respectively. We adopt a simple autoencoder architecture, where the encoder is simple five-layer discriminator backbone from DCGAN, and the decoder is the generator. The training and network hyperparameters are presented in Table 1.
z
f(z)
9
5
f(f(z))
6
9
5
Figure 4: Examples of MNIST and CelebA IGN generations from input Gaussian noise for IGN, Notice that some cases f(f(z)) corrects for artifacts f()).
7

---

Published as a conference paper at ICLR 2024
++ m f(x+n) f(f(r + n))
(x)
f((())) f(f((())))
s(x)
f(s(x))
f(f(s(x))
Figure 5: Projections of images from different distributions using IGN. We demonstrate that IGN can project noisy images + n (left), grayscale images g(x) (middle), and sketches s(x) (right) onto the learned natural image manifold perform image-to-image translation. See appendix for details the degradations.
Generation results. Figure presents qualitative results for the two datasets after applying the model once and consecutively twice. We report FID=39 (DCGAN FID=34). As shown, applying IGN once (f(z) results in coherent generation results. However, artifacts can be present, such as holes in MNIST digits, or distorted pixels at the top the head and hair in the face images. Applying again (z)) corrects for these, filling in the holes, reducing the total variation around noisy patches in the face. Figure 7 shows additional results, as well as applying ff three times. Comparing (f)()=))) f(((2) shows that when the images get closer the learned manifold, applying again results minimal changes. See large uncurated collection Figures 11-14.
Latent Space Manipulations. We demonstrate IGN has consistent latent space by performing manipulations, similarly as shown for GANs (Radford et al., 2015). Latent space interpolation videos can be found in the supplementary material. We sample several random noises, take linear interpolation between them and apply f. In The videos left to right: 2, (((()) Fig. shows latent space arithmetics. Formally, we consider three inputs &positive Enegative and 2, such that f( ((peaitive) has specific image property that f (Zmegative) and f(z) do not have (e.g. the faces in the two former images have glasses, while the latter does not have them). The result of J( (Zpositive Enegative z) is an edited version of f(z) that has the property.
onttrf-Distribution Projection. We validate the potential for IGN as a "global projector" by inputting images from a variety of distributions into the model to produce their "natural image" equivalents (i.e.: project onto IGN's learned manifold). We demon- strate this by denoising noised images n, coloriz- ing grayscale images g(x), and translating sketches s(x) realistic images in Fig. 5. Although the pro- jected images are not perfect reconstructions of the original images v, these inverse tasks ill-posed. IGN is able to create natural-looking projections that adhere the structure of the original images. As shown, sequential applications of ff can improve the image quality (e-g. it removes dark and smoky arti- facts in the projected sketches). Note that IGN was only trained on natural images and noise, and not see distributions such as sketches or grayscale images. While other methods explicitly train for this task (Zhu et al., 2017; Isola et al., 2017), this behavior naturally emerges in IGN product of its projection objective. Moreover, due to the autoencoding architecture IGN, we do not need to rely on inversion for editing. Instead, rely solely forward passes through the network.
Glasses Direction
Smile Direction
Pose Direction
Input noise arithmetic. Similar GANs, arithmetic operations can performed in the input noise space to idempotent generative networks to find interpretable axes of variation.
Figure 6:
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
Generative Adversarial Networks (GANs). IGN incorporates elements of adversarial train- ing (Goodfellow et al., 2014), evident in the relationship between Lidem and Lightt. which are neg- atives of each other. One could view as discriminator trained using free for real examples and Light for generated ones, while f serves as the generator trained by Lidem- Unique to IGN is a form of adversarial training we term "self-adversarial"" Put simply. f is both the generator and the discriminator. This streamlined architecture affects the optimization process. Rather than alter- nating between two networks, single model accumulates gradients from both discriminative and generative perspectives in each step.
Energy Based Models (EBMs). In Energy- Based Models (EBMs; Ackley et al. (1985)), a func- tion I is explicitly trained to serve as an energy metric, assigning higher values to less desirable examples and lower values to those that fit the model well. IGN introduces similar, yet distinct paradigm: rather than ʃ acting as the energy function, this role is filled by d(y) = D(f(y), y)- The model trains f to be idempotent, with the objective to minimize (f)).. A successful training pro- cedure would align the range of f with the low-energy regions as measured by 5. This reduces the need for separate optimization procedures to find the energy minimum. From another perspective, f can be viewed as transition operator that maps high-energy inputs toward a low-energy domain.
Energy Based Generative Adversarial Network (EBGAN) Zhao et al. (2017). Combining GANs and EBMs, EBGAN is the closest existing model to IGN. EBGAN is GAN where the discriminator is built as an autoencoder. The discriminator Like IGN, uses a reconstruction loss instead of a binary loss. The EBGAN discriminator is not trained project as it only sees real or generated images, but not latents or out-of-distribttion instances, It is trained to discriminate. IGN differs in its mentioned self-adversarialness having as both the generator and critic. The same model judges the quality while trying to improve it.
Diffusion Models. In both diffusion models (Sohl-Dickstein et al., 2015) and IGN, the model can be sequentially applied. Additionally, both methods train the model transition input along a path between a source distribution and a target data manifold. In diffusion models, this path is dictated by a predefined noise schedule. At inference, the model takes small, incremental steps, effectively performing a form of gradient descent to transition from complete noise representing the source distribution--to the target data manifold. IGN diverges from this approach. Instead of predefined path dictated by noise schedule or any other set rule, the trajectory between distributions determined solely by the model's learning process. Unlike diffusion models, IGN doesn't 't employ incremental gradient steps toward the data manifold. Instead, it is trained to approximate as closely as possible the target manifold in a single step. can be reapplied for further refinement if needed.
LIMITATIONS
Mode collapse. Similar to GANs, our model can experience mode collapse and is not practically guaranteed to generate the entire target distribution. Some methods attempt to overcome this failure mode in GANs (Mao al., 2019; Durall et al., 2020). We plan to investigate if these methods are applicable to our generative model as well.
Blurriness. Similar to VAEs and other autoencoders, our model suffers from blurry generated samples. Although repeated applications can fix artifacts to make images appear more natural, they may also smoothen them towards an werage-looking image. One possible solution to this problem to replace the naive reconstruction loss with perceptual loss Johnson et al. (2016). Another solution is to use a two-step approach and apply our model on latents instead of pixels (similar Rombach et al. (2021)). We plan to investigate future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors would like to thank Karttikeya Mangalam, Yannis Siglidis, Konpat Preechakul, Niv Haim, Niv Granot and Ben Feinstein for the helpful discussions. Assaf Shocher gratefully acknowl- edges financial support for this publication by the Fulbright U.S. Postdoctoral Program, which is sponsored by the U.S. Department of State. Its contents are solely the responsibility of the author and do not necessarily represent the official views of the Fulbright Program the Government of the United States. Amil Dravid is funded by the US Department of Energy Computational Sci- ence Graduate Fellowship. Yossi Gandelsman is funded by the Berkeley Fellowship and the Google Fellowship. Additional funding came from DARPA MCS and ONR MURI.
REFERENCES
David H. Ackley Geoffrey E. Hinton, and Terrence Sejnowski. A learning algorithm for boltzmann machines. Cogmitive Science, 9():1477-69, 1985, ISSN 0364-0213, doi: hitps:
URL https: //ww:///wwww/./.././//// sciencedirect. com/ scciene/arrricee//////6022222333
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012. 2012.
Ricard Durall, Avraam Chatzimichailidis, Peter Labus, and Janis Keuper. Combating mode collapse in GAN training: An empirical analysis using hessian eigenvalues. CoRR, abs/2012.09673, 2020. URL https::/// www.iaaxivvoo.oo org/ abs/2012...2/ 09673.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez, Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., inw.oeeeeediinn..nniiisss.s.s.ss nwwrips. wwpppppr..........e/..e files/paper/2020
2020. URL file/ http://cccccceeeaaaaaaaaaaaaaaa1
Phillip Isola, Yun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution, 2016.
Diederik Kingma and Max Welling, Auto-encoding variational bayes, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Qi Mao, Hsin-Ying Lee, Hung-Yu Yu Tseng, Siwei Ma, and Ming- Hsuan Yang. Mode seeking gen- erative adversarial networks for diverse image synthesis. CoRR, abs/1903.05628, 2019. URL http:// http:://arxvv... wwwp/bs//////oeeeeoooeee abs/ /100/ 05628.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CORR, abs/1511 1.06444, 2015. URL https: //api. semant www.oolacccccaara :ttp:/CoopsS::1115555
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
10

---

Published as a conference paper at ICLR 2024
Jascha Sohl- Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.). Pro- ceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL https://proeeeiioee
httpss:
mlr.press v37
Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukeuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In D. Lee, M. Sugiyama, U. Luxburg. Guyon, and Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016a. URL https: fi
http:s::///ccceedeeeseeeee www.:eeddddinnn.nnn.seee.e...... wwwrine.rr....eo. ctlesscc//pee wttp//pppeeeeee/epepepeeeeeeeeee files/paper/2016. Faa.
.pf..
Aaron den Oord, Oriol Vinyals, and koray kavukeuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach. R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https: http:/poooeeddinngee neur wttpoorrrp.ccppppppeeeeeeepp wwtppppr_aiiiiaeeeeeeeeee ffleef www.rfccpapppp........
paper/2017/file 7 Caa wwwp::eooeeeeeeeeeeeeeee
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural net- works. In Maria Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learn- ing Research, pp. 1747-1756, New York, New York, USA, 20-22 Jun 2016b. PMLR. URL ht.pps: http:///cceeeeeeg.eeeeem www.mrrss.............m..mmm...e / ooor16 6 html.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial networks. 2017. Publisher Copyright: ICLR 2019 Conference Track Proceedings. All rights reserved.; 5th International Conference on Learning Representations, ICLR 2017: Conference date: 24-04- 2017 Through 26-04-2017.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.

---

Published as a conference paper at ICLR 2024
A
VISUAL COMPARISON OF ITERATIVE APPLICATIONS OF f
Also see videos supplementary material.
f(z)
f(f(z))
f(f(f(2)))
Figure 7: Comparison of iterative applications f. As the generated images approach the learned manifold, sequential applications of f have smaller effects on the outputs.
1
1
5
6 8
9
f(z)
4/234
/
2
5
6
9
f(f(z)
F(F(f()))
4
6
2
8
Figure 8: Comparison of iterative applications of f on MNIST.
12

---

Published as a conference paper at ICLR 2024
B
MORE PROJECTIONS
X
X
f(x)
f((())
Figure 9: Projection-base basee edits. By simply masking out a region of interest and adding noise for stochastic variation, conduct fine-grained edits, such as closing the mouth, adding hair or facial hair, and putting on glasses.
f(z)
X mask
f (xmast)
Figure 10: Projection-based compositing. Given reference image f(z), we can use the noise spatially corresponding an attribute of interest, place on another image x, and project order to transfer an attribute, such as glasses, facial hair, etc.
13

---

Published as a conference paper at ICLR 2024
C
IMPLEMENTATION DETAILS
Operation
Kernel
Strides
Padding
Feature maps
BN?
Nonlinearity
Encoder -3 x 64 x 64 input
Convolution
4
x
64
Leaky ReLU
Convolution
4 4x4 4
2 2
I
128
Leaky ReLU
Convolution
4
2x
1
256
Leaky ReLU
Convolution
4 Ax4 4
2x
1
512
Leaky ReLU
Convolution
4x4 4
1x1
0
512
None
Decoder - 512 x 1 X 1 input
Transposed Convolution
4x 4
1x1
0
512
ReLU
Transposed Convolution
4X 4
2x2
1
256
ReLU
Transposed Convolution
4
2x2
128
ReLU
Transposed Convolution
4x 4
2
1
64
ReLU
Transposed Convolution 4x4
2
1
3
Tanh
Loss metric D L1: D1:1
82/11
Loss terms weights A 20, X
20,
X
2.5
Lthight clamp ratio a a= 1.5
Optimizer Adam (a 0.0001, 31 0.5, B2 0.999)
Batch size 256
# GPUs 8
Iterations 1000
Leaky ReLU slope 0.2
Weight, bias initialization Isotropic gaussian (μ1 = 0, 0.02), Constant(())
Table 1: CelebA-10 hyperparameters. We train simple autoencoder architecture with minimal hyperparam-
eter tuning.
Degradations.
The images are scaled to values [-1,
Noise:
We add Gaussian noise (0,0 0.15)
Grayscale: We take the mean each pixel over the channels and assign to each of the three channels the model expects three channels.
g(x) = x.mean dim = 1, keepdim = "rrue repeat),, 3, 1)
Sketch: We divide the pixel values by the Gaussian blurred image pixel values with kernel size of 21. The standard deviation is the default w.r.t. kernel size by PyTorch: a 0.3 ((kernel size 1) 0.5 1) + 0.8. we add and subtract to perform the division on positive values.
s(x) = (g(x + 1)/ (gausssian + 1), 21)) + 10) 1.
14

---

Published as a conference paper at ICLR 2024
D UNCURATED VISUAL RESULTS
Figure 11: Uncurated CelebA samples from applying IGN once: f()).
15

---

Published as a conference paper at ICLR 2024
TL
Figure 12: Uncurated CelebA samples from applying IGN twice: f(f(2)).
16

---

Published as a conference paper at ICLR 2024
Figure 13: Uncurated CelebA samples from applying IGN three times: f(f(f())))
17

---

Published as a conference paper at ICLR 2024
Figure 14: Uncurated CelebA samples from applying IGN four times: f(f(f((())))
18

---

Published as a conference paper at ICLR 2024
Regular DCGAN Architecture
full connected
roshape
PSC
FSC
rSC
}
64x64x3 x2 128 16x 256
8x8x
4
Generator Network
Discriminator Network
Flipped DCGAN Architecture, used in IGN for CelebA
conV
con
FSC
Discriminator Network
Generator Network
Figure 15: Illustration architecture used for CelebA generation. Top shows original DCGAN architec- ture. Bottom shows that we chop the binary decision head of the discriminator and flip the order to create an hourglass-like architecture. The exact sizes of all the layers are in table 1
19

---

Published as a conference paper at ICLR 2024
0
97
6
2
5
4
1
9
10
9
20
100
1000
10000
E
100000
8
300000
Figure 16: Stress lesting by extreme sequential application of IGN. We check what happens for f*((z for 30. The numbers on the left indicate the number of sequential applications. For the first few tens of iterations, the system seems to be stable but around 100 it diverges from the result of the first application. We hypothesize that the size of the model which influences the ability to get low reconstruction loss determines the stability.
20

---

Published as a conference paper at ICLR 2024
Z
f(f(z)
D
Lidem
θ
Map onto manifold
Expand the manifold
Z
f(z)
copy
f(f(z))
D
Lidem
Map Z to manifold
Stop Grad
//
f(f(z)
D
Ltight
0
Tighten manifold
An illustration supporting Fig. 2. Top: Naive implementation using Cisem without Leight Show- ing two paths of gradient- baadeent--ased based optimization of 6, the parameters of f. The update for simply consists of sum of gradients. The red gradient optimizes 0 s.t. ff maps better the current estimated manifold. The green gradient optimizes so that
Figure 17:
expand contain it. Bottom: Illustration gradients IGN. explained in 2.2, The idempotence term is split so that we can negate the term that expands the estimated manifold in the top illustrations, and therefore tighten it.
given f(z) the manifold
21


Reviews:
[{"review_id": "LNE9jfvBw2", "reviewer": "Reviewer_Sz9m", "summary": "This work presents what appears to me to be a novel way to model the generative process from noise (or corrupted images) using idempotence as the underlying principle for the objective(s). Notably, they train a model which at optimum should maintain given datapoints from the target distribution, yet generate samples whose manifold is maximally \"tight\". These objectives together yield a novel generative model algorithm, and there is theoretical and empirical support.\n\n--- Update post rebuttal / discussion\n\nI've looked over the discussion of EBGAN vs IGN, as well as revisited the EBGAN paper and reread the submission. While the public commenter's insights are useful, I feel that the reviewers have done a good job addressing them: the motivation of IGN is certainly different from those of EBGAN's, and the story is still clear and easy to follow. I'm pleased we've found a way to think about training generative models that leads to an algorithm that is very similar to a prior work, but I do see the author's points about the differences, both in terms of how the model is optimized and how it is evaluated, as being well aligned with the overall story of the paper. The added discussion wrt EBGAN also is important and sufficient.\n\nI'm less pleased with the intention of the public reviewer to bring the story under alignment EBGAN. It's true there are many open questions from the discussion, but it isn't necessarily the job of this paper to do so. It's an interesting motivation for generative modeling that bears fruit in something that is close, but not precisely the same as (nor subsumed by) prior works. If the public commenter wants to delve into this problem more, I'd encourage them to do so, rather than making that a criteria for acceptance to this conference.\n\nMoreover, EBGAN is in no way harmed by how the paper is presented. No incorrect claims are made that I can find, and the motivation (idempotence) *stands on its own*.", "questions": "Why does M:= sup(D(y1, y2)) show up in eq 19? Are we saying if y is outside the target data manifold + the margin that the repeated applications of f should maximize the distance? How is this guaranteed?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall I found the approach to be appealing and interesting. Though there isn't necessarily a strong compelling reason here why to use idempotence to train a generative model, I don't find this as a weakness (prefacing here), as there is scientific curiosity here, the intuition is compelling, the theory seems correct wrt the optima fitting the target distribution, and the empirical results support everything neatly. I'd also mention that the results are stellar / SOTA, but as noted in the paper: this is fine. Engineering might improve things or it might not, just generally it looks like good science with a solid story and results to support.", "weaknesses": "Some points could be clarified, notably I found the discussion around Figure 2 to be less clear than I would have liked. I think I understand the general gist, but I honestly didn't think I understood how the different objectives work together until I got through the theory section. Could this be cleaned up a bit / made a bit more clear? I'd enjoy a bit of some discussion here to see if we can arrive at a better way to present this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wuR76FeQvW", "reviewer": "Reviewer_Cz9y", "summary": "This paper presents a new kind of generative model for image synthesis called the Idempotent Generative Network (IGN). The key idea is to learn a function $f$ such that $f(x) = x$ if $x$ is on the data manifold, and $f(z)$ is on the data manifold if $z \\sim P_0$ for some prior $P_0$ (in which case $f(f(z)) = f(z)$). The network $f$ is learned with three loss terms: a reconstruction loss to encourage data samples to be mapped to themselves, an idempotent loss to encourage two applications of $f$ to a state $z$ from the prior to match a single application of $f$, and a tightness objective encouraging $f$ to map any state $z'$ not on the manifold as far away from the manifold as possible. A theoretical analysis is presented which claims that the proposed method will generated samples from the data distribution if $f$ is perfectly trained and has enough capacity. Experimental results show that the proposed method can generate MNIST and CelebA images from noise.", "questions": "Please see my questions about the theoretical justification of the method in the weaknesses section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The work is an ambitious and refreshing approach to generative modeling. Idempotence is an interesting foundation for building a generative model. Overall the paper was enjoyable to read.\n* The central ideas are clearly presented and I could easily follow the training algorithm and motivation of the method.\n* Empirical results show that the method is capable of image generation and editing the features of generated faces.", "weaknesses": "* I am not fully convinced of the theoretical validity of the method. In particular, Equation (19) does not appear fully rigorous. From my understanding, the value $M$ is the maximum distance between points in the state space. \n  * What if the state space is unbounded? Should we expect states to be restricted to an image hypercube?\n  * Why is it the case that $max_{\\theta*} \\delta_{\\theta*}(y) = M$ for any $y$ when $P_x (y) < \\lambda_t P_{\\theta*} (y)$? Wouldn't that mean the distance between $y$ and the furthest point from $y$ in the state space is $M$ for each $y$? I am not sure that makes intuitive sense to me. It seems that $M$ should not be a constant but rather a function $M(y)$ measuring the distance between $y$ and the furthest point in the state space, which greatly complicates the analysis that follows.\n\n  The proof heavily relies on the simple form of (19), but I am not sure this simple form is valid. Furthermore, in practice the distance $M$ is heavily restricted which leaves a gap between the theoretical development of the method and practical implementation. Since this method departs heavily from the established probabilistic principles used to justify generative models, clearly demonstrating the validity of the method is essential. I am willing to raise my score if these points can be addressed.\n* Although I understand that the current results are meant to be a proof of concept, it would still be good to provide a more formal comparison with prior generative models e.g. by calculating FID scores on CelebA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "diPJnAIYvD", "reviewer": "Reviewer_5afM", "summary": "This paper introduces a new class of generative model that is based on idempotent, i.e., $f(f(x)) = f(x)$. It acts like a projection function that project anything to the target data manifold, and it preserves identity for data already in the target manifold.To train such a model, first a reconstruction loss is used to let the model map latent z to the data manifold, and an idempotent loss to ensure the property of idempotent. An additional tightness loss is used for regularization. Some preliminary results of generation is shown.", "questions": "My questions are stated in the weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The biggest strength is that the idea is novel and refreshing. It introduces a new class of generative models with an interesting formulation.\n\n2. Overall the presentation is good. The idea is clearly stated in the pseudo code. The introduction of idempotent is clear and interesting in the first section.\n\n3. Although the results are preliminary, there are some interesting observations. For example, the method can naturally take things other than noise as input (e.g., Figure 5 c)", "weaknesses": "1. The biggest issue is that I don't get the motivation for designing such a model. When one designs a new class of generative model that is different from any previous approaches, it is necessary to clarify that it has the potential to address some limitations of previous approach. As a preliminary study, it does not need to have great results, but conceptually the advantage of the model has to be stated. Otherwise, it is just \"another class of generative models\". Since it suffers the same mode collapse and instability issue of GAN, the motivation becomes more important. Why do we need such a model? What are the potential advantages?\n\n2. I don't quite get the point of the tightness loss. Why maximize it is good for regularization? More explanation is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYCZhq4Aer", "reviewer": "Reviewer_j2FZ", "summary": "This work proposes a novel approach to generative modeling based on the idea of idempotence. An idempotent function $f: X \\rightarrow X$, is one for which $f(f(x)) = f(x)$ for any $x \\in X$. The aim of the model proposed in the paper, Idempotent Generative Network (IGN), is to achieve idempotence so that for any $x$ drawn from the source distribution, $f(f(x)) = f(x)$. The paper provides several justifications for why this property is desirable in the context of generative modeling that essentially amount to IGN having many of the nice properties held individually by either GANs, diffusion models, etc. The paper has an interesting discussion of the loss function needed to train such a model and provides one theoretical result justifying the optimization approach. Several experiments are run, including use of the idempotent property of IGN to project noisy or corrupted examples back to the target distribution.", "questions": "- One of the claimed strengths of IGN is that through the idempotence property, it can be used to project noisy or corrupted images back to the data manifold. Based on the reviewer’s understanding of the capability, there are already a range of methods for doing this (e.g., via image denoising with a diffusion model). Is there some additional nuance that the reviewer is missing? It is certainly the case that existing methods will not map an image plus noise $x + \\epsilon$ by to $x$ exactly, but it was our sense from reading the paper that neither would IGN.\n- It would be interesting to see the behavior of $f^{k}(x)$ as $k \\rightarrow \\infty$, does this converge or diverge? If it does converge, what does it converge to?\n- This reviewer did not understand the modified GAN network architecture described in Section 2.3.\n- As the authors note, the development of high-performing generative models has come about through numerous iterative improvements. Would it be possible to build an IGN model using the framework of a known generative approach?\n- What are the current drawbacks to scaling this method up?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Underlying idea:** The use of idempotence in deep learning is appealing given its deep connection to mathematics and theoretical computer science. Because deep learning is essentially a science of function composition, connecting it to category theory and related domains focused on composition in pure mathematics is a rich area for exploration. This is a great example of what first steps in that direction look like. Since idempotents have been studied in great depth, there is likely many additional constructions that could be explored using this work as a starting point.\n- **Simplicity of construction and clarity of writing:** IGN is described very clearly so that readers from a range of backgrounds can understand the constructions. The different parts of the model (particularly the loss function terms) are well-justified both mathematically and informally in the text. As might be expected when researchers utilize a truly foundational idea, constructions are surprisingly simple (as was the case in this paper). \n- **Quality of the outputs:** Despite the fact that this is a novel generative framework, this reviewer felt that the outputs looked quite good. It was especially exciting to see that that $f(f(f(x)))$ looked mostly the same as $f(x)$. As described below, the reviewer would be interested to see $f^k(x)$ as $x \\rightarrow \\infty$. \n- **Opening paragraph:** Though it is perhaps a small thing, this reviewer really appreciated the opening quote. It captures the underlying idea of the paper and made me laugh.", "weaknesses": "- **Motivation:** While the idea is interesting and utilizes foundational structures from mathematics, the motivation for idempotence in generative models is still a little weak. Several different consequences are described in the introduction (IGN provides outputs in a single step, yet allows additional refinements, etc.) Each of these properties is attractive, but already possessed by an existing mature method (as is pointed out in the paper). It was unclear to this reviewer whether starting from scratch with a completely new paradigm is the right way to obtain the desirable properties of GANs, diffusion models, etc. On the other hand, idempotence is a deep idea whose consequences have been explored in a wide range of settings and directions. It feels likely to this reviewer that there are good reasons for wanting idempotence in a model that go beyond properties we already have in alternative methods. However, these may yet need to be uncovered.\n- **Comparison to other methods:** While the reviewer agrees that it is not fair to hold this novel approach to the standards of more mature methods, it would still be useful to be able to compare the performance of IGN with known Approach. As it is, this reviewer was unable to tell how far IGN is from the performance of a comparably sized GAN, VAE, or diffusion model. \n- **Quantitative results:** It appears that the only way the reader can evaluate the experiments is through a handful of example generations. While these are helpful for getting a qualitative sense of the model (and look good, as described in the Strengths section), it would have made the analysis stronger if quantitative results were also presented. It would be interesting, for instance, to understand network performance for different types of initial $x$, especially since the domain of the network includes both $\\mathcal{P}_z$ and $\\mathcal{P}_x$. It would have also been interesting to quantitatively measure differences in $f^{(k)}(x)$ as $k \\rightarrow \\infty$.\n\n### Nitpicks\n\n- Idempotence is spelled wrong in the sentence below equation (7).\n- The tightness objective is a little subtle. It might make sense to utilize some of the notation already introduced (e.g., $\\mathcal{S}$) to help communicate why this loss term is necessary. \n- It may be a matter of personal taste, but this reviewer felt that while the PyTorch code was appreciated, it might be better to include in the Appendix and instead include additional analysis of the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)