LOGAN:  Latent Optimisation for Generative Adversarial Networks
Authors: Yan Wu
Jeff Donahue
David Balduzzi
Karen Simonyan
Timothy Lillicrap
Venue:  (2020)
Keywords: GAN
adversarial training
generative model
game theory

Abstract: Training generative adversarial networks requires balancing of delicate adversarial dynamics. Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes. In this work, we introduce a new form of latent optimisation inspired by the CS-GAN and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator. We develop supporting theoretical analysis from the perspectives of differentiable games and stochastic approximation. Our experiments demonstrate that latent optimisation can significantly improve GAN training, obtaining state-of-the-art performance for the ImageNet (128 x 128) dataset. Our model achieves an Inception Score (IS) of 148 and an Frechet Inception Distance (FID) of 3.4, an improvement of 17% and 32% in IS and FID respectively, compared with the baseline BigGAN-deep model with the same architecture and number of parameters.

Under review as a conference paper at ICLR 2020
LOGAN: LATENT OPTIMISATION FOR GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Training generative adversarial networks requires balancing of delicate adversar- ial dynamics. Even with careful tuning, training may diverge or end up bad equilibrium with dropped modes. In this work, we introduce new form of la- tent optimisation inspired by the CS-GAN and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator. We develop supporting theoretical analysis from the perspectives of differentiable games and stochastic approximation. Our experiments demonstrate that latent op- timisation can significantly improve GAN training, obtaining state-of-the-arr per- formance for the ImageNet (128 128) dataset. Our model achieves Inception Score (IS) of 148 and an Fréchet Inception Distance (FID) 3.4, an improve- ment of 17% and 32% in IS and FID respectively, compared with the baseline BigGAN-deep model with the same architecture and number of parameters.
INTRODUCTION
Generative Adversarial Nets (GANs) are implicit generative models that can be trained to match a given data distribution. GANs were originally proposed and demonstrated for images by Good- Tellow et al. (2014). As the field of generative modelling has advanced, GANs have remained at the frontier, generating high-fidelity images at large scale (Brock et al., 2018). However, despite growing insights into the dynamics of GAN training, most recent advances in large-scale image generation come from architectural improvements (Radford et al. 2015 Zhang et al., 2019), or reg- ularisation focusing on particular parts of the model (Miyato et al. 2018 Miyato & Koyama 2018). Inspired by the compressed sensing GAN (CS-GAN; Wu et al. 2019). further exploit the ben-
cfit of latent optimisation in adversarial games using natural gradient descent to optimise the latent variable at each step of training, presenting scalable and easy implement approach to improve the dynamical interaction between the discriminator and the generator. For clarity, we unify these approaches as latent optimised GANs (LOGAN).
To summarise our contributions:
1. We present a novel analysis of latent optimisation in GANs from the perspective of differ- entiable games and stochastic approximation (Balduzzi et al. 2018 Heusel et al.) 2017). arguing that latent optimisation can improve the dynamics of adversarial training.
2. Motivated by this analysis, we improve latent optimisation by taking advantage of efficient second-order updates.
3. Our algorithm improves the state-of-the-art BigGAN-deep model (Brock et al. 2018) by significant margin, without introducing any architectural change additional parameters, resulting in higher quality images and more diverse samples (Figure [ and (
2 BACKGROUND
2.1 NOTATION
We use In and @ to denote the vectors representing parameters of the generator and discriminator.
We use for images, and for the latent source generating image. The prime is used to denote

---

Under review as a conference paper at ICLR 2020
(a)
(b)
Figure 1: Samples from BigGAN-deep (a) and LOGAN (b) with similarly high IS. Samples from the two panels were drawn from truncation levels corresponding to points C and D in figure respectively. (FID/IS: (a) 27.97/259.4, (b) 8. 891/299.
(a)
(b)
Figure 2: Samples from BigGAN-deep (a) and LOGAN (b) with similarly low FID. Samples from the two panels were drawn from truncation levels corresponding to points and in figure respectively. (FID/IS: (a) 5.04/126.8, (b) 5.09/217.0)
a variable after one update step. e.g., 8 Op a 0f(=:0p.0). p(x) and p(z) denote the data 80p
distribution and source distribution respectively. Ep(x) [f(z)] indicates taking the expectation of function f(x) over the distribution p(z).
2.2 GENERATIVE ADVERSARIAL NETS
A GAN consists of a generator that generates image G(z; from latent source p(=), and discriminator that scores the generated images as D(:; (Goodfellow al. 2014). Training GANs involves adversarial game: while the discriminator tries distinguish generated samples
I = G (2; 0G) from data x ~ p(z), the generator tries to fool the discriminator. This procedure can be summarised as the following min-max game:
min max Ex<p(z) (DD(D (z; 0D))] Exsp(s) Iha(D (G; 0p))]
Op
2

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison of model scores. BigGAN-deep results are reproduced from Brock et al. 20118) "baseline" indicates our reproduced BigGAN-deep with small modifications. The 3rd and 4th columns are from the gradient descent (GD, ablated) and natural gradient descent (NGD) ver- sions of LOGAN respectively. We report the Inception Score (IS, higher is better, [allmaans et al. 2016) and Fréchet Inception Distance (FID, lower is better, [Heusel et al. 2017).
BigGAN-Deep
baseline
LOGAN (GD)
LOGAN (NGD)
FID
5.7 + 0.3 124.5 +: 2.0
4.92 + 0.05
4.86 + 0.09
3.36 0.14
IS
126.6 + 1.3
127.7 + 3.5
148.2 + 3.1
The exact form of h(-) depends the choice of loss function (Goodfellow et al. 2014 Arjovsky et al.] 2017 Nowozin et al. 2016). To simplify our presentation and analysis, we use the Wasserstein loss (Arjovsyy al. 2017), , that h() - and (t) I. Our experiments with BigGAN- deep uses the hinge loss (im & Yo 2017 Tran et al., 2017). which is identical to this form in its linear regime. Our analysis can be generalised to other losses as in previous theoretical work (c.g., Arora et al. 2017). To simplify notation, we abbreviate f(z; 0D fg) D (G (2; 0g) :dD), which may further simplified as f(z) when the explicit dependency on OD and Ag can be omitted.
Training GANs requires carefully balancing updates to D and G. and is sensitive to both architecture and algorithm choices (Salimans et al., 2016 Radiord et al. 2015). recent milestone is BigGAN (and BigGANN-dee Brock et 20.(2018). which pushed the boundary of high fidelity image generation by scaling up GANs to an unprecedented level. BigGANs use an architecture based on residual blocks (He et al. 2016), in combination with regularisation mechanisms and self-attention (Saxe ct al. 2014 Miyato et al., 2018 Zhang et al. 2019).
Here aim to improve the adversarial dynamics during training. We focus the second term in eq. [ which is at the heart of the min-max game, with adversarial losses for and G, which can be written
l(z) = [=)( La(=)) = [[==),=(()))
(2)
Computing the gradients with respect to OD and Og obtains the following gradient, which cannot be expressed as the gradient of any single function (Balduzzi et al. 2018):
[LD(2) OLG(2)
0f(z) 0f(=) T
g =
(3)
30p
30g
a0,- a0g
The fact that is not the gradient a function implies that gradient updates in GANs can exhibit cycling behaviour which can slow down or prevent convergence. In (Balduzii et al. (201)) vector fields of this form are referred to as the simultaneous gradient. Although many GAN models use alternating update rules (e.g.. Goodfellow et al. 2014;; Brock et al. 2018), following the gradient with respect to dD and GG alternatively in each step, they can still suffer from cycling, so we use the simpler simultaneous gradient (eq. 3 for our analysis.
2.3 LATENT OPTIMISED GANS
Inspired by compressed sensing (Candes al.) 2006 Donoho 2006). Wu et al. (2019) introduced latent optimisation for GANs. We call this type model latent-optimised GANs (LOGAN). Latent optimization has been shown to improve the stability of training well as the final performance for medium-sizze sized models such as DCGANs and Spectral Normalised GANs (Radford et al. 2015 Miy-
ato al. 2018). Latent optimisation exploits knowledge from D guide updates of 2. Intuitively,
the gradient V fz) (f()) better samples. Therefore, instead of using the randomly sampled 2 ~ p(2). Wu et al. (2019) uses the optimised latent
az points in the direction that satisfies the discriminator D, which implies
A Af(z)
2'=2tAe
(4)
Oz
! The general algorithm is summarised in Algorithm and illustrated in Figure a. We develop the natural gradient descent form latent update in Section
in eq. I for training]]
We single step of bradient-based optimisation during training, and justify this choice in section
3

---

Under review as a conference paper at ICLR 2020
of truncation
BgGAN-Deep LOGAN/GUI LOGANINGD)
f()
f(z')
D
D
x
.
G
G
z
2
150
Inception Score
(a)
(b)
Figure 3: (a) Schematic of LOGAN. We first compute forward pass through G and D with sampled latent z. Then, gradients from the generator loss (dashed red arrow) are used to compute an improved latent, z'. After this optimised latent code is used in a second forward pass, we compute gradients of the discriminator back through the latent optimisation into the model parameters 0p, Og- These gradients are used to update the model. (b) Truncation curves illustrate FID/IS trade- off for each model by altering the range of the noise source p(z). GD: gradient descent. NGD: natural gradient descent. Points A, B, C, correspond to samples shown Figure and
Algorithm Latent Optimised GANs with Automatic Differentiation
Input: data distribution p(z), latent distribution p(2), D (:; (:;0p), G (:;6), learning rate a, batch size N
repeat
Initialise discriminator and generator parameters 0p, 0g for i 1 to N do
Sample ~ p(z). x ~ p(x:
Compute the gradient @D())))) and use it to obtain A. from eq. 4 (GD) or eq. 12 (NGD)
Optimise the latent
+ Az]. [:] indicates clipping the value between -1 and
Compute generator loss L D(G(=))
Compute discriminator loss D = D(G(z)) - ((x) end for
Compute batch losses LG N Σm (0 and LD 1 ΣN (
N
Update dD and fG with the gradients OLn OLa until reaches the maximum training steps
89c
3 ANALYSIS OF THE ALGORITHM
To understand how latent optimisation improves GAN training, we analyse LOGAN as 2-player differentiable game following Balduzzi et al. (2018); Gemp & Mahadevan (2018) Letcher et al. 2019). The appendix provides a complementary analysis that relates LOGAN to unrolled GANs MMtz et al. 2016) and stochastic approximation (Heusel al. 2017 Borkar 1997
3.1 THE SYMPLECTIC GRADIENT ADJUSTMENT (SGA)
An important problem with gradient- gradient-based optimization in GANs is that the vector-field generated by the losses of the discriminator and generator is not gradient vector field. It follows that gradient descent is not guaranteed to find local optimum and can cycle, which can slow down convergence lead to phenomena like mode collapse and mode hopping. Balduzzi et al. (2018): Gemp &
4

---

Under review as a conference paper at ICLR 2020
Mahadevan (2018) proposed Symplectic Gradient Adjustment (SGA) improve the dynamics of
gradient- basee methods in adversarial games. For a game with gradient (eq. 3), define the Hessian
as the second order derivatives with respect to the parameters, H Vog. SGA uses the adjusted gradient
g - g - where a positive constant
(5)
and - H²) is the anti-symmetrie component of the Hessian. Applying SGA to GANs yields the adjusted updates (see Appendix B.1 details):
[0(()) D0D
O² GgG D0p/
0f(z)
0f(=) DBG
8)(2) b0 80g
0f(z)
g-
+
+
00g
D0D
Compared with in cq. the adjusted gradient g" has second-order terms reflecting the interactions
between and G. SGA has been shown to significantly improve GAN training basic examples
(Balduzzi et al. 2018), allowing faster and more robust convergence to stable fixed points (local
Nash equilibria). Unfortunately, SGA is expensive to scale because computing the second-order derivatives with respect to all parameters is expensive.
Explicitly computing the gradients for the discriminator and generator at z' after one step of latent optimisation (eq. 4 obtains
dLD dLG' T d0, d0c
05(=')
0A=) T
0f(z')
0f(z)
(AAz)
T
0f(=')
+
(7)
00D
00p
dA:
00G
00g
0A:
T
0f(2')
(02((z)
0f(=')
0f(²')
(02((2)
0f(=')
+
80D
0z00p
Oz'
00g
Oz00g
Oz'
(8)
In both equations, the first terms represent how f(=') depends on the parameters directly and the second terms represent how fzz') depends on the parameters via the optimised latent source. For
the second equality, we substitute A = a 0z as the gradient-based update of 2 and use -A=
#f(a)
af(=')
af(=')
02 The original GAN's gradient (eq. 3 does not include any second-order term, since Az
without latent optimisation. In LOGAN, these extra terms are computed by automatic differentiation when back-propagating through the latent optimisation process (see Algorithm I
The SGA updates in eq. 6 and the LOGAN updates in eq. g are strikingly similar, suggesting that the latent step used by LOGAN reduces the negative effects of cycling by introducing a symplectic gradient adjustment into the optimization procedure. The role of the latent step can be formalized in terms of third player, whose goal is to help the generator, see appendix B for details.
Crucially, latent optimisation approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second order terms involving parameters of both the discriminator and the generator which are extremely expensive to compute are not used. For latent 2's with dimensions typically used in GANs (e.g., 128-256 orders magnitude less than the number of parameters), these can be computed efficiently. In short, latent optimisation efficiently couples the gradients of the discriminator and generator, as prescribed by SGA, but using the much lower-dimensional latent source z which makes the adjustment scalable.
An important consequence of reducing the rotational aspect of GAN dynamics is that it is possible to use larger step sizes during training which suggests using stronger optimisers to fully take advantage of latent optimisation. Latent optimisation improve GAN training dynamics further by allowing larger single steps A towards the direction of df()) without overshooting.
8s
3.2 UNROLLING AND STOCHASTIC APPROXIMATION
Appendix B further explains how LOGAN relates to unrolled GANs (Metz et al. 2016) and stochas-
tie approximation. Our main finding is that latent optimisation accelerates the speed of updating D relative to that of G, facilitating convergence according to Heusel et al. (201)) (see also Figure 4b. b).
In particular, the generator requires less update compared with D to achieve the same reduction of loss, because latent optimisation "helps" G.
5

---

Under review as a conference paper at ICLR 2020
1.0
0.8
1.1
|| |A
0.6
scale
1.0
0.4
0.9
0.2
LOGAN (NKID)
LOGAN (GD) BigGanN Deep
0.0
10 100
200 300
400 500 600 700
|[l||
steps 10))
(a)
(b)
Figure 4: (a) Scaling of gradients in natural gradient descent. We use 3 = 5 BigGAN-Deep experiments. (b) The update speed of the discriminator relative the generator shown the dif- ference ||4 - | after each update step. Lines are smoothed with moving average using window size 20 (in total, there are 3007, 1659 and 1768 data points for each curve). For all curves oscillation strongly after training collapsed.
LOGAN WITH NATURAL GRADIENT DESCENT
Although our analysis suggests using strong optimisers for optimising 2, Wu et al. (2019) only used basic gradient descent (GD) with fixed step-size. This choice limits the size Az can take: in order not to overshoot when the curvature is large, the step size would too conservative when the curvature is small. We hypothesis that GD is more detrimental for larger models, which have complex loss surfaces with highly varying curvatures. Consistent with this hypothesis, we observed only marginal improvement over the baseline using GD (section 5.3 Table
Figure 3).
In this work, we instead use natural gradient descent (NGD, Amari 1998) for latent optimisation. NGD can be Martens 2014), and has been applied successfully many domains. By using the positive semi- definite (PSD) Gauss-Newtar matrix to approximate the (possibly negative definite) Hessian, NGD often works even better than exact second-order methods. NGD is expensive high dimensional pa- rameter spaces, even with approximations (Martens) 2014). However, we demonstrate it is efficient for latent optimisation, even in very large models.
approximate second-order optimisation method (Pascanu & Bengio 2013)
af(=)
NGD computes the update as Given the gradient of 2, dz
=== ===0a a F-1
(9)
where the Fisher information matrix F is defined
Ep(]= Lnn((t|2) V Vin((t)2)
(10)
The log-likelihood function p(((())) typically corresponds commonly used functions such as cross entropy loss. This correspondence is not necessary when NGD is interpreted as an ap- proximate second-order method, as has long been used practice (Martens 2014). Nevertheless, Appendix
C provides a Poisson log-likelihood interpretation for the hinge loss commonly used in GANs (Lim Ye 2017, Tran al., 2017). An important difference between latent optimisation and commonly seen senarios using NGD is that the expectation over the condition (2) is absent.
Since each = only responsible for generating one image, it only minimises the loss LG(z) for this particular instance. Computing per-sample Fisher this way is necessary to approximate SGA (see Appendix B.I for details).
More specifically, use the empirical Fisher F² with Tikhonov damping, as in TONGA (Roux ct al. 2008)
F' F g.g.g" 8T +1
(11)
F' is cheaper to compute compared with the full Fisher. since is already available. The damping factor 3 regularises the step size, which is important when Fr only poorly approximates the Hessian

---

Under review as a conference paper at ICLR 2020
60
200
Dotput Euclzdoan
atop_gradlent G
move between steps
50-
150
and
None
40
30
FID
100-
20
50-
10-
0
0- 0
100
200
300 400 500 600 700
800
50
100
150
200
steps (x10")
steps (x10')
(a)
(b)
Figure 5: (a) The change from A across training, in D's output space and ='s Euclidean space. The distances normalised by their standard derivations computed from moving window of size 20 (1007 data points in total). (b) Training curves from models with different "stop gradient" operations. For reference, the training curve from unablated model is plotted as the dashed line. All instances with stop.gro .rrdd ent collapsed (FID went up) early in training.
or when the Hessian changes too much across the step. Using the Sherman-Morrion formula, the NGD update can be simplified into the following closed form:
===a (+
99"
(1
19
52 g
9
3+ 11l1 3
(12)
which does not involve any matrix inversion. Thus, NGD adapts the step size according to the Figure 4 illustrates the scaling for different values of 3. NGD automatically smooths the scale of updates by down-scaling the gradients their norm grows, which also contributes to the smoothed norms of updates (Figure b). Since the NGD update remains proportional to g, our analysis based on gradient descent in section still holds.
1 curvature estimate c 1/2 "+
EXPERIMENTS AND ANALYSIS
We focus large scale GANs based BigGAN-deep (Brock et al. 2018 trained on 128 128 size images from the ImageNet dataset (Deng et al.) 2009). In Appendix we present results from applying algorithm on Spectral Normalised GANs trained with CIFAR dataset (Krizhevsky al., 2009), which obtains sfae--o-thhe-e scores on this model.
5.1 MODEL CONFIGURATION
used the standard BigGAN-ddepp architecture with three minor modifications: We increased the size of the latent source from 128 to 256, to compensate the randomness of the source lost when optimising z. 2. We use the uniform distribution 2d( (),1), instead of the standard normal distribution (00 1) for p(z), to be consistent with the clipping operation (Algorithm 11. 3. We use
leaky ReLU instead of ReLU as the non-linearity for smoother gradient flow for d2
Consistent with detailed findings in Brock et al. (2018) that these changes have limited effect, our experiment with this baseline model obtains only slightly better scores compared with those Brock et al. (2018) (Table see also Figure 8). The FID and IS computed as in Brock et al. (2018) 8), and IS values are computed from checkpoints with the lowest FIDs. The means and standard deviations computed from models with different random seeds.
To apply latent optimisation, we use damping factor 3 = 5.0 combined with a large step size of = 0.9. As additional way of damping, only optimise 50% of z's dimensions. Optimising the entire population of z was unstable our experiments. Similar to Wu et al. (2019), we found was helpful regularise the Fuclidean norm of weight-change Az, with regulariser weight of

---

Under review as a conference paper at ICLR 2020
300.0. All other hyper- parameters, including learning rates and large batch size of 2048, remain the same as in BigGAN-deep; we did not optimise these hyper-parameters. We call this model LOGAN (NGD).
5.2 BASIC RESULTS
Employing the same architecture and number of parameters as the BigGAN- baseline, LOGAN (NGD) achieved better FID and IS [T. As observed by Brock et al. (208), BigGAN training always eventually collapsed. Training with LOGAN also collapsed, perhaps due to higher-order dynamics beyond the scope have analysed, but took significantly longer (600k steps versus 300k steps with BigGAN-deep).
During training, LOGAN was 2 3 times slower per step compared with BigGAN-deep because of the additional forward and backward pass. We found that optimising = during evaluation did not improve sample scores (even up to 10 steps), we do not optimise for evaluation. There- fore, LOGAN has the same evaluation cost as original BigGAN-deep. To help understand this behaviour, we plot the change from A during training in Figure 5 Although the movement in Euclidean space |Az|| grew until training collapsed, the movement in D's output space, measured as [I + A=) - [=)]],, remained unchanged (see Appendix D for details). As shown in our anal- ysis. optimising z improves the training dynamics, so LOGANs work well after training without requiring latent optimisation.
5.3 ABLATION STUDIES
We verify our theoretical analysis in section 3 by examining key components of Algorithm via ablation studies. First, experimented with using basic GD optimising 2, in Wu et al.) (2019), and call this model LOGAN (GD). A smaller step size of a = 0.0001 was required; larger values were unstable and led to premature collapse of training. As shown in Table I the scores from LOGAN (GD) were worse than LOGAN (NGD) and similar to the baseline model.
We then evaluate the effects of removing those terms depending on 0f(+) in eq. 8 which are not in dz
the ordinary gradient (eq. (3). Since these terms were computed when back-propagating through the latent optimisation procedure, we removed them by selectively blocking back-propagation
with "stop_gradient" operations (e.g., in TensorFlow Abadi et al. 2016).
Figure b shows
the change of FIDs for the three models corresponding to removing
0A:
af(2') 0z
removing
306
and removing both terms. As predicted by our analysis (section stabilise training; training diverged early for all three ablations.
0A2 B0
T
af(=')
both terms help
5.4 TRUNCATION AND SAMPLES
Truncation a technique introduced by Broek et al. (2018) to illustrate the trade-off between the FID and IS in trained model. For a model trained with au p(2) from a source distribution symmetric around 0, such as the standard normal distribution NNO0, 1) and the uniform distribution u(-1,1), down-scaling (truncating) the source ह = s with 0
gives samples with higher visual quality but reduced diversity. This observation is quantified as higher IS and lower FID when evaluating samples from truncated distributions.
Figure 3 b plots the truncation curves for the baseline BigGAN-deep model, LOGAN (GD) and LOGAN (NGD), obtained by varying the truncation (value of s) from 1.0 (no truncation, upper-left ends of the curves) to 0.02 (extreme truncation, bottom-right ends). Each curve shows the trade-off between FID and IS for an individual model; curves towards the upper-right corner indicate better overall sample quality. The relative positions of curves in Figure 3
(b) shows LOGAN (NGD) has the best sample quality. Interestingly, although LOGAN (GD) and the baseline model have similar scores without truncation (upper-left ends of the curves, see also Table LOGAN (GD) was better behaved with increasing truncation, suggesting LOGAN (GD) still converged to a better equilibrium. For further reference, we plot truncation curves from additional baseline models in Figure
8

---

Under review as a conference paper at ICLR 2020
Figure and Figure show samples from chosen points the truncation curves. In the high IS domain, C and D on the truncation curves both have similarly high of near 260. Samples from batches with such high IS have almost photo-realistic image quality. Figure I show that while the haseline model produced nearly uniform samples, LOGAN (NGD) could still generate highly diverse samples. On the other hand, A and B from Figure 3 b have similarly low FID of near 5, indicating high sample diversity. Samples in Figure 2 show higher quality compared with those in (e.g., the interfaces between the elephants and ground, the contours around the pandas).
CONCLUSION
In this work present the LOGAN model which significantly improves the state-of-the- art large scale GAN training for image generation by online optimising the latent source z. Our results illus- trate improvements in quantitative evaluation and samples with higher quality and diversity. More- over, our analysis suggests that LOGAN fundamentally improves adversarial training dynamics. We therefore expect our method be useful in other tasks that involve adversarial training, including representation learning and inference (Donahue et al. 2017,/umooo
et al. (2017). text generation (Zhang al. 2019), style learning et. 2017 Karras et 2019), audio generation (Donahue et al. 2018) and video generation (Vondrick et al. 2016: Clark et al. 2019).
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In /2th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-
276. 1998.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: 170100777 2017.
Wasserstein GAN.
arXiv preprint
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets. In Proceedings of the 34th International Conference Machine Learning-Vani Volume 70, pp. 224-232. JMLR. org, 2017.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Grae- pel. The mechanics of n-player differentiable games. arXiv preprint
2018.
Vivek S Borkar. Stochastic approximation with two time scales. Systems & Control Letters, 29(5): 291-294, 1997.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint 1rXiv:1809.11000 2018.
Emmanuel J Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute Mathematical Sciences, 59(8):1277-122) 2006.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation complex datasets. arXiv preprint arXiv: 2019.
J. Deng. W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPR09, 2009.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. arXiv preprint 2018
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4): 1306, 2006.

---

Under review as a conference paper at ICLR 2020
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Advesarially learned inference. In İCLR, 2017.
Ian Gemp and Sridhar Mahadevan. Global Convergence to the Equilibrium of GANs using Varia- tional Inequalities. In Arxiv:8808.0153:3 2018
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Morris W Hirsch. Convergent activation dynamics in continuous time networks. Neural networks, 2(5):331-349, 1989.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Vijaymohan R Konda and Vivek Borkar. Actor-critic-type learning algorithms for markov deci- sion processes. SIAM Journal on control and Optimization, 38(1):94-123 1999.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alistair Letcher, David Balduzzi, Sébastien Racanière, James Martens, Jakob N Foerster, Karl Tuyls, and Thore Graepel. Differentiable game mechanics. Journal of Machine Learning Research, 20 (84):1-40, 2019.
Jac Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv preprint arXiv: arXiv: 725.02894, 2017.
James Martens. New insights and perspectives on the natural gradient method. arXiv:1412.1193, 2014.
arXiv preprint
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. CoRR, abs/1611.02166 2016. URL
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. arXiv preprint 18X2v:1002.0566 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: arXiv:/802.00 htt::/802.0557. 2018.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural sam-
plers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint 1rXi::1301.3584, 2013.
Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 15N1::5511.044::: 2015.
Nicolas L Roux, Pierre-Antoina Manzagol, and Yoshua Bengio. Topmoumoute online natural gra- dient algorithm. In Advances in neural information processing systems, pp. 849-856, 2008.
10

---

Under review as a conference paper at ICLR 2020
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, 2016.
pp.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. ICLR, 2014.
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood-fii variational inference. In Advances in Neural Information Processing Systems, pp. 5523-5533, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, pp. 613 (21.,21,22 2016.
Yan Wu, Mihacla Rosca, and Timothy Lillicrap. Deep compressed sensing. In International Con- ference on Machine Learning, pp. 6850-6860, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learming, pp. 7354-7363, 2019.
Jun Yan Zhu, Tacsung Park, Phillip Isola, and Alexei A Efros. Unpaired taaage-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A ADDITIONAL SAMPLES AND RESULTS
Figure and 7 provide additional samples, organised similarly as in Figure additional truncation curves.
and Figure shows
B DETAILED ANALYSIS OF LATENT OPTIMISATION
In this section we present three complementary analyses IOOANN, particular, we show how the algorithm brings together ideas from symplectic gradient adjustment, unrolled GANs and stochastic approximation with two time scales.
APPROXIMATE SYMPLECTIC GRADIENT ADJUSTMENT
To analyse LOGAN differentiable game treat the latent step Az adding third player to the original game played by the discriminator and generator. The third player's parameter, Az, optimised online for each p(z). Together the three players (latent player, discriminator, and generator) have losses averaged over a batch of samples:
(13)
where 7 = N (N is the batch size) reflects the fact that each As is only optimised for a single sample so its contribution to the total loss across batch is small compared with 0p and Og which are directly optimised for batch losses. This choice of rj is essential for the following derivation, and has important practical implication. It means that the per-sample loss Lo(z'), instead of the loss summed over batch Σm= Lo(=(,), should be the only loss function guiding latent optimisation. Therefore, when using natural gradient descent (Section 4. the Fisher information matrix should only computed using the current sample z.
The resulting simultaneous gradient is
g aLo(s') aLD(=') oLo(z') T
0s(=)) 0f(=') T
8As
80p
D9G
= -n dAz
af(=)
00p
80G
(14)
Following Balduzzi et al. (2018), we can write the Hessian of the game as:
o² f('')
823(2)) AA:00D 3 Ff((")
f(2') DA200a
TI
BA2
-1I
8 81(())
8 f(22
H
(15)
36
79 80g
8 ffz2)
82 F(2'))
8(2'
80g0Az
80G00D
062
11

---

Under review as a conference paper at ICLR 2020
The presence of non-zero anti-symmetric component in the Hessian
1+4 f'e('))
1 1- 8 (xx))
A A= 1 2 1²)= 1+ 8²(())
8A200D
BAzitle (²
(16)
88 BA
a0p d6c
1- 8 f(x')
8 f''('))
80c00p
implies the dynamics have a rotational component which can cause cycling or slow down conver- gence. Since 77 < 1 for typical batch sizes (e.g.. for DCGAN and 2048 for BigGAN-deep), abbreviate
1- to simplify notations.
Symplectic gradient adjustment (SGA) counteracts the rotational force by adding adjustment term the gradient to obtain g" + g AA which for the discriminator and generator has the form:
0f(2')
8f((2))
0f(z')
0f(z')
DD
+
+1
(17)
00D
0h:00p
OAz
30g
00
80G
0f(=')
8 O1((2)
0f(z))
0f(²')
0f(z')
96
Xy
(18)
00G
0A:00G
dZz
a 00c
a0D
The gradient with respect to A is ignored since the convergence of training only depends on Op and 8G.
If we drop the last terms in eq and 18
which are expensive to compute for large models with
high-dimensional @D and Cg. and use
0f(=")
the adjusted updates can be rewritten as
0z'
0f(z') xy
(8
0f(z')
ID
(19)
00D
02'00p
O2
0f(z))
(0²((=))
05(²')
96
Xy
(20)
00g
dr00G
Oz'
Because of the third player, there are still the terms depend on 0f(z') ciently computing 8² f²') and 0² f(=') non-trivial (e.g.,
to adjust the gradients. Eff-
Pearlmutter 1994.. However, if intro-
0z'00p
duce the local approximation
8 O(
8 {²)(=)
8 8 8²
(21)
0z000p
Dz00p
0z000p dz00I
then the adjusted gradient becomes identical to from latent optimisation.
In other words, automatic differentiation by commonly used machine learning packages can com- pute the adjusted gradient for Op and fG when propaaaaggg through the latent optimisation process. Despite the approximation involved in this analysis, both our experiments in section and the results from Wu et al. (2019) verified that latent optimisation can significantly improve GAN training.
B.2 RELATION WITH UNROLLED GANS
Latent optimisation can be seen unrolling GANs Meetz et al. 2016) in the space of the latent, rather than the parameters. Unrolling in the latent space has the ad advantages that:
1. LOGAN is more scalable than Unrolled GANs because it avoids second-order derivatives over a potentially very large number of parameters.
2. While unrolling the update of D only affects the parameters of G (as in Metz et al. 2016 latent optimisation effects both D and G as shown in eq. g
We next formally present this connection by showing that SGA can be seen approximating Un- rolled GANs (Metz et al. 2016). For the update 0', = Op + AOD, we have the Taylor expansion
approximation at 0p:
(ff(2: (ff(=(D..a)) T
f(z:0p + A0p,0a) f(z:0p.Oa)
A@D
(22)
d0D
12

---

Under review as a conference paper at ICLR 2020
Substitute AdD -a
(=f(2)0000)) p0p
and take the derivatives with respect to fg on both sides:
0f(2;0D AdD. 0G)
0f(z;0p.0c) 2a
O ((; p0
T (f.2
(23)
bbG
d0a
d0p
which is the same as eq. unroll:
18
(taking the negative sign). Compared with the exact gradient from the
0f(2;0p 0D Adp. 8G)
0f(z:0p..cc 0G)
8 ((2;;p, 0G)
T 0f(z;0p.00c 06)
2a
(24)
00g
00c
p0D00G
a(0))
The approximation in eq.
23 comes from using
0f(=)p.00a
0f(2;0) 0c)
and 0f(z:0p.0)
30p
00
00c
(f(= result of the linear approximation.
86c
At this point, unrolling D update only affects 0p. Although it is expensive to unroll both D and G. in principle, we can unroll G update and compute the gradient of On similarly using Alg = @f(=;9)ba))
86g
0f(z; 1D. Alc) 0f(z;0p,0c) + 2a 00p
(8 (0.(()0).00)
T 0f(z;0p.0c) 80G
(25)
D0p
00g00n
which gives the update rule SGA (eq. 17). This correspondence based first order
Taylor expansion is unsurprising, as SGA is based on linearising the adversarial dynamics (Balduzzi
al. 2018
B.3 STOCHASTIC APPROXIMATION WITH TWO TIME SCALES
Heusel et al. 2017) used the theory of stochastic approximation to analyse GAN training. View- ing the training process as stochastic approximation with two time scales (Borkar) 1997 Konda & Borkar 1999). they suggest that the update of D should be fast enough compared with that of G.
Under mild assumptions, Heusel et al. (2017) proved that such two time-scale update converges to local Nash equilibrium. Their analysis follows the idea of (r, 8) perturbation (Hirsch 1989), where the slow updates (G) is interpreted as a small perturbation over the ODE describing the fast update (D). Importantly, the size of perturbation 5 is measured in the magnitude of parameter change, which is affected by both the learning rate and gradients.
Here show that LOGAN accelerates discriminator updates and slows down generator updates, thus helping the convergence of discriminator according to Heusel et al. (2017). We start from analysing the change of 0g. We assume that, without LO, takes Alg 0g Og make a small constant amount of reduction in loss Lo:
P = f2; 0. AlG) + (2; 9c)
(26)
Now using the optimised z' Ac, we assess the change 80g required achieve the same amount of reduction:
P = -=f(2z+ Az; 9G + 50c) f(z; (2:00,,00
(27)
Intuitively, when "helps" 0G to achieve the same goal of increasing f2 0G) by p. the respon- sible of dg becomes smaller, it does not need change much as Adg. thus ||80Gl| < ||Af|]]
Formally, ((2;0p, 0a) and f(z + A; 0g + +80g) have the following Taylor expansions around and fc:
f(2;0d, PG + 80G) = f1 0f(2;0p.0G)
T
Alc + c(AfG)
(28)
00G
0G)
T
f(z Az;0s, 06 + 80c) f[;;
Oz
[0[2++ Az; 0n. Az;0n, 0a)
Az +
50G (A2,80c)
(29)
13

---

Under review as a conference paper at ICLR 2020
Where e(-)'s are higher order terms of the increments. Using the assumption of eq.
26 and 27
can combine eq. 28 and 29
05 0s(=;0p;0c) 00G
T
(0f(z; (01(::00:00) A
T
(Of(2 + Az; 0c))
T
DdG
80G + (30)
Oz
d0g
where c c(2z, 80c)) c(A0g). Since A x
0f(=:0,000) 0s
in gradient descent (eq. 3
0f
A > 0
(31)
0z
Therefore, we have the inequality
Of(2;0p..0 0c)
T
(ffl=+ A2; 0G)
Alc
80G te
(32)
00G
00c
we further assume learning rate,
Agg and 80g are obtained from stochastic gradient descent with identical
0f(z;0p,
0f(z;0p.0c
Adc
80c
(33)
aAG
a0g
substituting eq.
33 into eq.
32 gives
| < ||80gl| +
(34)
The same analysis applies to the discriminator. The similar intuition is that takes the discriminator additional effort compensate the exploitation from the optimised z'. We then obtain
(ff(200,8G) D0D
A@D -
(0f(2;0p,6c)
[ff(z + 0z; 0c)
A
50D + (35)
Oz
D0D
However, since the adversarial loss LD
LG., have A0p -a 0f(.:p..aa) and 80p
80D
0f(x:9p.o)) taking the opposite signs of eq 33 For sufficiently small Az, AOG and ò0c, is 80p close zero, ||0p|| ||80>|| under our assumptions of small Az, AOG and 50g.
Importantly, the bigger the product @f(x) Az is, the more robust the inequality to the error from d2
Moreover, bigger step increases the speed gap between updating D and G, further facilitating convergence according to Heusel et al. (2017). Overall, our analysis suggests:
More than one gradient descent step may not be helpful, since As from multiple GD steps may deviate from the direction of @f(z)
8s
2. Large step of A is more helpful in facilitating convergence by widening the gap between D and G updates (Heusel al. 2017).
3. However, the step of As cannot be too large. In addition to the linear approximation used throughout our analysis, the approximate SGA breaks down when [1 is strongly
violated when "overshoot" brings the gradients 0f(=') the opposite sign of (=()) 0z"
C POISSON LIKELIHOOD FROM HINGE LOSS
Here we provide probabilistic interpretation of the hinge loss for the generator, which leads natu- rally to the scenario of family of discrininators. Although this interpretation is not necessary for our current algorithm, may provides useful guidance for incorporating multiple discriminators.
We introduce the label & = for real data and f 0 fake samples. This section shows that the generator hinge loss
La - (G(z))
(36)
be interpreted negative log-like liheelihoodd function:
Le In p(t D; G(=)
(37)
14

---

Under review as a conference paper at ICLR 2020
Here p(t = D.G) is the probability that the generated image G(2) can fool the discriminator D.
The original GAN's discriminator be interpreted outputting Bermuulli distribution p(t; 8c) = 3/g (1 - Be)²=² In this case, if we parameterise Bg = D (G(z)), the generator loss is the negative log- likelihood
In P(t = D, G(z)) = in pnt 1;8c) - In 3g = - Dn (G(2))
(38)
Bernoulli, however, not the only valid choice as the discriminator's output distribution. Instead of sampling "I" or "0", assume that there many identical discriminators that can independently vote to reject an input sample as fake. The number of votes a given interval can be described by Poisson distribution with parameter À with the following PMF:
x
pk:;
(39)
k!
The probability that generated image can fool all the discriminators is the probability of G(z) receiving no vote for rejection
p(k = A)
(40)
Therefore, we have the following negative log-likelihood the generator loss if we parameterise = -D (G(2)): - In 0; D. G(=)) = In = 0; -D
(41)
This interpretation has caveat that when D (G(z)) > 0 the Poisson distribution is not well defined. However, general the discriminator's hinge loss
LD min (0, 1 D(x)) min (0, - D(G(=)))
(42)
pushes D (G(=)) via training.
D DETAILS IN COMPUTING DISTANCES IN FIGURE 5
For temporal sequence 3 #2, Pr (changes of = or f(=) at each training step in this paper), to
normalise its variance while accounting for the non-stationarity. we process it as follows. We first compute the moving average and standard deviation over window of size N:
Fx
(43)
1+N-1
Σ u=!
(Ix
M2)
(44)
N
Then normalise the sequence
Ft
Ft
(45)
The result in Figure 5 is robust to the choice of window size. Our experiments with N from 10 to 50 yielded visually sumilar plots.
E EXPERIMENTS WITH DCGAN AND CIFAR
To test latent optimisation works with models at more moderate scales, we applied on SN-GANs Miyato et al. 2018). Although our experiments this model are less thorough than in the main paper with BigGAN-deep. we hope to provide basic guidelines for researchers interested in applying latent optimisation smaller models.
The experiments follows the same basic setup and hyper-paramen settings as the CS-GAN in Wu al. (201)) There is class conditioning in this model. For NGD, found smaller damping
15

---

Under review as a conference paper at ICLR 2020
factor B 0.1, ||||| regulariser weight of 3.0 (the same as in Wu et al. 2019), combined with optimising 70% of the latent source (instead of 50% for BigGAN-ddep) worked best for SN-GANs.
In addition, found running extra latent optimisation steps benefited evaluation, of latent optimisation in evaluation for results in this section, although the models were still trained with single optimisation step. We reckon that smaller models might not be pover-parametrised" enough to fully amortise the computation from optimising 2, which can then further exploit the architecture in evaluation time. On the other hand, the overhead from running multiple iterations of latent optimisation is relatively small at this scale. We aim to further investigate this difference in future studies.
ten steps
Table shows the FID and IS alongside SN-GAN and CS-CAN which used the same architec- ture. Here we observe similarly significant improvement over the baseline SN-GAN model, with an improvement of 16.8% in IS and 39.6% in FID. Figure FigureUshows shows random samples from these two models. Overall, samples from LOGAN (NGD) have higher contrasts and sharper contours.
Table 2: Comparison of Scores. The first and second columns are reproduced from Miyato et al. (2018) and Wu et al. (2019) respectively. We report the Inception Score (IS, higher is better, Sali- mans et al. 2016) and Fréchet Inception Distance (FID, lower is better, Heusel et al. 2017).
SN-GAN
CS-GAN
LOGAN (NGD)
FID
29.3
23.1 + 0.5
17.7 0.4
IS
7.42 + 0.08
7.80 = 0.05
8.67 = 0.05
16

---

Under review as a conference paper at ICLR 2020
(a)
Figure 6: Samples from BigGAN-deep (a) and LOGAN (b) with the similarly high inception scores. Samples from the two panels were draw from truncations correspond to points C, D in figure respectively. (FID/IS: (a) 27.97/259.4, (b) 19/9999..
17

---

Under review as a conference paper at ICLR 2020
(a)
(b)
Figure 7: Samples from BigGAN-deep (a) and LOGAN (b) with the similarly low FID. Samples from the two panels were draw from truncations correspond to points A, B in figure respectively. (FID/IS: (a) 5.04/126.8, (b) 5.09/217.0)
18

---

Under review as a conference paper at ICLR 2020
18.0
125
17.
S GAN
20.
25.3
41-3
37 DAN
302
LLDAAN 1001
3.3
31.
no 1/2 20 too
Inception
Figure 8: Truncation curves with additional baselines. In addition to the truncation curves reported in Figure b. here we also include the Spectral-Normalised GAN (Miyato al. 2018). Self- Attention GAN (Zhang et al. 2019), original BigGAN and BigGAN-deep as presented in Brock al. 2018
(b)
Figure 9: (a) Samples from SN-GAN. (b) Samples from LOGAN.
19


Reviews:
[{"review_id": "HJx7zgxTFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel training scheme for GANs, which leads to improved scores w.r.t. state-of-the-art. The idea is to update the sampled latent code in a direction improving the inner maximization in the min-max problem. The work considers both, the gradient direction and a direction motivated by the natural gradient which is shown to yield excellent performance. The overall scheme is motivated as an approximation to the (prohibitively expensive) symplectic gradient adjustment method. \n\nWhile the work contains some typos, it was easy to read and overall very well written. The experimental results are impressive and clearly validate the usefulness of the approach. Therefore, I recommend acceptance of the paper.  \n\nOn the theoretical side, I feel some parts can be improved. I'm willing to further increase my rating if some of the following points are addressed:\n\n1. The connection to SGA is a bit hand-wavy, as terms are dropped or approximated with the only reasoning that they are difficult to compute. In some approximations (e.g. Gauss-Newton approximation of the Hessian) one can argue quite well that certain second-order terms can be dropped under some assumptions (e.g. mild nonlinearity, or vanishing near the optimum).\n\n2. I had some troubles Sec 4 related to the natural gradient.\n(a) What is p(t | z)? Is it the same p as in (37), Appendix C? I find the argument that the hinge loss pushing D(G(z)) < 0 during training hand-wavy, as for natural gradient p(t | z) should always be a valid distribution. There are also some typos which made it hard to follow the arguments there. It probably should read \"an ideal generator can perfectly fool the discriminator\" (not perfectly fool the generator). \n\n(b) Maybe it would be easier to directly argue that one wishes to approximate SGA as well as possible, rather than taking a detour through the natural gradient.\n\n3. Why is the increment \\Delta z clipped in Algorithm 1? Is there a theoretical justification? If the goal of the clipping is to stay inside the support of the uniform distribution, shouldn't it rather be z' = [z + \\Delta z]? A soft clipping (e.g. performing a mirror descent step) might give better gradients.\n\nTypos, minor comments (no influence on my rating):\n- In Eq. (5), it should be \\partial^2 in the last \"Hessian-block\" multiplication.\n- presribed -> prescribed\n- Eqs. (7) and (8) might be easier to parse if one uses a different notation to distinguish between total derivative and partial derivative, i.e., write on the left side df(z') / d\\theta. Also, I think it is clearer to write in the last terms in (7) and (8) \\partial f(z') / \\partial \\delta z instead of \\partial f(z') / \\partial z'.\n- In Appendix B.1, shouldn't it be \\gamma=(1+\\eta)/2 instead of  \\gamma=\\eta (1+\\eta)/2?\n- I've found ELU activations to work well in GAN models which involve the Jacobian w.r.t. z. Maybe it can stabilize things here as well.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxDhRyCtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nLOGAN optimizes the sampled latent generative vector z in conjunction with the generator and discriminator. By exploiting second order updates, z is optimized to allow for better training and performance of the generator and discriminator.\n\nPros:\n+ A relatively efficient way of exploiting second order dynamics in GAN training via latent space optimization.\n+ A good set of experiments demonstrating the superior performance of the proposed method on both large and small scale models and datasets.\n\nCons:\n- Lack of code\n\nComments:\nAll in all, this appears to be a solid contribution to the GAN literature, addressing some of the limitations of CS-GAN [1]. The lack of open source code accompanying the paper (in this day and age) does it a serious disservice. I have already tried and failed to replicate the cifar10 results. There appears to be some nuance in implementation that would probably clear up if the authors release their code along with the paper.\n\n[1] - http://proceedings.mlr.press/v97/wu19d.html", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to overcome challenges in GAN training through latent optimization, i.e. updating the latent code, motivated by natural gradients. The authors show improvement over previous methods.  The work is well-motivated, but in my opinion, further experiments and comparisons need to be made before the work can be ready for publication.

The authors write that "Unfortunately, SGA is expensive to scale because computing the second-order derivatives with respect to all parameters is expensive" and further "Crucially, latent optimization approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second-order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used. For latent z’s with dimensions typically used in GANs (e.g., 128–256, orders of magnitude less than the number of parameters), these can be computed efficiently. In short, latent optimization efficiently couples the gradients of the discriminator and generator, as prescribed by SGA, but using the much lower-dimensional latent source z which makes the adjustment scalable."

However, this is not true. Computing the Hessian vector product is not that expensive. In fact, it can be computed at a cost comparable to gradient evaluations using automatic differentiation (Pearlmutter (1994)). In frameworks such as PyTorch, this can be done efficiently using double backpropagation, so only twice the cost.  Based on the above, one of the main claims of improvement over existing methods, which is furthermore not investigated experimentally, is false. 

It is unacceptable that the authors do not compare with SGA: both in terms of quality and computational cost since that is the premise of the paper. The authors also miss recent works that successfully ran methods with Hessian-vector products: https://arxiv.org/abs/1905.12103 https://arxiv.org/abs/1910.05852
Decision: Reject