Kati8/ICLR_2023_2025 huggingface dataset:
authors: Anuradha Kumari
authors: Ritik Mishra
authors: M. Tanveer
reviews: 
Comments: The author claims that the zephyr loss improve stability, prevent overfitting, and makes it robust to outliers. They provide some theorems about the properties of the loss and showing weak experimental results.
Strengths: Novel loss function, could potential help stability (not enough proof for that claim yet). Needs a lot more work to validate it.
Weaknesses: Theorem 2 is framed incorrectly, its approximately similar, not equal. It would be great to instead give lower and upper bounds.

As mentioned in WGAN, https://arxiv.org/abs/1701.07875, the Total Variation is not continuously differentiable with respect to the parameters. Since Zephyr loss is equivalent to estimating the Total Variation distance, this is not ideal.

The experiments alone prevent giving this paper a good score. The generated images are very poor quality, often completely broken and the inception score is extremely low (For CIFAR-10, the authors get an FID of 2.9 when the smallest FID of the WGAN-GP 7 years old paper is 5.34 and they can reach up to 8.59). We are not in 2016, currently they were done with the original setup from DCGAN. The experiments need to be done with modern implementations using Adamw(beta1=0), EMA, and modern methods. The baselines should be taken from their respective papers or re-implemented with modern methods with similar or better FID/IS than original results. 

At this point in time the big problem with GANs is stability when scaling. Making a new loss function is unlikely to solve this problem. The authors needs better experiments and it would be useful to also have DiracGAN (https://github.com/ChristophReich1996/Dirac-GAN) experiments and convergence proof.

IS is not a good metric, this has been mentioned many times in the literature. FID is better, but not perfect. Still, its a better choice.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 2
reviews: 
Comments: To overcome the limitations such as susceptible to outliers, vanishing gradients, and training instability in GAN’s training, this paper introduce zephyr loss—a novel, convex, smooth, and Lipschitz continuous loss function designed to enhance robustness and provide flexible gradient control. 
Leveraging this new loss function, the article propose ZGAN, a refined GAN model that guarantees a unique optimal discriminator and stabilizes the overall training dynamics.
Extensive experiments further demonstrate that ZGAN surpasses leading methods in generative modeling.
Strengths: The article introduces a novel loss function, called zephyr loss, for GANs, which is referred to as ZGAN. This loss function is convex, smooth, and Lipschitz continuous, designed to enhance robustness and provide flexible gradient control.
Weaknesses: 1. The empirical results do not strongly support the claims made in the article. For instance, in Figure 4, the visual results of your ZGAN are unsatisfactory and significantly worse than those of state-of-the-art GAN models.

2. The article makes a mistake: even if the loss function is convex, smooth, and Lipschitz continuous, this does not ensure that the training process of GAN is Lipschitz continuous and dynamically stable. I recommend that the author include more analysis on how zephyr loss can stabilize GAN training and ensure that ZGAN meets Lipschitz continuity.

3. The article does not provide proof of dynamic stability near Nash equilibrium points. I advise the author to add this proof about local stability analysis around equilibrium points.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This work proposes using a smooth approximation to the Huber loss function, termed the 'zephyr loss', as part of GAN training. Authors show in theorem 4 that when the discriminator is optimal at each training step, minimizing the GAN objective is equivalent to minimizing a distance function that is close to the total variation distance. Experiments are provided demonstrating improved inception scores on cifar10, cifar100, STL10, and svhn datasets when compared to WGAN and LSGAN
Strengths: * Presentation of the loss function is clear, and paper is well organized
Weaknesses: * Much of the theory presented in this work seems incremental (especially Theorems 1-3), essentially reproducing the original convergence results of Goodfellow et al 2016 and Nowozin et al 2016 for a specific loss function. Accordingly, there is the same limitation that convergence results assume the discriminator is fully trained at each step, which is not realistic in practice.

* Concerningly, I do not see any references to the Huber loss or f-gan related papers [1,2]. It seems to me that this method is most likely a specific instance of an f-gan, and would benefit from analysis under this framework.

* While the experimental results seem promising, these datasets are fairly simple and I am not convinced that hyperparameter tuning will not be an issue for more complex datasets.


[1] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems 29 (2016).

[2] Kurri, Gowtham R., et al. "α-GAN: Convergence and estimation guarantees." 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022.
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: Motivated by increasing the stability of GAN training, the paper proposes an alternative objective that is close to that of Least-Squares GAN (LSGAN) but where the squared loss is replaced by a soft-L1 loss that resembles the Huber loss. The generator’s objective is also reportedly replaced by a form of expected feature matching (as in Salimans et al. 2016a). Experiments on small image datasets show improvements in inception scores.
Strengths: Originality and significance: To the best of my knowledge this is the first work to propose this type of GAN training objective. Improving stability of GAN training by increasing robustness properties via losses more robust to outliers and with bounded gradients is an interesting idea worth investigating. 

Clarity and Quality: The paper is well written, and the approach is properly related and contrasted with the related GAN literature. Motivations are clearly stated. The algorithm is well described and easy to understand. The paper provides both a theoretical analysis of the algorithm’s property and quantitative and qualitative empirical evaluation.
Weaknesses: The main weakness of the paper is what I believe may be an unsound equation derivation (in Eq 2 and 4) which invalidates the proof that the stated generator’s optimization objective approximately minimizes total loss variation, and thus invalidates the larger part of the theoretical analysis.
Moreover I think that how the generator’s objective is implemented in the code does not even correspond to what is stated in the paper (see question section for details).

The Zephyr-loss looks very close to a Huber loss, this connection is neither referenced nor discussed.

Another weakness is that the paper claims benefits of the approach – compared to its closest parent LSGAN – but without providing evidence that LSGAN training actually suffers from the ills that ZephyrGAN purports to cure, s.a. Gradient (in)stability and (insufficient) robustness to outliers. It would have been more convincing to measure and highlight these failures/difficulties, to show that they are indeed represent in LSGAN and then cured by ZephyrGAN.
One claimed advantage, “Flexibility in tuning with alpha”, clearly isn’t one, since alpha plays the same role as the learning rate (which is present in all other GAN variants).

Lastly, the difference in generation quality cannot really be assessed qualitatively on low resolution images (CIFAR-10, STL-10, SVHN). These experiments are much too low scale and quality for claiming the superiority of a generative modeling approach by nowadays’ standards.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 2