Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies Wasserstein gradient flows for a certain class of objective functionals. These functionals form are an approximation of the f-divergence defined using its variational form. In particular, the optimization domain in the variational formulation is restricted to the class of functions with bounded Lipschitz constants. As the bound on the Lipschitz constant grows, these approximations become exact. The gradient flow is constructed with the objective of generating new samples from a target distribution (similar to the objective in GAN). Another presented idea is to consider generating the gradient flows in the latent space with an encoded-decoder mechanism in order to take advantage of the low-dimensional structure present in the target distribution.   
strength_and_weaknesses: Strength: 
- The idea of considering gradient flows for Lipschitz constrained f-divergences is new to me
- Also, considering the gradient flows in the latent space seems interesting and valuable
- Numerical experiments are complete and informative
- the comparison Table 2 is very nice, though I wished to see more discussion. 

Weakness: 
- Not enough motivation for constraining in terms of Lipschitz functions, since neural networks are not a good representative of all Lipschitz functions with a certain Lipschitz constant. Why not gradient-norm penalty?
- The claim that Lipschitz constant makes the particle system "stable" should be made more precise. Stability in what sense? 
- The infinite-speed of diffusions is true, but diffusion is not the only way to implement Fokker-Plank. It can also be implemented deterministically using \nabla \log (density) which is Lipschitz under standard assumptions on density. 
- Not in depth discussion of the numerical results, conclusions, and limitations.  


   
clarity,_quality,_novelty_and_reproducibility: The paper is very clear, and has high quality. The idea of using restricted variational forms of f-divergences for gradient flow appears in the following paper. But the combination of using the Lipschitz constrained class and latent space is original. 

J. Fan, Q. Zhang, A. Taghvaei, Y. Chen. "Variational Wasserstein gradient flow", ICML 2022
summary_of_the_review: The paper is well written in general and contain original ideas. However, it needs more motivation for considering Lipschitz constrained class. Also, it discusses two very nice and independent ideas in one single paper. Gradient flows in the latent space is independent of Lipschitz constrained gradient flows. And it is not clear that which of these two ideas are responsible for the improved numerical results compared to GAN. 

####
I thank the authors for their response. But I am still not convinced about the motivation for considering Lipschitz constrained since most gradient flows are Lipschitz under mild assumptions on the initial and target density. After reading other reviews I am changing the score to 5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers a new class of f-divergences that incorporate Lipschitz continuous functions in the variational representation of the f-divergence. These new class of divergences interpolate between the 1-Wasserstein metric and f-divergence when the Lipschitz parameter of the function class varies from 0 to infinity. This class also allows one to define a gradient flow and transportation equation from one distribution to another. Using this theory, the authors introduce a scheme to learn a generative model of a data distribution by learning a particle flow in the latent space of an auto encoder. This structure leverages the fact that many datasets of interest have intrinsic low-dimensionality. Experimental results are showcased on small synthetic and toy datasets such as MNIST.
strength_and_weaknesses: **Strengths:**
- The proposed methodology of the paper is mathematically grounded, with some nice results on the approximation side showcasing that the error between the two distributions can be quantified.
- The method appears to perform at its best, and fairly well in low-data regimes.

**Weaknesses:**
- The motivation for the approach could be strengthened. In particular, it is unclear what the benefits are in terms of utilizing this Lipschitz regularized formulation versus the original variational formulation over continuous and bounded functions. More motivation and discussion along these lines would improve the readability of the paper.
- Some experimental results show that the generative capabilities do not improve with more data. For example, the MNIST generation results do not appear to improve with more samples both visually and quantitatively (as given by the FID score).
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is fairly clear given the technical nature of the results. There are some areas that can be improved. In particular, there is a lack of motivation for consideration of Lipschitz regularized functionals over continuous and bounded functions. I can see similarities with W-GAN in this regard, but more discussion along these lines would be nice. Also, Figure 1 comparing the differences between GPA and GAN is difficult to read and is quite unclear.

**Novelty and significance:** To the reviewer’s knowledge, the use of this general Lipschitz regularized class of divergences for generative modeling appears novel. The technical tools, however, appear heavily inspired by Birrell et al., and it is not clear what are the new contributions on the theoretical side.

**Reproducibility:** The empirical results appear reproducible with the code provided by the authors. I have not run the code myself, however.

**General comments:**
- It is curious that the sample generations do not appear to improve with more examples. In fact, in the MNIST example, the FID score is slightly worse for 2000 samples versus 200 samples. Do the authors have a sense of why this is the case? The theory seems to suggest that having a perfect encoder decoder pairing is important. Is there a connection here to a lack of a perfect encoder decoder pairing? Also, why does it seem to be that the model performs well with few samples? 

summary_of_the_review: Overall, I think that there are interesting mathematical ideas presented in this work, and the experimental results showcase some potential in the applications of such ideas as well. However, there are certain points about the core contributions on the theoretical side and puzzling properties about the model that I would like the authors to comment on before increasing my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a particle flow algorithm that approximates a target empirical distribution with particles. The suggested algorithm updates the particles using the Wasserstein gradient of Lipschitz-regularized $f$-divergences, which is shown to exhibit a variational formulation as a supremum over Lipschitz continuous functions. An additional autoencoder can be used to flow the particles in a latent space. Various experiments are done to demonstrate the performance of the proposed method compared to GANs. 

strength_and_weaknesses: ## Strengths:
* The idea of facilitating Wasserstein gradient flow by optimizing a learnable neural network from which the Wasserstein gradient can be obtained is a very interesting one.
* The proposed algorithm is simple and works for all kinds of $f$-divergences.

## Weaknesses:
* I found the novelty of the present paper limited. To me, the main novelty is (6), although I think the derivation is more or less identical to that of Dupuis & Mao (2022) from the KL case.
* The application prospect of the proposed algorithm is questionable. Without using the autoencoder, to me, the proposed algorithm is just to subsample the target empirical distribution. I think there is a lot to be explored here. For instance, if we use a very small number of particles (compared to the number of samples in the target distribution), can we still represent the target distribution faithfully? If so can we use the proposed method as an alternative to k-means?
* I don't think comparing with GAN is reasonable. GAN is capable of generating endless streams of new samples, whereas for the proposed method you must fix the number of particles ahead of time (it is also unclear to me how many particles were used in the experiments). A good baseline to compare would be, say minimizing the Wasserstein-1 (or W2) distance between particles and the target distribution and comparing the resulting particles. As a result, I do not find the experimental results convincing.

clarity,_quality,_novelty_and_reproducibility: The writing of the paper can be much improved. 

* Quite a lot of sentences are not precise and thus confusing. For example, just above Section 4, "This is in sharp contrast with the Fokker Planck equation". How is the finite speed of propagation related to FP equation?
* Theorem 2.1 (except the 4th point) seems somewhat irrelevant to the rest of the paper
* I found the Data Processing Inequality and the argument on the mobility concept in Section 5 confusing and very handwavy. Aside from applying some change of variable in (13)(14) I don't get what the selling point is in this section.
* The experiments section is very confusing. It feels like the authors try to pack all the numbers and figures in the main text without properly explaining most of them. I think it would be better to focus on two experiments with greater details in the main text and move the rest to the appendix.
* Reproducibility seems good.

summary_of_the_review: Overall I don't think the paper has enough novelty or has justified its application prospect well enough. The writing could also use a lot of improvement. As a result, I'm leaning toward rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper uses Lipschitz regularization (in the context of the dual for f-divergences) to construct gradient flows for sampling from target distributions. These distributions can be empirical (i.e. discrete), and the construction can be applied to build generative models. Experiments show improved performance for generative modeling in low-data regimes.
strength_and_weaknesses: Strengths
-----------
* The method is very nicely developed with a good interplay between f divergences, lipschitz regularization, and gradient flows. By using these three perspectives, it is able to introduce a unified perspective and build a real technical model.
* The presentation is very clear and is able to present the mathematically technical details without it being overly cumbersome. In particular, I appreciated the usage of notation.
* Additionally, the overall mathematical construction does not seem to have any flaws.

Weaknesses
---------------
* The experiments are somewhat weak. First, the experiments only show results for MNIST and Gene Expression (student-t seems more like a proof of concept than an experiment). These datasets are pretty toy (MNIST is rather low dimensional and easy to solve with simpler methods, and the Gene Expression dataset was just introduced to show merging).
* Furthermore, for MNIST in particular, it is very disconcerting that, while the method does perform well in the low-data regime, increasing data from 200->2000 samples doesn't improve the method at all (either visually or numerically). This seems like a potentially big drawback for the purported applications. In particular, if the comparison is on the low data regime only, the authors should compare with other methods that also look at this.
* The MNIST results also should be compared with modern methods. In particular, Wasserstein GANs are around 4 years old, and modern methods such as diffusion.
* There should be a connection with continuous diffusion models (in particular diffusion also has a similar connection with PDEs like the Fokker-Planck Equation) that the authors should elaborate on and include.
* It seems like the method might be pretty computationally expensive (e.g. Table 2 (b)). Could the authors comment more on the training time, as GANs are reasonably fast to train compared with most differential equation methods.
clarity,_quality,_novelty_and_reproducibility: Clarity
-------
The paper is very clear. In particular, notation is tight, statements are supported well, and overall the paper is quite easy to read.

Quality
--------
The paper is of reasonably high quality. In particular, the theoretical developments are quite advanced and are executed well. The experiments may be a bit lackluster, however.

Novelty
---------
The paper is quite novel. It builds on prior work on gradient flows and f-divergences (for GANs) but offers a unique perspective. In particular, it should be noted that the central flow constructions are entirely novel and serve as a helpful bridge between the two (somewhat disconnected) communities.

Reproducibility
-----------------
The authors include a reproducibility statement which describes all relevant information.
summary_of_the_review: Overall, I lean (slightly) to accept the paper. This is mostly due to the theoretical niceness of the paper (including it's many connections with gradient flows, f-divergences, and PDEs). What's stopping me from being fully supportive is the experimental section, which includes relatively mixed results on small toy datasets. However, given the theoretical nature of the work, I do believe that the technical developments outweigh experimental shortcomings.

I would also ask the authors to include a section on diffusion models and the connections therein if possible.

Update
--------
Several of my questions have gone unanswered, which is rather concerning. Furthermore, in the rebuttal, the authors referenced "engineering issues" or "full optimization of algorithms" when talking about toy results (MNIST) or algorithmic running time. Given that there still seems to be a lot of necessary changes to be made, I believe it is better to revise and resubmit the paper and have thus changed my score to leaning reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the Wasserstein gradient flow of some regularised version of f-divergences. These regularizations  are obtained by restricting the variational formulation of f-divergences to functions with bounded Lipschitz constant. The gradient flow allows transporting an initial distribution toward a target by the regularised divergence. The paper then proposes a latent generative model obtained by performing a particle flow in the latent space of an auto-encoder.

After discussion with the reviewers, it appears that the paper would benefit from additional work to address the concerns on novelty and motivation. 

Novelty: It appears that the idea of constraining the Lipschitz constant in the variational formulation of f-divergences is not novel as pointed out by Reviewers Ds4C, VqE2 and VduJ. The response of the authors suggests that one of the main contributions are to consider the Wasserstein gradient flow of these functionals, which is rather limited given that it results from classical constructions of the Wasserstein gradient flows. Therefore, the paper would require major changes to focus more on the third contribution (gradient flow in latent space) and provide more context on the known results. 

Motivation: the motivation for the proposed approach is limited, as pointed out by Reviewers VduJ in response to the authors. This could be improved by considering concrete applications and comparing the proposed approach with some known baselines (see the reviewer's comments). 



justification_for_why_not_higher_score: Lack of motivation and novelty of the proposed method.
Decision: Reject