Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel architecture and training process for learning differentially private synthetic data generators. 

Rather than taking an explicit adversarial GAN approach, the paper looks at minimizing a regularized Wasserstein distance (called the Sinkhorn loss) between the empirical distribution of the data and the synthetic distribution. 

Their approach brings two benefits: first, the Sinkhorn loss is more straightforward to optimize than traditional GANs. The authors don’t really discuss why this is, but my understanding is that the “adversary” is contained in the dual formulation of Wasserstein distance (as the minimum over Lipschitz functions of the difference in expectation between two distributions). Considering such a Lipschitz adversary simplifies optimization. 

Second, the paper uses an idea first published by Chen et al (NeurIPS 2020) but described as independent work here: Instead of measuring the gradient of the full parametric distribution of the generator, the authors measure the gradient “at the generated image level”. This gives a lower-dimensional gradient (requiring less noise). 

Overall, the paper achieves a moderate improvement over the work of Chen et al with respect to a few simple measures of accuracy (namely, how well two DNN models do at classifying real data when they are trained on synthetic data). It does considerably worse than Chen et al’s method with respect to the FID score, a measure of visual similarity. 

I generally like the idea of exploring alternate training strategies. Nevertheless, I have several reservations about the paper: 

1. The relationship to Chen et al. It seems like much of the gain relative to previous work comes from the way gradients are compressed. But this idea appears already in the NeurIPS paper of Chen et al. It is unclear to me how to handle the relative priority of the papers. 
2. Overly simplistic accuracy measures: Wasserstein (or Sinkhorn) distance is a natural loss function, but it is only likely to be a really good measure of accuracy when it is very small. The DP literature, since the pioneering work of Blum, Ligett, and Roth, has focused on using as a loss function the minimum over a (potentially enormous but) task-specific set of queries as the ultimate loss. We understand as a field that general-purpose synthetic data isn’t really possible (it preserves “too many statistics, too accurately” to avoid attacks). How can the framework here be adapted to such accuracy measures? How well does DP Sinkhorn do when measured against such task-specific accuracy? Can DP Sinkhorn be applied to anything except images (and if so, how well does it do)?

Overall, I find the paper interesting but perhaps borderline. 

review_title: new loss function for DP synethic image data

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a method for training OT GANs using differentially private sinkhorn algorithm. The idea is very simple - train GANs with sinkhorn divergences and add Gaussian noise to the gradient of output wrt generated samples. So, the novelty by itself is minimal as sinkhorn GANs have previously been proposed. The method merely adds Gaussian noise to gradients. The DP analysis is also not very different.

I am generally ok with incremental improvements in method if experimental results are strong. This doesn't seem to be the case in this paper. Two datasets are considered - MNIST and CelebA. In MNIST, we observe that the algorithm performs poorly compared to GS-WGAN both in inception score and FID. The method gets better accuracy though. This suggests that the method does not produce diverse samples. For example, consider a case of a mode collapse where a GAN generates only one sample per class. In this case, FID scores will be poor. However, if the generated sample correcly corresponds to the class, classification score will be high. My guess is similar thing is happening here. This is evident even in Fig 2 where the method has much poor sample diversity compared to GS-WGAN. Hence, the model itself is not that great.

In CelebA, despite using BigGAN style architecture, there is significant blur. The real challenge in differential private GANs is to generate samples with good quality while still satisfying privacy constraints.  This doesnt seem to be happening here. Also, comparison with GS-WGAN or other differentially private GANs are missing in CelebA experiment.

For conditional generation, is concatenation of data and labels in cost function a good strategy? It looks like both data and label space is very different, and I am wondering if such type of concatenation might be weak. Can you comment on this.


review_title: Method not so novel, experiments not so convincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a differentially private method for training a generative model. The proposed method takes advantage of the Sinkhorn divergence to achieve robustness against the hyperparameters' choices. The authors also introduce a cost function enabling the generative model to generate images associated with a specific class label. The experimental results show that the proposed method outperforms the existing methods for learning a generative model in a differentially private manner. Furthermore, such a high accuracy can be achieved without the use of publicity available data.

The strong points of this paper are as follows:
-  The authors bring the Sinkhorn divergence-based learning of a generative model into the differentially private generative model learning problem.

The weak points of this paper are as follows:
- The proposed method is not clearly explained and thus is suspicious in the guarantee of differential privacy.
- The presented method is a direct application of Wang et al.'s moment account technique with Zhu et al.'s Poisson sampling. The originality is considerably low.
- This paper has no theoretical and experimental analysis about robustness against a hyperparameter choice, while the authors claim it as a contribution.
- The proposed approach is not well motivated. We can use some differentially private classification algorithm if the objective is high accuracy in downstream classification.

I recommend rejection of this paper because the proposed algorithm has low originality and is not well motivated. Also, the unclarity of the privacy guarantee is problematic.

The authors do not give a clear explanation of the proposed method. In particular, it is unclear if the proposed algorithm guarantees differential privacy. I guess the authors employ either the composition theorem or moment account technique to prove the algorithm's differential privacy; however, there is no privacy proof of the proposed algorithm. I could not confirm that the proposed method ensures differential privacy.

The proposed method is a straightforward application of the techniques from Wang et al. and Zhu et al. Also, defining the cost function as in Eq. 4 is a straightforward way to combine the multidimensional real-valued data and discrete label. I cannot find any original idea, except introducing the Sinkhorn divergence, in the proposed method.

The authors claim that the proposed method is robust against the choice of its hyperparameters. However, there is no evidence to support the claim. A theoretical or experimental analysis of robustness is necessary to claim it.

Why don't we employ the differentially private classification algorithm, such as M. Adabi et al. Deep Learning with Differential Privacy. In CCS'16. When the objective is high accuracy in the downstream classification, we can utilize such an algorithm directly. What is the benefit of employing the generative model-based privacy preservation? The differentially private classification algorithm can achieve high classification accuracy; for example, in Adabi et al.'s paper, the classification accuracy for MNIST with eps=10 is 97%; this value is significantly higher than that of the proposed method.


### Minor comments

- What is the definition of $\hat{S}$? 
review_title: Proposed method is not well motivated and is not novel

Meta-review: The paper proposes a DP method for generative modelling based on optimal transport. The reviewers agree that the novelty is limited in relation to prior work, while the results are not especially compelling either. So, even though this is a valid approach, correctness is not sufficient for acceptance at ICLR. 

Decision: Reject