Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new loss function to improve the performance of neural network models trained by DP-SGD. The new loss function is a weighted average of the sum of squared error, the focal loss, and a penalty on the squared norm of the pre-activation output of different layers. The new loss achieves state-of-the-art accuracy on the CIFAR-10, FashionMNIST, and MNIST datasets. It is also shown that the new loss can reduce the bias of gradient clipping and encourage learning on hard examples.
main_review: Overall I think this is an interesting paper that explores the impact of loss function on the performance of models trained by DP-SGD. Also, the experiments are quite comprehensive uncovering the importance of choosing loss functions. Yet, there are still some aspects that could be improved.

1. Though the loss can improve upon cross-entropy, it introduces three extra hyperparameters to tune. It seems the paper only reported the best performance after hyperparameter searches. However, the sensitivity of performance with respect to these hyperparameters is not reported or discussed. It would be better if such results can be added.

2. The impact of loss function on model performance is an important and interesting topic. The loss function proposed by the paper consists of three components, all from existing literature. I wonder have the authors tried other components (e.g. using hinge loss of Huber loss to replace the sum of squared errors or using other penalties)? 

-----------------------after rebuttal--------------------------------------

My concerns are mostly addressed. As pointed out by other reviewers, some intuitive arguments for motivating the losses are not well-supported and misleading (e.g. using Allen-Zhu et al. 2019 to argue faster convergence in practice mentioned by Reviewer VgSd). However, I believe this paper should be viewed as an empirical paper and I am satisfied with the authors' efforts in exploring the impact of loss function on the performance of DP-SGD. Thus, I would like to keep my score.

summary_of_the_review: Overall I think the paper is interesting and could be further improved if more results are added.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a tailored loss for DP-SGD, which includes summed squared error, the focal loss, and a regularization penalty. The summed squared error is for fast convergence at initial stage. The focal loss is used for identify hard samples. The regularization penalty is used for reducing the gradient/weight norm and avoid explosion. For each component of the tailored loss, the paper has empirical/theoretical evidence to argue the necessity. It is a good try to improve the performance of DP-SGD from amending the loss.
main_review: There are several weak points of this paper.

The paper uses the theoretical evidence to argue that summed squared error is good for fast convergence. We have to admit that the convergence result in Allen-Zhu et al. 2019 has strict conditions: NTK regime, ReLU and square loss, which may not hold in the practical experiments. Even if these results hold, the fastness of exponential convergence is more evident for the final stage of convergence. However, here it is used to argue for the initial stage's fastness, which is not persuasive. In practice, the convergence rate is more related with the choices of the optimizers SGD or Adam, different learning rates. rather than different losses. The paper does not have good comparison with these choices.

The regularization penalty is quite similar to the function of the weight decay. How does it compare with normal weight decay? It is supposed that DP-SGD naïve application with weight decay can achieve better results than 59% on CIFAR10.

Most importantly, CIFAR10 may not be a good dataset for benchmarking DP algorithms. For one obvious reason, it is not a sensitive dataset from every aspects. It is a too small dataset, with each class 5000 samples so that the DP algorithm may not perform well as expected unless using pretrained model. Benchmarking DP algorithms on CIFAR10 may lead to over-optimized models and/or losses that are not be able to generalize to other real privacy-sensitive scenarios, i.e. language models.

summary_of_the_review: Good intuitions about DP algorithm but the results and arguments are not convincing. Overall, the review would like to give a weak reject to this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to improve the accuracy of the DP-SGD training from the perspective of loss function design. The authors propose a loss composed of SSE loss, focal loss, and L2 regularization penalty. Experiments are conducted to demonstrate the effectiveness of the 
proposed loss. 

main_review: Strength: 

1.	Improving the utility of private deep learning from a loss design perspective is novel to me. This may be also valuable to the community. 
2.	The model obtained by optimizing the proposed loss achieves the SOTA result.

Weakness: 

The main disadvantage of this article is that it does not explain clearly the motivation for designing the loss and why the proposed loss can work. Although the author tried to analyze the mechanism of loss in section 3.2, these analyses were not convincing enough and did not clearly explain why the loss can work. Specifically,
1. The author indicated "Our loss limits information loss from clipping" through Fig. 1, but did not explain why the proposed loss can prevent logit values and weights from exploding.
2. The author claimed "Our loss yields faster convergence", but did not provide proof.
3. The author mentioned "Preventing logits from exploding may also help recover the generalization boost of batch normalization", which can only be used as a guess based on observations. A rigorous theoretical analysis may be required to strengthen the claim.
4. It seems that Figure 2 doesn't serve as an indication that the proposed loss function works because it enhances smoothness, because according to Fig 2, the smoothness of the network trained by CE loss is getting better and better, while the smoothness of the proposed loss training model is It's getting worse.

Another question: Are the parameters robust to different hyper-parameters settings in the loss?

summary_of_the_review: None
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new loss that consists of three parts, namely MSE+focal loss+regularization loss. The new loss is supposed to benefit DP deep learning in comparison to the cross-entropy loss. Accuracy improvement has been observed on MNSIT, FashionMNIST and CIFAR10. 
main_review: I think this paper is clear and easy-to-follow. The underlying philosophy that maybe DP learning should not use the same loss function as regular non-DP learning is interesting. This empirical paper has analyzed a set of computer vision datasets and the improvement does exists. I appreciate the authors efforts to do ablation study and to give details of network architectures etc. in the appendix.

However, my main concern is the message that this paper is trying to convey. First of all, the experiments are not comprehensive: only computer vision tasks are included and only on toy datasets. I believe including recommendation system and NLP datasets will make the statement of using their new loss much more convincing. Secondly, the empirical improvement is not significant. The seemly most significant improvement (as is the one presented in the abstract) is 4% on CIFAR10. Given that non-private model can easily get over 90% on CIFAR10, I believe the gap is not really closed by introducing the new loss. Lastly, the lack of theory may make people wonder, does this improvement really hold for general DP training? At most we can say on specific datasets and specific models, using the new loss is beneficial. Also all three components of the new loss already exist in regular training, so the current approach seems a simple combination without sufficient justification. 
summary_of_the_review: The paper is clear and easy-to-follow, combining three existing losses to form a new one that empirically improves DP accuracy on some vision datasets. However, the experiments should include non-vision tasks and provide (at least discuss) the insight from a theoretical viewpoint. Also the improvement is not significant enough for ICLR venue.
review_title: null

Meta-review: The reviewers all seemed to agree that the investigation of other losses is an interesting direction of study, and acknowledged there was some empirical performance improvement for standard computer vision tasks. However, they felt the justification of the specific form of loss was a bit shaky and heuristic, and were furthermore unconvinced by results exclusively for image classification (one reviewer was unmoved by the magnitude of improvement). This was a borderline decision, but we hope the authors refine and resubmit their work as this is an interesting but underexplored direction within DPML. 

As one recent related work which investigates the effect of other architecture differences in the DP setting, the authors may be interested in https://arxiv.org/abs/2110.08557.
Decision: Reject