Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a simple modification of $\ell_\infty$ net training, which boosts the accuracy for certified robustness under $\ell_\infty$ attack. It provides a trainable scale on the output of the network and uses a clipped hinge loss. The paper also proves the expressive ability of $\ell_\infty$ nets for classification problems.
main_review: Major comments:
- The main focus of the paper is on the training of $\ell_\infty$ net. The authors emphasize that the original training is hindered by the  $\ell_p$ relaxation, and they use clipped hinge loss to relieve the issue. The authors also implemented scaled cross-entropy loss. From my point of view, the authors do not explain intuitions very well. Why clipped hinge loss can maintain the Lipschitz property of $\ell_\infty$ even when $\ell_p$ relaxation is used? How should we decompose the effect of scaled cross-entropy and clipped hinge loss?
- The empirical results of this paper seem pretty decent to me. The certified accuracy is able to beat IBP. I wonder how it compares to randomized smoothing?

Minor comments:
- When you search on the hyper-parameters, is any cross-validation performed?
- How do you explain the generalization gap between the certified train and certified test? It seems larger than the previous training scheme.
- How large is $\theta$ in Figure 1(a) and 2(a)?
summary_of_the_review: My current assessment of this paper is slightly below the acceptance threshold because I think the intuitions are not well explained and there are questions that I think the authors should explain to readers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study to improve the certified robustness of $l_\infty$ distance nets by introducing a regularization term to train the network. Guided by the fundamental assumption of "existing a two-layer $l_\infty$ net which has 100% robust accuracy", this method shows better performance over other baselines.
main_review: Strengthens:
1. The paper overall is easy to read and follow.
2. The authors study an important problem of improving the robustness of deep network from the perspective of reegurozation. 


Weakness:
1. Moderate novelty. This paper follows a similar setting with prior work (Zhang et al. 2021) published in ICML 2021, specially targeted in $l_\infty$ distance net for improving certified accuracy. The proposed method serves as a regularization term for the hinge loss to train a more robust network. Apart from that, this work has moderate novelty which shows a large portion of prior work with the same setting and experiment design. 

2. Limited experimental design. To the best of my knowledge, this work shares the same setup with prior work where experiments are not extensive enough. For example, the empirical study is limited to improving the robustness in small datasets such as MNIST. What about other bigger datasets? And also, is this proposed method applicable to other machine learning tasks other than the listed task in the paper?
summary_of_the_review: This paper is largely based on prior work (Zhang et al. (2021))  which follows the same setting and experimental design. The authors propose to prove the effectiveness of $l_\infty$-distance net where they come up with a regularized hinge loss to learn robust classifiers guided by their proposed Theorem 3.2.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new loss to improve the performance of Linf distance network a new network for Linf robustness and achieved impressive empirical results.
main_review: The paper is well-written and the improvement on empirical result is impressive. 
My main concern is on the experiments. 
* The paper proposed multiple modifications on the loss function, including the scale factor on cross-entropy, clipped hinge loss and weight coefficient. The authors should provide comprehensive ablation study on the several modifications.
* I recommend the authors to include sensitive analysis for the hyper-parameter $\lambda$ and hinge threshold $\theta$, which are important if we want to apply this method to different models.
* For reproducibility, since the code is not provided, I recommend the author to provide a detailed instruction on how to reproduce the paper based on https://github.com/zbh2047/L_inf-dist-net.
* After checking the hyper parameters, I found that the authors have tuned the beta_2 and eps of Adam optimizer. Is there any explanation for this tuning?

summary_of_the_review: The paper is well-written and the proposed method has an impressive empirical performance. My concern is mainly on the experiments and reproducibility.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is a follow-up paper of Zhang et al. (2021). In Zhang et al. (2021), the authors proposed a new network architecture, l_infty distance net. By construction, the network is 1-Lipschitz w.r.t. l_infty distance. However, the training procedure therein is problematic. This paper resolves the issue by a new loss design of scaled cross entropy loss + clipped hinge loss. Without using MLP on top of the l_infty distance net backbone, the proposed new training method outperforms the original one in Zhang et al. (2021) and improves over the state-of-the-art by more than 5% for 8/255 and other radiuses. Theoretically, the paper shows the expressive power of l_infty distance net for well-separated data.
main_review: Strength:
1. It is very impressive to achieve 40% certified accuracy for 8/255 attack. The result significantly close the gap between the state-of-the-art of empirical robustness (~53% by TRADES) and that of certified robustness.
2. The paper has a thorough analysis for the training problem in Zhang et al. (2021), and proposes a new method to resolve the issue.
3. The theoretical analysis of expressive power of l_infty distance net is interesting, though it is a little out of the selling point of this paper.

Weakness:
1. The 2/255 experiment in Table 2 does not compare with randomized smoothing. Randomized smoothing is able to achieve 62.6% certified robustness for 2/255 radius (see Table 1 in [1]), which outperforms the proposed method by ~8%.
[1] https://arxiv.org/pdf/2002.03517.pdf

Questions:
1. Is there a particular reason that in Table 2, the 2/255, 8/255 and 16/255 experiments use different l_infty distance nets (they have different clean accuracy)?

2. We know that adversarial training helps in randomized smoothing (see [2]). Does adversarial training helps in improving the certified robustness of l_infty distance net?
[2] https://arxiv.org/pdf/1906.04584.pdf
summary_of_the_review: Overall, I like the paper (in particular, the design of l_infty distance net, which is not the focus of this paper but of Zhang et al. (2021)). The design goes towards solving the robustness issue by new network design, rather than regarding the neural network as a black-box. The experiment result on 8/255 is promising and exciting.
review_title: null

Meta-review: This paper is a follow-up paper of Zhang et al. (2021), that proposed a new network architecture for adversarial robustness, l_\infty distance net. Although the l_\infty network is provably 1-Lipschitz w.r.t. the l_\infty distance, its training procedure exploits the l_p relaxation to overcome the non-smoothness of the model but suffers from an unexpected large Lipschitz constant at the early training stage, an issue to be solved. This paper resolves this issue by a new loss design of scaled cross entropy loss and clipped hinge loss. Without using MLP on top of the l_\infty distance net backbone, the proposed new training method empirically outperforms the original one in Zhang et al. (2021) and improves over the state-of-the-art by more than 5% for 8/255 and other radiuses. Moreover, the paper shows the theoretical expressive power of l_\infty distance net for well-separated data.

There are some concerns about the moderate novelty and reproducibility of the results. Since the empirical results are indeed impressive, the paper could be accepted conditional on that the authors release their reproducible codes to the public.
Decision: Accept (Poster)