Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new formulation for designing 1-Lipschitz neural networks, which is more general than existing ones. To guarantee 1-Lipschitzness, existing works use a variety of parameterizations. This paper proposes a more general parameterization which covers existing ones (existing works can be seen as special cases of this parameterization), and the new parameterization allows more flexibility on designing the network. Additionally, experimental results on CIFAR-10 and CIFAR-100 show that the 1-Lipschitz network trained using the new parameterization performs better compared to baselines.
strength_and_weaknesses: Strengths:

1. The new formulation for representing the 1-Lipschitz layer is novel. The theoretical analysis provide many insights on how to design neural network layers that are one Lipschitz.

2. Empirical results look very promising - the proposed method achieves better provable robustness (one main application of 1-Lipschitz network) than prior works consistently, although in some cases the improvements are marginal.


Weaknesses:

1. One question is not clearly answered by this paper: Is it possible to show that the formulation proposed in this paper can universally approximate any 1-Lipschitz functions? If it is hard to prove theoretically, we hope to know if the new factorization for 1-Lipschitz neural network is more representative compared to previous ones. The paper provides some experimental comparisons, however some baselines use models of different sizes. Ideally, for a fair comparison the number of optimizable parameters should be same in each method.

2. Table 1 should include other recent baselines for provable robustness for L2 norm, such as (Leino et al., 2021) and (Huan et al., 2021); it should be easy to add the results directly. In addition, I think it is important to report training time of each method.

References:

[1] Leino, Klas, Zifan Wang, and Matt Fredrikson. "Globally-robust neural networks." International Conference on Machine Learning. PMLR, 2021.  
[2] Huang, Yujia, et al. "Training certifiably robust neural networks with efficient local lipschitz bounds." Advances in Neural Information Processing Systems 34 (2021): 22745-22757.

clarity,_quality,_novelty_and_reproducibility: The paper is overall well written and has good novelty. The paper provides good insights on how to design 1-Lipschitz neural networks. Sufficient details on experiments are given in appendix.
summary_of_the_review: Training 1-Lipschitz neural network is an important and challenging problem and I like the new formulation in this paper because it provides new insights to this problem. The empirical results also look positive. So I tend to accept this paper. I hope the authors can address the weaknesses questions above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a unified perspective on 1-Lipschitz layers. The 1-Lipschitz layers in previous literature can be recovered by the algebraic conditions in Theorem 1. The theoretical results also lead to new parameterizations of 1-Lipschitz neural network layers. Experiments on certified robustness shows the effectiveness of the new paramterizations.
strength_and_weaknesses: Strength:
1) The authors provide a unified perspective on the development of 1-Lipschitz layers, which provides better understanding of the connections between different methods, and allows for new parameterizations of 1-Lipschitz layer.
2) Theorem 1 can recover a wide range of previous methods on parameterization 1-Lipschitz layers, and leads to varaitions of new parameterizations.
3) The authors show some computational efficient parameterizations as an extension to CPL. The performance on certified robustness surpasses previous 1-Lipschitz neural networks and further reduces the gap between certified accuracy and AutoAttack accuracy, implying the new parameterization is more expressive. 

Weaknesses:
1) The architectures that are used for SLL seem to be much larger than those used in OrthoConv (Trockman et al., 2021). For instance, Cayley-Large uses an architecture with 4 convolutional layers and 3 fully connected layers, while in Table 3, SLL-small has 20 convolutional layers and 7 fully connected layers. It may be good to also put number of parameters in the table performance comparison table, and comment on the effectiveness of different architectures of 1-Lipschitz neural network.
2) What activation functions do the authors use for SLL? It seems that the authors follow the architecture choice in Meunier et al. (2022) but it'd be good to mention the activation function choice because it is important for training 1-Lipschitz neural networks. Some previous works (e.g. Trockman et al., 2021) finds that gradient-norm-preserving GroupSort is much more effective than ReLU in training 1-Lipschitz neural networks. I am curious on whether this is also the case for the residual architecture. 
clarity,_quality,_novelty_and_reproducibility: This paper is well written, and the proposed perspective and techniques are novel. 
However, the template of the paper seem to be wrong and should be fixed.
summary_of_the_review: This paper proposes a unified perspective on the development of 1-Lipschitz neural network layer, and provides new parameterizations of 1-Lipschitz layer. The experiments show strong certified robustness results which demonstrate the flexibility of the new parameterization.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a unified framework for a group of well-known Lipschitz regularization methods in deep learning. Theorem 1 presents the algebraic condition on normalizing matrix $T$ which requires $WW^\top - T\preceq \mathbf{0}$. As a result, the framework includes spectral normalization, orthogonal regularization, Almost-Orthogonal-layer (AOL), Convex Potential Layers (CPL) schemes. Subsequently, Theorem 2 gives a new interpretation of AOL methods by observing that the chosen matrix $T$ results in the closest normalized matrix to the set of orthogonal matrices which is an important step toward achieving a $1$-Lipschitz network. Then, in Section 5 the authors generalize the CPL method by introducing new optimization variables in a diagonal matrix $Q$ which is simultaneously optimized with weight matrix $W$. The paper discusses several numerical results to show the merits of the proposed SLL method on robust learning problems over CIFAR-10 and CIFAR-100 datasets.

**Review Update**

The authors' response satisfactorily addresses my comments and I therefore raise my score to 8. 
strength_and_weaknesses: Strengths:

1- The paper has an interesting theoretical contribution and proposes a nice unified framework for Lipschitz regularization methods in deep learning.

2- The paper is well-written and easy to follow. The theorems are sufficiently explained and well supported by several examples.

Weaknesses:

1- The numerical results are somewhat preliminary and discussed for only CIFAR-10 and CIFAR-100 datasets. Discussing the results for one or two large-scale image dataset could provide a stronger support for the SLL approach. Also, the performance scores for the SLL method and AOL baseline are quite close and in several cases, AOL performs better than the proposed SLL apporach.

2- While the theorem proofs are interesting, I think they could have been presented in the Appendix. The proofs' space could have been used to discuss additional numerical results.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: The paper is written clearly and it is rather easy to follow the paper's discussion.

**Quality**: The paper discusses an interesting unified framework for Lipschitz regularization methods.

**Novelty**: While the paper revisits several existing Lipschitz regularization schemes, the unified framework and discussed extensions look novel to me.

**Reproducibility**: While I have not checked the details of numerical experiments, the numerical evaluation seems reproducible to me. 
summary_of_the_review: This paper proposes an interesting unified framework for Lipschitz regularization methods, and the theorems help extend and generalize existing Lipschitz regularization schemes. The numerical contribution could be further improved by experimenting the proposed methods on more datasets and finding factors in which they outperform the baseline AOL.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an SDP based formulation for the development of 1-Lipschitz architecture. They show that many of the existing techniques for generating 1-Lipschitz neural networks can cast as solution to this SGD condition (W^TW < T).

Furthermore, based on the SDP formulation the authors derive new conditions for Lipschitz Layers (based on different values of T) and show through experimentations that the SDP based lipschitz layers get better performance than previous approaches on natural and certified accuracy. 

Based on their setup, the paper also provides motivation for why Almost-Orthogonal-Layers promote orthogonality of layers during training.
strength_and_weaknesses: [Strengths]

- The paper is well written and easy to follow, and provide a theoretical framework that can unify the existing methodologies to get 1-Lipschitz functions.
- Theorem 1 provides shows that spectral normalization and convex potential layers can be looked at as the linear and residual block for the same value of T.
- The authors apply theorem 3 to come up with a new formulation of T with a new diagonal matrix Q.

[Weakness]

- The empirical verification is lacking. The authors have provided with a general framework for designing Lipschitz Layers, however they only show empirical results with one result from this formulation.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow, and very well motivated. 

One minor point is that it would be useful to write the statement for Gershgorin circle theorem so that the proof for Theorem3 could be easier to follow.
summary_of_the_review: I think the unifying framework introduced by the authors is interesting. However, the paper lacks enough empirical validation since the authors only show results for one type of layer from their SDP formulation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The contributions include introducing a unifying framework for various methods for training Lipschitz-constrained neural networks (useful for adversarial robustness, control systems, etc.) and using this framework to discover a novel and non-obvious Lipschitz-constrained architecture. Impressively, the general framework can easily handle convolution layers, which have often required complicated constraints. Experiments show improved results on provable adversarial robustness compared with a variety of prior approaches.

The reviewers are uniformly enthusiastic about this paper, and appreciate the generality of the approach and the clarity of the exposition. Some reviewers had concerns about experimental details (e.g. choice of architecture for the baselines), but the authors addressed these concerns with a variety of new experiments. Overall, I think this is a solid paper and recommend acceptance.
justification_for_why_not_higher_score: This work may meet the bar for an oral presentation, depending on the strength of other contenders.
justification_for_why_not_lower_score: This is a non-obvious unifying framework which captures a variety of existing methods, and it leads to a novel Lipschitz-constrained architecture which outperforms existing approaches. It strikes me as a solid paper overall, and will probably be in the top quarter of papers at the conference.
Decision: Accept: notable-top-25%