LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models
Authors: Kai Hu
Haoqi Hu
Matt Fredrikson
Venue: ICLR 2026 Poster (2026)
Keywords: robustness
Lipschitz
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A new architecture for scalable Lipschitz-based training

Abstract: Lipschitz-based certification offers efficient, deterministic robustness guarantees but has struggled to scale in model size, training efficiency, and ImageNet performance. We introduce \emph{LipNeXt}, the first \emph{constraint-free} and \emph{convolution-free} 1-Lipschitz architecture for certified robustness. LipNeXt is built using two techniques: (1) a manifold optimization procedure that updates parameters directly on the orthogonal manifold and (2) a \emph{Spatial Shift Module} to model spatial pattern without convolutions. The full network uses orthogonal projections, spatial shifts, a simple 1-Lipschitz $\beta$-Abs nonlinearity, and $L_2$ spatial pooling to maintain tight Lipschitz control while enabling expressive feature mixing. Across CIFAR-10/100 and Tiny-ImageNet, LipNeXt achieves state-of-the-art clean and certified robust accuracy (CRA), and on ImageNet it scales to 1–2B large models, improving CRA over prior Lipschitz models (e.g., up to $+8\%$ at $\varepsilon{=}1$) while retaining efficient, stable low-precision training. These results demonstrate that Lipschitz-based certification can benefit from modern scaling trends without sacrificing determinism or efficiency.

Published as a conference paper at ICLR 2026
LIPNEXT: SCALING UP LIPSCHITZ-BASED CERTIFIED ROBUSTNESS TO BILLION-PARAMETER MODELS
Kai Hu, Haoqi Hu, Matt Fredrikson Carnegie Mellon University ka i Chuêcs emuueedu
ABSTRACT
Lipschitz-based certification offers efficient, deterministic robustness guarantees but has struggled to scale in model size, training efficiency, and ImageNet per- formance. We introduce LipNeXt, the first constraint-free and convolution- I-Lipschitz architecture for certified robustness. LipNeXt is built using two tech- niques: (I) manifold optimization procedure that updates parameters directly on the orthogonal manifold and (2) a Spatial Shift Module to model spatial pat- tern without convolutions. 'he full network uses orthogonal projections, spatial shifts, a simple I-Lipschitz 3-Abs nonlinearity, and Lg spatial pooling to maintain tight Lipschitz control while enabling expressive feature mixing, Across CIFAR- 10/100 and Tiny-ImageNet, LipNeXt achieves state-of ootee--f--heaarr the-art clean and certified robust accuracy (CRA), and ImageNet it scales 1-2B large models, improv- ing CRA over prior Lipschitz models (e.g.. up to +8% at s=1) while retaining efficient, stable low-precisiog training. These results demonstrate that Lipschitz- based certification can benefit from modern scaling trends without sacrificing de- terminism or efficiency.
INTRODUCTION
Adversarial robustness represents fundamental challenge in machine learning (Szegedy et al., 2014). Numerous defense mechanisms have been developed to enhance model robustness against adversarial attacks (Gong et al. 2021; Kundu et al. 2021 Poursaeed et al. 2021 Liu et al. 2021 Pang et al., 202T). However, these approaches are predominantly empiiri defenses that cannot pro- vide formal guarantees of robust predictions. Consequently, models deemed robust under current evaluation protocols may remain vulnerable to more sophisticated attack strategies as they emerge.
This limitation is particularly concerning for safety-critical applications such as autonomous driv- ing (Huang et al. 2025). medical image processing (Laousy et al. 2023), and malware classifica- tion (Saha et al. 2024), where failures have severe consequences. To address this challenge, certified robustness has emerged as a promising research direction. Certifying the robustness of a test requires mathematical guarantee that the model's outputs remain unchanged within pre- defined fy-norm ball of radius around the input. The performance of these certification methods typically measured by the certified robust accuracy (CRA), which quantifies the proportion of correctly predicted inputs that are also provably robust within a specified radius.
methodological strands. The first, ran- domized smoothing (RS) (Cohen et al. 2019b] Yang et al. 2021 Jeong et al., 2021 Carlini et al., 2022). provides probabilistic guarantees by averaging a classifie's predictions under additive noise. The second exploits the Lipschitz properties of neural networks to yield deterministic (worst-case)
Research on robustness certification largely follows
certificates (Huang al. 202T Araujo al., 2023 Hu al., 2024). this work, we focus on advancing the latter direction; detailed comparison between the two appears in Appendix
Despite their theoretical appeal, Lipschitz-based certification methods have struggled to scale in practice. A central critique is the weak performance on large-scale benchmarks: models often un- derfit even small-sized datasets such as CIFAR-100 and degrade markedly on ImageNet (Hu et al, 2023). Most systems still rely small, VGG-style (Simonyan & Zisserman, 2014) architectures with <32M parameters. Although recent work has explored deeper/larger architectures for certified robustness (Hu et al. 2023) Araujo al. 2023). the gains plateau quickly model size increases.

---

Published as a conference paper at ICLR 2026
Examining existing work, find that orthogonal matrices are fundamental to building 1-Lipschitz networks because they enable tight Lipschitz bounds. However, they are also a major bottle- neck that prevents Lipschitz-based certification from scaling. Existing methods either explicitly re-parameterize orthogonal matrices or implicitly re- re-parameterize Lipschitz-bounda operations to learn near-orthogonal matrices; both introduce substantial computational overhead (see Section To address this issue, we propose directly optimizing orthogonal matrices on the orthogonal mani- fold. Although orthogonal manifold optimization been exploited for certification. We further observe that in the haree-odel regime, where learning rates are small, the matrix exponential can be accurately and efficiently approximated. Combining these ideas, we reduce the additional per-update overhead to at most five matrix multiplications.
mature technique, to our knowledge has not
natural path to scalability is transformers (Vaswani et al., 2017). which scale to billion- parameerr models and exhibit emergent abilities across a wide range of tasks (Wei et al.] 2022). How- ever, attention lacks straightforward Lipschitz control. Fortunately, ConvNeXt (Liu al. 2022)
and MetaFormer (Yu et al. 2022b) suggest that Lipschitz-bounded architectures can benefit from transformer-era design choices. Motivated by the simple token-mixing mechanisms in these modern architectures Yuu et al. 2022b),
design convolution -ree shifting module paired with positional encoding that uses simple shift operations to model spatial relations as the building block of our model. Compared prior work using heavy FFT-based convolution designs (Trockman & Kolter 2021 Lai et al. 2025) or power-iteration-basse I. Lipschitz regularization (Leino et al.) 2001] et al,,
2023
the shifting module adds minimal computational cost while preserving the capacity to model
spatial patterns.
Combining the aforementioned techniques, we propose LipNeXt, the first convolution-free and constraint-free architecture for certified robustness. Benefiting from this design, we scale to billion- parameter models and observe non-satine con-saturating gains with increasing model size. Our experiments show that LipNeXt outperforms state-of-tee ofat-oof-the-an the-art methods in both clean accuracy and certified robust- ness across CIFAR-10, CIFAR-100, Tiny-ImageNer and ImageNet, highlighting the effectiveness of our approach for scalable, provably robust deep learning.
Contributions. (i) A constraint-free manifold optimization scheme for learning orthogonal pa- rameters at scale with efficient exponential approximations and stabilization; (ii) a theoretically mo- tivated, convolution-fis Spatial Shift Module that preserves Lipschitz control while enabling spatial mixing; and (iii) state-of-the-an CRA and clean accuracy on standard benchmarks, with successful scaling to billion-paramen models.
PRELIMINARY AND RELATED WORK
2.1 PRELIMINARIES
Lipschitz-based Method for Certified Robustness Consider a neural network f Rd + RR and its corresponding binary classifier F(x) sign(f(x)), We say that classifier F is e-locally robust at point æ if for all perturbations x' satisfying || I'Ilp < we have F(x) = F(z'). Certified robustness represents a stronger condition where the robustness property cannot only be empirically verified but also mathematically guaranteed. In this work, we only focus on 2, i.e., certified robustness under lg norm.
Given an upper bound K on the Lipschitz constant of f. we can certify that F is locally robust at x with a guaranteed robustness radius of [(x))/K This certification follows from the Lipschitz property, which bounds the maximum change in ((x within the e-ball around x.
For multi-class classification with classes, we decompose the problem into N - binary classi- fication tasks using one-vs-rest approach. The certified robustness radius of the N-class classifier is then defined as the minimum certified radius across all constituent binary classifiers, thereby en- suring robustness for the prediction.
To compute Lipschitz bounds for deep networks, we leverage the compositional property that the Lipschitz constant of a composite function satisfies Lip(f g) < Lip(f)Lip(g), where (z) fg(x)). Consequently, for a feed-forward neural network, an upper bound on the overall Lips- chitz constant can be efficiently computed as the product of the Lipschitz constants of all constituent
2

---

Published as a conference paper at ICLR 2026
layers. However, this bound can be very loose, leading to extremely small certified radius. There- fore, designing a network architecture for which a tight Lipschitz bound can be computed is crucial to achieving satisfactory certified performance.
Orthogonal Manifold Optimization onal matrices: Ma {X E Rddd X X Ia), where I, denotes the identity matrix. Optimization problems on the orthogonal manifold can be formulated as:
The orthogonal manifold is defined as the set of all orthog-
min XEM
f(X).
A standard Euclidean gradient descent update of the form X + X5: - -VV()))(s))) with step size ? > 0 will generally cause X to leave the manifold, since this update does not preserve orthogo- nality constraints. Manifold optimization first projects the Euclidean gradient onto the tangent space of the manifold, thereby removing the component of the gradient that is orthogonal to the manifold. The Riemannian gradient (projected gradient) is given by:
grad f(x) = V f(() - Xsym X Vf(X),,
where sym(A) (A + //2 2 denotes the symmetric part of matrix A. Once the Riemannian gra- dient is computed, Exponential Map is employed to update Xk41 while preserving orthogonality:
X s++ X exp {-2
? skew grad [
where skew(A) = (A A 1/2,, and (x)(A) denotes the matrix exponential. Orthogonality is preserved since exp(skew(( (A)) orthogonal for any matrix A. [ééeigneul & Ganea (2018) ex- tended this optimization to adaptive optimizers like Adam, and algorithm provides an example for Manifold Adam Optimizer.
2.2 RELATED WORK
Re- paraameterizationnn for Lipschitz-based Certified Robustness The Lipschitz upper bound dis- cussed in Section 2.1 can be tight when all weights in the feed-forward network are orthogonal or near-orthogonal. Many methods have been proposed to parameterize weights as orthogonal ma- trices to enhance Lipschitz-based certification performance. [Sinla & Feizi (2021) employed the matrix exponential of skew-symmetric matrix to construct orthogonal (I-Lipschitz) linear layers and extended this approach to convolutions via Taylor expansion. Trockman & Kolter (2021) intro- duced the Cayley transform, Cayley(A) A) = (I- - + A), ), which yields orthogonal matrix when A is skew-symmetric. To apply this method to convolutions, the weights are transformed into the frequency domain via FFT before applying the Cayley transform. Xu et al. (2022) pro- posed LOT-Orth, defined as (AAT) A A. To mitigate the computational overhead introduced by the matrix square root in LOT-Orth, Hu et al. (2024) proposed Cholesky decomposition with Cholesky-Orth Orth(A) (A) Cholesky(AA ( -
Alternative approaches focus on near-orthogonal architectures. Prach & Lampert 2022) intro-
duced the AOL layer: f(x, A) Ad, AA Alxs) Ala))_aa -lx. Meunier al. (2022 proposed the
1-Lipschitz CPL layer defined as A)r = x- 2A T o /Aa))/ Al Araujo et al. 2023) introduced
the SLL layer given by f(x, A.q) 9) = x 2Adiag(E, AA A &j / (A x). While the AOL, CPL. and SLL. layers may not be orthogonal initialization, experimental evidence shows that these layers converge to nearly orthogonality at the end of Lipschitz-basee based training.
Efficient Manifold Optimization for Orthogonality While the matrix exponential operation en- sures orthogonality in manifold optimization (Section 2.J). its computation is computationally pro- hibitive. Several methods have proposed efficient
geodesic retractions as alternatives to ex- ponential mapping. Projection-based approaches by Absil & Malick (2012); Gawlik & Leok (2018) map gradients back to the orthogonal manifold but rely on computationally expensive SVD oper- ations. Other methods utilize the closed-form Cayley transform (Jiang & Dai 2015 Zhu 2017), yet require costly matrix inversions. Wen & Yin (2013) reduced the computational cost of the Cay- ley transform by imposing the restrictive assumption that for matrix dimensions
p if 2p < n. However, this algorithm becomes inefficient when 2p 2 n, which represents the majority of cases in deep neural networks. Shustin & Avron (2024) employ randomized sketching methods to handle
3

---

Published as a conference paper at ICLR 2026
large update matrices efficiently. Ablin & Peyré (0022 proposed landing algorithm that avoids costly matrix exponential retractions by using an efficient potential energy-based baaeyd update scheme that progressively pulls iterations onto the manifold.
3
METHOD
Our proposed method, LipNeXt, is built upon two core technical innovations designed for scalable certified robustness: a constraint-free optimization technique for learning orthogonal matrices and novel convolution-free spatial mixing operator. In this section, we first detail our manifold optimiza- tion approach, which enables efficient and stable training of orthogonal parameters directly on the Stiefel manifold. We then introduce the Spatial Shift Module, providing theoretical justification for its design as the unique norm-preent corm-preserving operator within the family of depth- convolutions. Finally, describe the overall LipNeXt architecture, which integrates these components into scalable and effective model.
3.1 CONSTRAINT-FREE LEARNING OF ORTHOOGNNAL MATRICES ON THE MANIFOLD
As discussed in Section 2.2 prior works use parameterization to learn orthogonal matrices ex- plicitly or implicitly. We name these as constrained approaches, as the optimization variables do not reside directly on the orthogonal manifold Mg. In contrast, we adopt a constraint-free man- ifold optimization perspective, where parameters are updated directly on the manifold, inherently preserving orthogonality. Section 2.1 introduces the details of orthogonal manifold optimization.
significant bottleneck manifold optimization the computational cost of the matrix exponen- Section 2.2 discussed solutions from prior work to address this issue.
tial, exp(-) in Equation 3
However, we observe that there exists very simple solution in context of training neural net- works. As the model size increases, the optimal learning rate for optimizers like Adam needs to decrease, (c.g., 10- Thus the exponential component in Equation EquationnJofter Jffte has a smaller Frobenius norm (because multiplied by the learning rate 77). We propose approximating the exponential with norm-adaptive Taylor series truncation. For a given skew-symmetric update matrix A, we define:
I A+ + 1 A2 I+ A
if |||[|| P 0.05,
2 A2 + 9 A
if 0.05 - |||]|p < 0.25, if 0.25 I ||]| <1,
FastExp(A)
=
I A+
2 12 + 13
A1
(4)
6
24
exp(
if ||A|| 1.
Here the byper-parameten 0.05, 0.25 and 1 are chosen empirically to balance accuracy and stability. One issue of FxptEEpp-)) is the truncation introduces small errors that violate exact orthogonality, which can accumulate and destabilize training over time. To address this issue, we introduce two stabilization techniques and mark them in Algorithm I
First, to control the accumulation of numerical errors from the truncated series, we perform pe- riodic polar retraction. fold. This is achieved by computing the Singular Value Decomposition X = UVVT and resetting X <- UVT, which finds the closest orthogonal matrix to X in the Frobenius norm. Although SVD is also expensive, we only need to perform it once per epoch. This step is detailed in lines 20-22 (red text) Algorithm
the end of each epoch, we project the matrix X back the mani-
Second, we adapt the Lookahead optimizer wrapper (Zhang et al. 2019) to the manifold setting. In nutshell, Lookahead optimizer wrapper updates the optimized parameters with an interpolation between the correct weight and the weight K steps earlier every K steps:
(t + 1) mod K 0, X, + 0.5X,, + 0.5X,-K
However direct interpolation orthogonal matrices breaks orthogonality. Applying the polar re- traction after interpolation is also expensive if applied every K steps.
4

---

Published as a conference paper at ICLR 2026
We instead interpolate the skew-symmetris updates in the tangent space. Suppose the learning rate is small, and the exponential component of update (A,) at every step is small:
X, = X,- exp(A.) = Xt-K
II
exp(A,) 2 Xt- exp(
Σ
A
(5)
f=t-K4
-=t-K+1
1
Thus we can approximate 0.5., + 0.5X,-K as X exp(
Σ
A,). To understand this,
j=t-k+1
starting at K, we update the parameters K steps and collect the update trajectories {Aj}ī K+ then we go back X,-K. and update X,-K using the half of all K trajectories. Lines 12~28 (blue text) of Algorithm T reflect this method.
Algorithm Stabilized Manifold Adam Optimizer with FastExp
learning rate 17 (on the order of 10 10-3), momentum coefficients 31 and 32. Number of steps to perform a Lookahead update K, and number of steps in one epoch N.
Input:
2: Goal: Minimize f(X) on the orthogonal manifold beginning at orthonormal weight E Ma.
3: Set the fast weight Xo +- X and slow weight sslow <- X
4: Set the Lookahead updating buffer Bo f 0 dxx
5: Set the first and second moment mo 0axd 10 dxd /d.
6: for step in , do
7: Compute the Euclidean gradient vf((x-1)).
8: Compute the projected gradient grad f(X+-1) using (f(x,-1) and X,-1 by Equation
9:
Update the first order moment: m, < f1m--1 + (1 3n )rrad f((X,-1).
10:
Update the second order moment: 812 - 32
32 32)grad d f(X(-1) * ggad f (Xt-1).
11:
Compute the exponential component A, s- -17 17. m,/D2
12:
Update the Lookahead updating buffer B, +- B, + A.. if (I + 1) mod K * 0 then
13:
14:
Update the fast weight: X; 1-++xx -X-1-1 FastExp( A.). else
16:
Update the slow weight: Xslow + Xsow East (Exx(BBB2))) B2/ Update the fast weight with the slow weight: X, X slow. Unset the updating buffer B, +- xxxn
18:
19:
end
20:
if (L+ mod N = 0 then
21:
Perform SVD on the fast weight: X, = U.VT. Force orthogonalization: X, t UV Xslow f- Xx.
22:
23:
end
24:
end for Output: Xslow-
25:
Relation to skew re-parameterization Skew re-parameterization constructs orthogonal matrices via the matrix exponential of skew-symmetric argument, i.c., exp(S) with S = X -X Building on this idea, Singla & Feizi (2021) applies a truncated Taylor expansion of the matrix exponential to implement orthogonal convolutions. In contrast, manifold optimization exploits the regime of small per-step updates, allowing the exponential map to be accurately approximated with lightweight computations. As result, retain orthogonality with significantly lower cost. By comparison, skew parameterization cannot leverage small-update structure in the same way, and SOC typi- cally requires larger truncation order reliably preserve orthogonality, increasing computational overhead and potential numerical eттor.
3.2 SPATIAL SHIFT MODULE: A CONVOLUTION-FREE DESIGN
Depthwise separable convolutions have demonstrated that effective spatial pattern learning requires neither extensive parameterization nor computational overhead (Howard et al. 2017 Tan & Le 2019 Liu et al. (2022). MetaFormer YYu et al. 20226) further showed that parameter- free operations replace depthwise convolutions while maintaining performance. Building this trajectory
5

---

Published as a conference paper at ICLR 2026
toward parameter efficiency, we introduce convolution-fre Spatial Shift Module tailored for certified robustness.
1D formulation. For input sequence X [2] . Fnl E Rd of length n with feature dimen- sion d, we partition each token's features into three parts: a, IRdd by Rad, and o (1-2dodd The spatial shift operation S(X) applies circular shifts to the first two partitions:
Q1
22
an bn
a2
Q1
an-
an-1
X =
bj
bg
= S(X)
b2
by
bn
bi
(6)
C1
C2
Cn
C1
C2 Ga-1
Cn
This shifts the first partition right by one position and the second partition left by one position, mixing adjacent token information without parameters. For 2D data, we extend to five partitions enabling shifts along horizontal and vertical axes. Empirically, shift ratios E {1/8, 1/16] yield optimal results.
Theoretical justification. Our design motivated by corm-preservation requirements in certi- fied robustness. The following theorem establishes fundamental constraints Lipschitz-preserving convolutions:
Theorem Let X E IR NxW be a cingle-channel tensor and fK be spatial convolution with kernel K bx unit stride, and circular padding (crucially relies on circular padding and does not hold under zero-padding). The operator JK is norm-preserving (tight I I-Lipschitz isometric):
lx fx (Y)Il = |X -X-YYle.
if and only if kernel K contains exactly one non-zero element with value +1.
Theorem reveals that norm preserving depthwise convolutions reduce to spatial shifts. Our module directly implements this principle. Proof of Theorem Tis is in Appendix B33
Circular padding and positional encoding. The strict norm preservation guaranteed by Theo-
rem [ necessitates the use of circular padding. Nevertheless, in related applications such as FFT- based convolution, zero-padding is a common alternative (Xu et al. 2022; Lai et al. (025).. The rationale for this deviation is to avoid the artificial mixing of boundary features (e.g., On with b2) that can occur with circular padding.
On
On-
Szcro-pad (X)
b2
by
bn 0
Cc1
C1-1
We also observe Spco-pad outperforms S in our convolutional-free architecture. However, we would explain this with another hypothesis that zero-padding introduces position information to the model (Islam et al. 2024). Under this hypothesis, we can address the issue by add explicit po- sitional encoding to the input and keep the circular padding. Our experiments in Section 4 verifies that this hypothesis is correct: by using circular padding S and explicit positional embedding, the model achieving superior performance because of norm preservation guarantees.
3.3 THE LIPNEXT ARCHITECTURE
We now define the LipNeXt architecture, which integrates our manifold optimizer and Spatial Shift Module. The core of the architecture is the LipNeXt block.
Let X C be the input tensor. First, we add a learnable positional embedding p f x1. which is broadcast across the channel dimension: X' = X + p. Next, we apply the core mixing operation. We use an orthogonal matrix R E Mc to project the channel-wise features, apply the 2D Spatial Shift Module S, and then project back using R.
Y R S(RX') HX
(8)

---

Published as a conference paper at ICLR 2026
The projections R and R' ensure that the shift operation is not always applied to the same fixed sub- set of channels, enabling comprehensive feature mixing across all chamnels over successive blocks. Without S, this operation reduces to identity mapping Y R RX' X'. To learn more complex transformations, we apply another orthogonal matrix M E Mc and an activation a:
= o + (MM S (R(X + p)) + b).
For the activation, we propose 3-Abs, a flexible and GPU-friendly 1-Lipschitz function. For an input vector R2, it is defined channel-wise
[B-Abs(<)]: = [|aal if ,dd
(10)
Zis
otherwise.
3-Abs is simple yet powerful non-linearity, We can show that the commonly used MinMax acti- vation can be expressed by B-Abs if 0.5:
3R E M2a, Vx E R2d MinMax(x) = R² 3-Abs(Rtr.
(11)
The hyperparameter B E [0, controls the degree of non-linearity. We defer further discussion to the Appendix B.I] A PyTorch-like code of the LipNeXt block is given at Appendix B.2
The overall LipNeXt architecture adapts the macro-structure of LiResNet (Hu al. 2023 We replace the original LiResNet blocks with our LipNeXt blocks and substitute the Neck module with simple pooling layer,
manifold optimizer is designed for square matrices. Following the philosophy of Vision Transformers (Dosovitskiy et al. (2220), we rely on the scalability of the backbone to learn effective features. To ensure the entire network is 1-Lipschitz, we Spatial Pool, which computes the 1.2 norm over the spatial dimensions for each channel. For an input X & RHxW'CC, the output is vector in R.C where the e-th component is:
L.2
w
[L2SpatialPool(X))..
Σ
X2
(12)
h,w.ce
This operation is 1-Lipschitz and serves as the final global pooling before classification.
EXPERIMENTS
In this section, we present the empirical evaluation of our approach. We first compare our method with the SOTA certified robustness methods on the widely used CIFARIO, CIFAFR 100 and Tiny- ImageNet benchmark. Next we conduct scaling experiments to show the scalability of our model in terms of network depth, network with and dataset size. We the EMMA loss (Hu al. 2023) for our training and follow the same training receipts in LiResNet++ (Hu et al. 2024p. See Code for implementation details. By default, we do use ceffusiion-gneerane synthetic data.
Main Results We compare LipNeXt against SOTA baselines. T clean accuracy, cer- tified robust accuracy (CRA), and parameter counts. For LiResNet (Hu et al, 2023). follow the setting without ceffusiion-geeeraa synthetic data, using the numbers reported by Lai et al. (2225) Results that leverage
data are presented separately in the next table. Lip
achieves the strongest performance on nearly all metrics; the only exception is CRA at = 108/255 on CIFAR-10. where AOL (Prach & Lampert. (2022) higher. However, AOL attains this point by incurring a substantial drop in clean accuracy and in CRA at the other radii. Even under the similar parameter budgets, our smaller configuration L32W 1024 outperforms prior work by clear margins.
Performance with extra data As we further scale up LipNeXt, we continue to observe improve- ment in clean accuracy, but decrease in CRA. This is a known issue called robust overfitting (Rice) et al. 20200 Yu et al. 2022a) due to the small dataset size. Following prior work (Hu et al., 2024] Lai et al. (025).
add diffusion-gentt generated synthetic data to the training set to solve this issue, and use the generated data provided by Hu et al. (2024). Table 2 shows LipNeXt can effectively leverages these synthetic datasets enhance performance.
7

---

Published as a conference paper at ICLR 2026
Table 1: Clean and Certified robust accuracy (CRA) of prior works and our LipNeXt models CIFAR-10/100 and TinylmageNet. The best results are marked in bold.
Clean
CRA. (%) at (E)
Datasets
Models
#Param.
Acc.
36/255
72/255
108/255
Cayley Large
[Trokkman & Kolter 2021
21M
74.6
61.4
46.4
32.1
SOC-20 Singla & Feizi 2021
27M
76.3
62.6
48.7
36.0
LOT-20 Xu al. 2022
18M
77.1
64.3
49.5
36.3
CPL XI (Meunier
236M
78.5
64.4
48.0
33.0
CIFAR-10
AOL Large
Prach & Lamper.
2022
136M
71.6
64.0
56.4
49.0
SLL X-Large
et.l
N123
236M
73.3
64.8
55.7
47.1
LiResNet. Hu 2024
83M
81.0
69.8
56.3
42.9
BRONet
68M
81.6
70.6
57.2
42.5
LipNeX 1.32W 1024
64M
81.5
71.2
59.2
45.9
LipNeXt 1.32W0048
256M
85.0
73.2
58.8
43.3
Cayley Large [Trockman & Kolter 2021
21M
43.3
29.2
18.8
11.0
SOC-20 (inglla
27M
47.8
34.8
23.7
15.8
LOT-20 Xa al. 2022
18M
48.8
35.2
24.3
16.2
CPL XL (Munier et al. 2022)
236M
47.8
33.4
20.9
12.6
AOL Large Prach & Lamper.
2022
136M
43.7
33.7
26.3
20.7
CIFAR-100
SLL X-Large (Armjo
236M
47.8
36.7
28.3
22.2
Sandwich (Wang & Manchester
223
26M
46.3
35.3
26.3
20.3
LiResNet
Hu al. 2024
83M
53.0
40.2
28.3
19.2
BRONet et
68M
54.3
40.2
29.1
20.3
LipNeXt T.32W 1024
64M
53.3
41.3
30.5
21.8
LipNeXI 1.32W2048
256M
57.4
44.1
31.9
22.2
SLL X-Large (Aranjo et 2023
1.IB
32.1
23.2
16.8
12.0
Sandwich
(Wang Munchester. 2023
39M
33.4
24.7
18.1
13.4
Tiag-em
LiResNet Ha 2024
83M
40.9
26.2
15.7
BRONet
Lai ct 2025
75M
41.2
29.0
19.0
12.1
LipNeXt 332W1024
64M
42.5
32.0
21.8
15.2
LipNeXt L33W2048
256M
45.5
35.0
25.9
18.0
Table 2: Comparison clean and certified accuracy using extra diffusion generated data from.
Results of other methods reported by Lai al. (2025). The best results are marked in bold.
Lipschitz Backbone
CIFAR-14 10
CIFAR- --00
#Param.
Clean
36/255
72/256
108/255
Clean
72/255
108/256
255
LOT
59M
85.7
76.4
65.1
52.2
59.4
47.6
36.6
26.3
Cayley
68M
86.7
77.7
66.9
54.3
61.1
48.7
37.8
27.5
Cholesky
68M
85.4
76.6
65.7
53.3
59.4
47.4
36.8
26.9
SLL
83M
85.6
76.8
66.0
53.3
59.4
47.6
36.6
27.0
SOC
83M
86.6
78.2
67.0
54.1
60.9
48.9
37.6
27.8
Lip-reg
83M
86.7
78.1
67.0
54.2
61.1
48.9
37.5
27.6
BRO
68M
87.2
78.3
67.4
54.5
61.6
49.1
37.7
27.2
Ours L32W2048
256M
88.2
79.2
68.0
54.9
62.1
51.2
38.5
27.5
Ours L32W2896
512M
92.7
81.7
68.6
55.8
63.6
55.2
39.2
28.3
Performance on ImageNet The main criticism of Lipschitz-based certification is the poor per- formance on large-scale datasets like ImageNet. We show that LipNeXt is feasible to scale up and achieve non-vanishing performance as model size increases. We consider two certification radius 36/253, widely used for Lipschitz based method and ş 1, widely used for randomized smooth- ing based method. Since the difference between the two radii is large, we train two models with different paramtearameters. Specifically, the maximum training radius is set 1.5 times of the test radius. Table B shows the comparison with prior work. By scaling to billon-level large models, LipNeXt outperforms prior work by 8% for 1 and 3% for 36/255.
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of clean aceuracy and CRA ImageNet. "Use EMD" indicates training with
additional diffusion-generatee data, and training speed is reported in minutes per epoch.
Training Speed
Use
30/255
Lipschitz Model
#Param.
EDM
Clean Acc.
CRA
Clean Acc.
CRA
LiResNet
51M
5.3
X
18.8
14.2
45.6
35.0
LiResNett+
83M
49.0
38.3
BRONet
86M
10.5
X
49.3
37.6
BRONet
86M
52.3
40.7
Ours L32W4096
1B
8.9
X
40.2
21.1
55.9
40.3
Ours L32W5792
2B
17.8
X
41.0
22.4
57.0
41.2
Table (a):
Fix depth as 32
Table (b):
Fix width as 2048
Table (c): Fix #Param as 1B
width
Acc.
CRA.
depth
Acc.
CRA.
depth, width
Acc.
CRA.
1024
40.5
22.9
8
30.7
22.4
(8, 8192)
49.5
28.6
1456
43.5
24.5
16
43.5
24.7
(16, 5792)
50.4
29.2
2048
45.8
26.2
32
45.8
26.2
(32, 4096)
51.7
30.0
2896
48.5
28.2
64
47.0
26.9
(64, 2896)
51.2
29.6
4096
51.7
30.0
128
47.5
26.8
(128, 2048)
50.1
28.9
Table 4: Clean accuracy and CRA at = 1 on ImageNet (400 classes). Each sub-table fixes one
factor (depth, width, or parameters) to study different configurations.
Training stability and efficiency. The wall-clock training time in Table measured using single 8xH100 machine. The L32W5792 configuration trained with two such machines, and we equivalent single-node throughput. LipNeXt adopts constraint-
normalize the reported speed free optimization whose forward and backward passes involve only orthogonal matrix multiplica- tions and spatial shift operations, and train it using bfloatis 6. By contrast, LiResNet em- ploys power iteration to estimate per-block Lipschitz constants and regularizes the backbone Lips- chitz (the product across blocks) in the loss. On ImageNet we observe numerical instability when training LiResNet with bfloat 16 (numeric overflow occurs occasionally), so we can only use float 32 for LiResNet instead. BRONet builds I-Lipschitz convolutional networks via FFT-based frequency convolutions, which require complex arithmetic in both forward and backward passes. Because mainstream training frameworks currently support only complex33 ex32, this effectively curs fl oat 64-like memory and compute overheads and prevents BRONet from fully leveraging low-precision accelerators. As result, LipNeXt can continuously benefit from hardware advances due to its more stable architecture and optimization. Despite scaling to substantially larger models than LiResNet and BRONet, LipNeXt achieves training throughput on par with prior work, enabling further scaling. We leave training LipNeXt on large-scale image-text pair datasets future work.
Scaling experiments We demonstrate clear scaling trends for LipNeXt across both backbone depth and width dimensions. Due to computational constraints, we conduct experiments on ran- domly sampled subset of 400 ImageNet classes, reporting clean accuracy and CRA
1 in Table We systematically evaluate three scaling scenarios: (a) fixing backbone depth at 32 layers while varying width, (b) fixing backbone width W 2048 while varying depth, and (c) constrain- ing the total parameter count to 1B while exploring different depth-width configurations. Our results reveal that LipNeXt exhibits favorable scaling properties with respect to both architectural dimen- sions, with performance improvements observed when increasing either depth width. Notably, under fixed parameter budgets, a depth 32 layers yields optimal performance.
Ablation Studies Due to space constraints, extended ablations are provided in Appendix C Ta-
ble 5 evaaluates the two stabilization techniques for FastExp; Table 6 isolates the contribution of the spatial shift module: Table compares padding and positional embedding choices; and Table g ablates activation functions.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We presented LipNeXt, scalable 1-Lipschitz architecture that is both constraint-free and convolution-free. On the optimization side, we update orthogonal parameters directly on the man- ifold using stabilized scheme, eliminating power-iteration penalties and avoiding the numerical fragiliiy that inhibits low-precision training in prior work. On the architectural side, we replace depthwise convolutions with a theoretically grounded Spatial Shift Module. Together, these choices enable Li to scale cleanly in depth and width and to realize consistent gains in both clean ac- curacy and CRA. Empirically, LipNeXt establishes new state of the art across CIFAR- 10/00000 Tiny- ImageNet, and ImageNet with billion- billion-parameter models, while maintaining competitive throughput and stable 6 training. These results indicate that deterministic, Lipschitz-based certification can track modern scaling trends and further scale up.
REFERENCES
Pierre Ablin and Gabriel Peyré. Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference on Artificial Intelligence and Statistics, pp. 5636-5657. PMLR, 2022.
P-A Absil and Jérôme Malick. Projection-like retractions matrix manifolds. SIAM Journal on Optimization, 22(1):135-158, 2012.
Alexandre Araujo, Aaron J Havens, Blaise Delattre, Alexandre Allauzen, and Bin Hu. A uni-
fied algebraic perspective on lipschitz neural networks. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httpps://openeeiieets http:/ewwwweeeeeeeee new.:orumttotrreoooo
K71IGLC8cfc:
Gary Bécigneul and Octavian Eugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arxiv:1810.0070 2018.
Nicholas Carlini, Florian Tramer, Zico Kolter, et al. (certified!!) adversarial robustness for free! arXiv preprint arXiv:2206.105500 2022.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), 2019a.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learming, pp. 1310-1320. PMLR, 2019b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR 2021, 2020.
Evan S Gawlik and Melvin Leok. High-order retractions on matrix manifolds using projected poly- nomials. SIAM Journal on Matrix Analysis and Applications, 39(2):801-88) -888, 2018
Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. Maxup: Lightweight adversarial train- ing with data augmentation improves neural network training. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 2483, 2021.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1701v:1704.0466.1 2017.
Kai Hu, Andy Zou, Zifan Wang, Klas Leino, and Matt Fredrikson. Scaling in depth: Unlocking robustness certification on imagenet. arXiv preprint arXiv:2301.12549, 2023.
Kai Hu, Klas Leino, Zifan Wang, and Matt Fredrikson. A recipe for improved certifiable robustness. In The Twelfth International Conference on Learning Representationss 2024.
Yujia Huang, Huan Zhang, Yuanyuan Shi, J. Zico Kolter, and Anima Anandkumar. Training certifi- ably robust neural networks with efficient local lipschitz bounds. In NIPS, 2021.
10

---

Published as a conference paper at ICLR 2026
Zijian Huang, Wenda Chu, Linyi Li, Chejian Xu, and Bo Li. Commit: Certifying robustness of multi-sensor fusion systems against semantic attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17528-175360 2025.
Md Amirul Islam, Matthew Kowal, Sen Jia, Konstantinos G Derpanis, and Neil DB Bruce. Position, padding and predictions: A deeper look at position information in cnns. International Journal of Computer Vision, 132(9):388933900 2024.
Jongheon Jeong, Sejun Park. Minkyu Kim, Heung-Charn Lee, Do-Guk Kim, and Jinwoo Shin. Smoothmix: Training confidence-calibratee smoothed classifiers for certified robustness. Ad- vances Neural Information Processing Systems, 34:30153- 3016688, 2021.
Bo Jiang and Yu-Hong Dai. framework of constraint preserving update schemes for optimization on stiefel manifold. Mathematical Programming, 153(2):535-5777 2015.
Souvik Kundu, Massoud Pedram, and Peter A Beerel. Hire-snn: Harnessing the inherent robustness of energy-efficiene deep spiking neural networks by training with crafted input noise. Proceed- ings of the IEEE/CVF international conference computer vision, pp. 5209-5218, 2021.
Bo-Han Lai, Pin-Han Huang, Bo-Han Kung, and Shang-Tse Chen. Enhancing certified robustness via block reflector orthogonal layers and logit annealing loss. In International Conference on Machine Learning (ICML), 2025. Spotlight.
Othmane Laousy, Alexandre Araujo, Guillaume Chassagnon, Nikos Paragios, Marie-Pient Revel, and Maria Vakalopoulou. Certification of deep learning models for medical image segmentation. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 61 1-211. Springer, 2023.
Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning, pp. 6212-6222. PMLR, 2021.
Aishan Liu, Xianglong Liu, Hang Yu, Chongzhi Zhang, Qiang Liu, and Dacheng Tao. Training robust deep neural networks via adversarial noise propagation. IEEE Transactions on Image Processing, 30:5769-517 -7811 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022.
Laurent Meunier, Blaise Delattre, Alexandre Araujo, and Alexandre Allauzen, A dynamical system perspective for lipschitz neural networks. In International Conference Machine Learning, pp- 15484-15500. PMLR, 2022.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In International Conference on Learning Representations, 2021.
Omid Poursaeed, Tianxing Jiang, Harry Yang. Serge Belongie, and Ser-Nam Lim. Robustness and
generalization via generative adversarial training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15711-15720, 2021.
Bernd Prach and Christoph H Lampert. Almost-orthogonal layers for efficient general-purpose lipschitz networks. In European Conference on Computer Vision, pp. 350-365. Springer, 2022.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International conference on machine learning. pp. 8093-8104. PMLR, 2020.
Shoumik Saha, Wenxiao Wang, Yigitean Kaya, Soheil Feizi, and Tudor Dumitras. DRSM: De- randomized smoothing on malware classifier providing certified robustness. In The Twelfith In- ternational Conference on Learning Representations, 2024. URL httpps:/opeezeviies
fuet/Eorue
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Ad- vances in Neural Information Processing Systems, 32, 2019.

---

Published as a conference paper at ICLR 2026
Hadi Salman, Mingjie Sun, Greg Yang. Ashish Kapoor, and J Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, 33:21945-219577 2020.
Boris Shustin and Haim Avron. Faster randomized methods for orthogonality constrained problems. Journal of Machine Learning Research, 25(257):1-59, 2024.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 140:::409915000 1556, 2014.
Sahil Singla and Soheil Feizi. Skew orthogonal convolutions. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9756-9766. PMLR, 18-24 Jul 2021.
Christian Szegedy. Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
Asher Trockman and Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. In International Conference on Learning Representations (ICLR), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Ruigang Wang and Ian Manchester. Direct parameterization of lipschitz-bounder deep networks. In International Conference Machine Learning, pp. 36093-36110. PMLR, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0700 2022.
Zaiwen Wen and Wotao Yin. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 142(():397-434, 2013.
Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anand- kumar, Bo Li, and Dawn Song. Densepure: Understanding diffusion models towards adversarial 2022.
robustness. arXiv preprint ar arXiv:2211.00330
Xiaojun Xu, Linyi Li, and Bo Li. Lot: Layer-wise orthogonal training on improving 12 certified /620,
robustness. arXiv preprini arXiv:2210. /6220, 2022.
Zhuolin Yang, Linyi Li, Xiaojun Xu, Bhavya Kailkhura, Tao Xie, and Bo Li. On the certified robustness for ensemble models and beyond. arXiv preprint arXiv:2107.10873 2021.
Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Under- standing robust overfitting of adversarial training and beyond. In International Conference Machine Learning, pp. 25595-25610. PMLR, 2022a.
Weihao Yu, Mi Luo, Pan Zhou Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, pp. 108 108999 2022b.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, step back. Advances in neural information processing systems, 32, 2019.
Xiaojing Zhu. A riemannian conjugate gradient method for optimization on the stiefel manifold. Computational optimization and Applications, 67(1):33-110, 2017.
12

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
In contrast to the deterministic robustness guarantees emphasized in this work, Randomized Smooth- ing (RS) (Cohen et al. (019a) provides probabilistic guarantees and has been extensively studied at ImageNet scale (Salman et al. 2019 Jeong et al. 2021 Salman et al. 2020). Diffusion meth- ods (Carlini et al. [022|Xiao et a., a 2022) are introduced denoise the addictive noise and further improve the performance.
Despite these successes, RS methods face two fundamental limitations that constrain their practical applicabiliity. First, their inherently probabilistic nature introduces the possibility of false positives, where adversarial examples may be incorrectly certified as robust. While existing RS approaches typically maintain false positive rates below 0.1 16, even this level of uncertainty renders them un- suitable for security-critical applications where certification guarantees must be absolute. Although one can reduce the false positive rates by improve the confidence level a, it will then require sig- nificant greater number of noised images, which leads the second limitation: the computational overhead of RS AS-based certification is prohibitive, necessitating thousands to tens of thousands of forward passes per image due to reliance on concentration inequalities with tight tail bounds. This computational burden has confined most empirical evaluations to small test subsets of at most 1 1,000 images, limiting the scope of practical validation.
B ADDITIONAL METHOD EXPLANATION
Algorithm 2 shows the naive Adam Optimization. The major issue of this optimization is the high computational cost of matrix exponential.
Algorithm Manifold Adam Optimizer
1: Input: learning rate 77, momentum coefficients 31 and B2, optimization objective f(X).
2: Initialize X as an orthonormal matrix, and the first and second moment m 0
3: for step in .... T do
4: Compute the Euclidean gradient vx());
5: Compute the projected gradient grad f(X) using V (X) and X by Equation 2
6: Update the first order moment: m +- 3,m= + (1 - 31 grrad f(X);
7: Update the second order moment: V 32U + (1 32)grad f(X) grad f(X);
8: Rescale m + m/(1 {1) 0 t v/(11 - 65);
9: Update the weights: X +- X exp(-1) m
end for
B. FURTHER DISCUSSION ABOUT 3-ABS ACTIVATION
We show that the commonly used MinMax activation can be expressed by B-Abs if 3 = 0.5.
Theorem Consider 2d-dimensional input (21, (21,
where 01, E Rd. Define
Ia -Id]
22
I8
Let 3 0.5, we have
MinMax (x) = R. 8-Abs(Rz)
Proof: First
3-Abs(Rr)=BBAbbb
= A-Abs(
J1 x22
[1 x22
V2
F1 + x2
V2
31 + 32
Then
R 3-Abs(Ra)
1
[.
32l
+ 21
+ 02
max [ (11, x2
x2
22
(min(21,20)
Here max and min are element wise operations for vectors. Theorem shows that MinMax and 3-Abs should have the same expressive ability. Increasing 3 introduces more non-linearity and decreasing 3 introduces more linearity. However, MinMax corresponds to a fixed non- coneaiiear,,,, whereas A-Abs allows continuous interpolation between linear and piecewise-linear regimes.
13

---

Published as a conference paper at ICLR 2026
B.2 PYTORCH-LIKE CODE FOR THE LIPNEXT BLOCK
Below a PyTorch-like Code for the LipNeXt block for a better understanding.
def shift in (x, alpha=1/16);
x. shape|31
d = int (c * alpha)
a0, al, a2, a3, a4 = forch.split
x, [c d 4, d, d, d, dl, dim=3)
al terch. roll (al, dims=1, shifts-1)
a2 terch. roll (a2, dims=1, shifts--1)
a3 torch, roll (a3, dims-2, shifts-1)
a4 torch. (a4, dims-2, shifts--1)
11
12
x torch. cat [a0, al, a2, a3, a4), dim=3)
return
15
def beta_abs (x, beta-0. :
16
d int (x. shape[1] beta)
17
x = torch.cat ( [[x[..., :d] .bs (),, xl..., d:]], dim=-1)
return
20 def lipnext_block (x, R, M, b, pos)
21
shapo of x: (B, H, W, C)
22
shape of R and Mi (C, C)
shape of b: (C)
shape of pos: (8, 1)
25
x x pos
26
x = F.linear (x, R)
27
= shift shift_fn(x: fn(x)
F.linear (x, N R.T, b))
beta_abs (x)
30
return
B.3 PROOF THEOREM I
Part 1: (-) Sufficiency
Assume the kernel K contains exactly one non-zero element with value #1, Let this element be at position (do.ju), so Kig.jo and all other elements of K are zero.
The convolution fK is linear operator. Therefore, we can write:
(X - fx = IIfx(X Y)IEF
Let Z X - Y. We need show that []fx(Z))l= |Z]|r. The result the convolution, let's call it W = =fx(Z), is given by:
k-1 R-1
Wr,
Km,n Zi-m.i-n
x=
where the indices for Z are taken modulo H and W due to circular padding. Since only Kig is non-zero, this sum simplifies to single term:
Wiji Kio Zi-io.i- Zi-io
This means the output tensor W is a circularly shifted version of the input tensor Z, with each element multiplied by
Now, let's compute the squared Frobenius norm of W:
|
Σ
W
Σ
Σ cc Zi-
14

---

Published as a conference paper at ICLR 2026
Since = =1, we have c² 1.
#-1 W-1
W
Σ 1=
Σ
22-
==0
The mapping (i,j) + (i in (mod H).j - jo (mod W)) is a bijection on the set of indices. Therefore, the sum on the right is simply reordering of the sum of the squared elements of Z.
H-1W-1
H-1W-1
4-
7
=
|
1=0
<=0 ,'=0
Thus, we have shown that [| (2) =| which implies ||xx Z| This completes the first part of the proof.
Part 2: (=) Necessity
Assume that fx is norm- cemm-peservingg i... fxx = =I
for all X,Y e Rxx As before, let Z = X Y. The condition is equivalent to stating that fx is linear isometry with respect to the Frohenius norm:
(2) = |
VZ HxW
We analyze this condition in the frequency domain. Let F denote the 2D Discrete Fourier Transform (DFT), and let Å F(A). The convolution theorem for circular convolution states:
F(fx(Z)) K' @ 2
where K" the kernel K zero-padded to size x W, and @ denotes element-wise (Hadamard) product.
Parseval's theorem relates the Frobenius norm of a tensor to the Frobenius norm of its DFT:
||
|
HW
Applying Parseval's theorem to our isometry condition
= 11
HW [I HW
1
1
121
HW
| の 21
11
Expanding the norms in terms of their elements:
W
H-1W-1
Σ
K 0
Zu,,l 2 = Σ Σ
==0
u=0 ==0
#-1 W-1
11 W-1
Σ
Σ
K' u.n 11 2= Σ Σ
1,,,r
u=0
m
This can be rewritten as:
H H-IW-
Σ
Σ
==0
Since this equality must hold for any tensor Z, must hold for any possible DFT Ž. Let us choose a 2 that has only one non-zero element, say Zu0.zn and all other elements are zero. For this choice, the equation simplifies to:
R un..t 2 11
12 12= 12
K uo 2 2=
As we can make this choice for any frequency pair (wo, vo), it must be that RR' w,ol = 1 for all Now this property derive constraints the spatial domain kernel K.
15

---

Published as a conference paper at ICLR 2026
1. Sum of Squares of Elements: Apply Parseval's theorem the padded kernel K' itself:
H
W
K = ||
R
K u
HW
HW
==0
Since we found that [K" 0, = for all (u, v), the sum becomes En, 1 = HW.
|
HW (HW) = 1 Σ
K =
1
2. Sum of Elements: Consider the DC component of the DFT, ie., (u, v) (0,0):
H-1 W-1
k-1 R-1
K'o = Σ
Kg
e =
Kis
1=
1=0
2=0 ==
Since Roooo 1, we must have:
Σ
Kis =1
Let (k1, k2..... kN be the set of N non-zero elements in the kernel K. From our derivations, we have two conditions on these elements:
1.
=1
2.
Σ 1= ki =1
We can expand the second condition:
Σ
Σ
k2 +
Σ ISiCiEN
hyky
Substituting the first condition into this expansion:
1
Σ
ki =1
E kigki = 0
ISiCJEN
ISKSSN
We now have Σ 2o and
Σ 122
= ( k.) Let's consider the Cauchy-Schwand inequality on and b (ki, .2,,..,, ,NN) E RN
the vectors a (1, 1) E R
(a b) < ||a||||b||3
12
k
Plugging in our conditions:
NSN1 -
Equality holds if and only if one vector is a scalar multiple of the other, i.e., = ca for some scalar This means all non-zero elements must be equal: k₃ ka kN c.
Let's impose this equality on our conditions:
1. = =
2. - e) 2 Substituting d² = 1/N from the first equation into the second gives:
1
2 =
This shows that there must be exactly one non-zero element (N 1).
Let this single non-zero element be k11 From condition 1, A 1, which implies k +1. Therefore, for fR to be norm- norm-preser,ing. the kemel K must contain exactly one non-zero element with value of +1 -1. This completes the second part of the proof.
16

---

Published as a conference paper at ICLR 2026
C
ADDITIONAL RESULTS
Table 5: Ablation study on the effectiveness the two stabilization techniques for FastExp.
LipNeXt
CIFAR-10
CIFAR-100
L32W2048
Clean
36/255
T2/255
108/255
Clean
72/255
108/255
255
Algorithm
85.0
73.2
58.8
43.3
57.4
44.1
31.9
22.2
No Periodic polar retraction
84.8
72.3
57.2
42.0
57.2
43.3
31.4
21.3
No Lookahead
84.4
72.4
57.8
42.7
57.4
43.5
31.4
21.7
Table shows the effectiveness of the two stabilization techniques for FastExp. Removing either would lead constant performance drop.
Table 6: Ablation study on the effectiveness of the spatial shift module
LipNeXt L32W2048
CIFAR-10
CIFAR- 100
Clean
36/255
?2/255
108/255
Clean
36
72/255
108/255
255
1/16 (baseline)
85.0
73.2
58.8
43.3
57.4
44.1
31.9
22.2
G (no shift)
62.2
53.6
43.7
32.2
40.8
32.0
22.3
14.3
1/4 (shift all channels)
79.0
67.4
51.4
38.3
52.5
39.1
26.4
18.2
Table the effectiveness of the spatial shift module. Setting a 1/16 shifts one quarter the channels (distributed across four directions). We examine two extremes: a: O, which applies no spatial shift, and a 3/4, which shifts all channels. As expected, a performs poorly because the model cannot capture spatial interactions. Interestingly, 1/1 also degrades performance; hypothesize that shifting every channel removes absolute positional cues, preventing the model from retaining information about the original locations.
Table 7: Ablation study on the design choices of the padding and positional embedding
Padding
Positional Encoding
CIFARIO clean Acc.
CIFARIO CRA
CIFARI00 clean Acc.
CIFAR100
Type
CRA
Circular
85.0
73.2
57.4
44.1
Circular
X
84.3
72.1
56.2
43.2
Zero
X
84.5
72.4
56.5
43.5
Zero
84.6
72.6
56.8
43.7
Table presents an ablation padding and positional encoding choices. As discussed Section 3 without positional encoding, zero padding outperforms circular padding because it implicitly intro- duces positional information. With positional encoding enabled, the trend reverses: circular padding is preferable since it guarantees Lipschitz--ight transformation.
Table 8: Ablation study on the choice of activations
LipNeXt L32W2048
CIFAR-10
CIFAR-100
Clean
36/255
72/255
108/255
Clean
36
72/255
108/255
255
MinMax
84.4
72.7
58.0
42.6
56,9
43.5
31.2
21.4
0.5
84.6
72.9
58.3
42.9
57.1
43.8
31.5
21.7
3 = 0.75 (default)
85.0
73.2
58.8
43.3
57.4
44.1
31.9
22.2
1.0
84.3
72.5
58.1
42.3
56.5
43.2
31.0
20.8
3-Abs and MinMax should have the same expressive ability when 0.5. Experiments show that MinMax is slighter worse than = 0.5. We can adjust 3 to control the non-linearity, and 3 degrades to the absolution function optimal because it breaks identical mapping.
Table ablates the activations. As we shown in Appendix B.1
17


Reviews:
[{"review_id": "pZ6w0vaA3x", "reviewer": "Reviewer_BPtS", "summary": "The paper proposes a new approach for constructing Lipschitz neural networks and demonstrates improved certified robustness compared with several baselines. The method primarily focuses on two key components. They are Manifold Optimization and Spatial Shift Module. The latter one is novel to me.\nOverall, I lean toward accepting this paper. I may further increase my score if the authors can address my concerns properly.", "questions": "1.\tI suggest separating Sections 3.1 and 3.2–3.3. This would allow the paper to present Sections 3.2–3.3 together with other orthogonal methods such as LOT, BRO, or Cholesky for comparison. I also recommend adding an ablation study combining those orthogonal baselines with Sections 3.2–3.3 to better highlight their contribution.\n2.\tIn Table 2, the proposed model appears larger than the baseline. Could the authors include LipNeXt L32W1024 in the table for a fairer comparison? Additionally, I am curious about the scaling ability of the baselines, as the proposed method shows clear scaling behavior when synthetic data are used.\n3.\tIn Table 3, could the authors provide the EDM results for the proposed method? Including these results would help illustrate how scaling up combined with EDM could further enhance performance.\n4.\tI do not fully understand the relationship between β-Abs and min–max. In Theorem 2, is it necessary to use this $R$? Also, please verify whether line 751 is correct.\n5.\tSOC introduces other activation functions such as Householder and GroupSort. Can the paper compare with these? Otherwise, the advantage of β-Abs is unclear.\n6.\tWhat does the dagger ($\\dagger$ in line 351) symbol in Table 1 represent?\n7.\tWhy does the paper use R in Equation (8)? Is this orthogonal mapping essential and trainable?\n8.\tThe periodic polar retraction seems somewhat unusual. The paper claims it helps eliminate errors, but could the authors clarify its specific effect? Why not use other orthogonal projections such as LOT or Cholesky?\n9.\tIt would strengthen the paper if the authors could include AutoAttack results to verify that the certified robustness implementation is correct.\n10.\tRegarding Section 3.1, I believe the following reference may be missing:\nBader, Philipp, Sergio Blanes, and Fernando Casas. \"Computing the matrix exponential with an optimized Taylor polynomial approximation.\" Mathematics 7.12 (2019): 1174.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-organized and clearly written.\n2.\tThe proposed manifold optimization and spatial shift techniques are interesting and technically sound.", "weaknesses": "1.\tSome important baselines are not discussed in the related work section, such as Sandwich and BRONet.\n2.\tThe method integrates several known techniques, making it somewhat difficult to assess the individual effectiveness of each component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9y7um8x1WH", "reviewer": "Reviewer_cWBh", "summary": "LipNeXt introduces a novel 1-Lipschitz architecture that provides efficient, deterministic robustness guarantees for large models. By leveraging manifold optimization and a Spatial Shift Module, it achieves state-of-the-art certified robustness accuracy on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet.", "questions": "* Could the authors provide a formal definition of certified robust accuracy (CRA) and clarify how the corresponding data is generated? As mentioned in line 40, ε represents the radius of a ball in the p-norm. However, the specific value of p does not seem to be provided. Additionally, since the introduction begins by discussing adversarial robustness, I assume that CRA refers to accuracy against adversarial examples. Could the authors clarify which attack methods were used for the assessments?\n\n* I find the role of the β-Abs nonlinearity somewhat unclear. It is introduced between lines 320-357 as a replacement for MinMax or MaxMin. While I do not doubt the correctness of the proof, the connection to the overall proposed method is not immediately clear from a narrative perspective. Additionally, the improvement gains from β-Abs, as shown in Table 7, appear to be relatively modest compared to those in Table 1. Besides, could the authors clarify whether β-Abs could be applied to existing models, and if so, under what conditions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides solid empirical evidence, along with an ablation study, to support the main claims.", "weaknesses": "Table 2 presents results with additional data; however, I noticed that the total number of parameters for the proposed model is 256M, which is significantly larger than the competitors. Could the authors provide results for a smaller model configuration, such as L32W1024?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "24jYW2Vgcf", "reviewer": "Reviewer_ZYaL", "summary": "This paper studies certified robustness in the direction of Lipschitz networks. Algorithmically, it (1) designs an approximate-then-correct optimization algorithm which speeds up the optimization in the orthogonal space, and (2) proposes a convolution-free method via shifting (rolling) data spatially in the orthogonal embedding space. The final method is able to scale to networks with billion parameters and achieves strong performance.", "questions": "1. An orthogonal projection, $R$, is conducted before the spatial shifting. Is this a learned orthogonal matrix or a manually constructed matrix?\n\n2. Is the spatial shifting method meaningful without projecting into an orthogonal space first? The current design arguably introduces much more parameters (the $R$ matrix is a full-dimensional square matrix) than convolutions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method exhibits strong rigor and novelty. Almost all designs are supported by strong theoretical analysis and well-motivated. The final overall performance demonstrates the effectiveness of the general algorithm. Detailed ablation studies are provided in the appendix to demonstrate the effectiveness of individual modules.", "weaknesses": "The overall algorithm seems costly. On the memory side, the optimizer requires a copy of the full parameter, thus doubles the memory cost, which is especially concerning for a model with billion-parameters. On the computation side, the main results on conducted on 8xH100 GPUs, which seems hard to reproduce by academic labs and not scalable to harder tasks. I will not attack the main contribution due to the costs though.\n\nThe parameter efficiency is of question. All comparisons, although meaningful in the story, are conducted with regard to models with far fewer parameters. I will not deny that it is meaningful to further scale and the baselines might suffer from scaling, but the key question is, are those parameters necessary to achieve strong performance; which part of the model consumes the most parameters?\n\nAs a minor comment, the introduction about certified training is misleading. Line 43 introduces randomized smoothing and Lipshitz networks as the two major trends, while other deterministic certified training based on bound propagation is as active, see [1] as an example.\n\n[1] https://arxiv.org/abs/2406.04848", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)