SONIC: Spectral Oriented Neural Invariant Convolutions
Authors: Gijs Joppe Moens
Regina Beets-Tan
Eduardo H P Pooch
Venue: ICLR 2026 Poster (2026)
Keywords: Spectral Neural Networks
Spectral Parameterization
Resolution Invariance
State-Space Models
Spectral Factorization
Convolution Alternatives
Oriented Filters
Global Receptive Fields
Robust Representation Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: SONIC filters combine global receptive fields with directionality, interpretability, and resolution invariance, achieved in a highly parameter-efficient way

Abstract: Convolutional Neural Networks (CNNs) rely on fixed-size kernels scanning local patches, which limits their ability to capture global context or long-range dependencies without very deep architectures. Vision Transformers (ViTs), in turn, provide global connectivity but lack spatial inductive bias, depend on explicit positional encodings, and remain tied to the initial patch size. Bridging these limitations requires a representation that is both structured and global.  We introduce SONIC (Spectral Oriented Neural Invariant Convolutions), a continuous spectral parameterisation that models convolutional operators using a small set of shared, orientation-selective components. These components define smooth responses across the full frequency domain, yielding global receptive fields and filters that adapt naturally across resolutions. Across synthetic benchmarks, large-scale image classification, and 3D medical datasets, SONIC shows improved robustness to geometric transformations, noise, and resolution shifts, and matches or exceeds convolutional, attention-based, and prior spectral architectures with an order of magnitude fewer parameters. These results demonstrate that continuous, orientation-aware spectral parameterisations provide a principled and scalable alternative to conventional spatial and spectral operators.

Published as a conference paper at ICLR 2026
SONIC: SPECTRAL ORIENTED NEURAL INVARIANT CONVOLUTIONS
Gijs Joppe Moens Regina Beets-Tan" Eduardo H. P. Pooch 1.3+ Netherlands Cancer Institute, University of Amsterdam, Maastricht University {g moens, E.beetstan, e e.pais pooch) (rocch]rki....
Local convolution
Self-attention
SONIC (Ours)
ABSTRACT
Convolutional Neural Networks (CNNs) rely fixed-size kernels scanning lo- cal patches, which limits their ability to capture global context or long-range de- pendencies without very deep architectures. Vision Transformers (ViTs), turn. provide global connectivity but lack spatial inductive bias, depend on explicit po- sitional encodings, and remain tied to the initial patch size. Bridging these lim- itations requires a representation that is both structured and global. We intro- duce SONIC (Spectral Oriented Neural Invariant Convolutions). continuous spectral parameterisation that models convolutional operators using small set shared, orientation-selecee responses across the full frequency domain, yielding global receptive fields and filters that adapt naturally across resolutions. Across synthetic benchmarks, large- scale image classification, and 3D) medical datasets, SONIC shows improved bustness to geometric transformations, noise, and resolution shifts, and matches exceeds convolutional, attention-based and prior spectral architectures with order of magnitude fewer parameters. These results demonstrate that continuous, orientation-aware spectral parameterisations provide principled and scalable al- ternative to conventional spatial and spectral operators.
components. These components define smooth
INTRODUCTION
Human visual processing effortlessly recognises objects, interprets motion, and comprehends com- plex scenes across varying orientations, scales, resolutions, and degraded conditions capabiiiies where computer vision methods still fall short. Bridging this gap remains central challenge, driving development of models that more closely approximate the versatility and robustness of human perception.
Convolutional Neural Networks (CNNs) (LeCun et al., 2015) rely on fixed-size kernels scanning local patches. While effective for capturing local features, this design limits long-range dependency modelling without very deep architectures (Luo et al. 2017) and is sensitive to geometric variations such translations, rescalings, rotations, and distortions (Azulay & Weiss 2018). Vision Trans- formers (ViTs) (Dosovitskiy et al. 2021) overcome locality constraints via self-attention over image
"Code available wwtttts://ggiihaa/ com/Gi jaMoens/Soni

---

Published as a conference paper at ICLR 2026
patches, but their quadratic cost in image area poses challenges for high-resolution inputs. More- over, VITs lack spatial inductive biases, requiring explicit positional encodings, and their accuracy- compute trade-off is tied to the chosen patch size. With the proposed method, which enables global receptive fields using significantly fewer parameters, we aim to narrow this gap and move toward resolution-invarian perception, drawing inspiration from the robustmess of human visual processing.
Contribution. In this paper, we introduce theoretically grounded spectral framework for multi- dimensional signals that naturally provides global receptive fields, full convolutional expressiveness, and inherent resolution invariance, offering lightweight yet versatile foundation that can support progress toward more scalable and adaptable vision models. The remainder of this paper is organ- ised as follows. Section 2 introduces the mathematical preliminaries and related work. Section presents the formulation of the SONIC approach together with implementation details. Section reports the experimental results. Section discusses the limitations of the proposed method and outlines directions for future research.
BACKGROUND
Modern vision requires integrating information long spatial ranges, yet standard convolutions remain bounded by local receptive fields. This section reviews operators that achieve global recep- tive fields and develops the spectral framework underpinning our approach.
Large Receptive-Fieb Operators
Resolution-Invarieee Invariant
Desolution-Dependenn
Spectral- Doomann
Spatial-Domain
Continuous
Discrete Grid
Large-kemel CNNs
Structured
BNO 84ND SONIC
Unstructured
Global Filler Spectral
FNO
A-NNst
Vision tansformers
Noc-local networks
Orientation-awaan
Continuous FNO
Low-rank
Figure 1: Taxonomy of large receptive-field operators.
Spatial-domain operators. In the spatial domain, enlarging receptive field requires expanding the discrete kernel support. Large-kernel convolutions (Ding et al. 2022), dilated convolutions (Yu & Koltun 2015), and non-local methods (Wang et al. 2018) each target long-range interactions, but all operate ranges. Self-attention (Chen et al. 2018: Dosovitskiy et al. (021) computes pairwise interactions across all positions, yet scales quadratically with the number of tokens. As resolution increases, both large kernels (O(K2)) and attention become prohibitively costly in high-resolution domains.
fixed sampling grid whose locality assumptions break down over large spatial
Spectral-domain operators. An alternative paradigm achieves global receptive fields by repre- senting operators directly in the frequency domain (Rippel et al. 2015). This approach exploits the fact that every linear, shift-invariant operator on E2D) fully characterised by its frequency response, enabling information to propagate globally through a single frequency- -wie multiplication.
Let D N and consider vector-valued signals
x : + Cin. y: y:R + Coont
For sufficiently regular scalar function R:RD + C, the Fourier transform is
Folf(ww) = [ f(x) f(x)e-iwx
-iw iix dx,
W E IR
(1)
JRD
and extends component-win to vector-valued functions. linear, shift-invariant operator acting on has a convolution representation:
v(x) = /
k(x - 2) xx()) dz,
(2)
RD
2

---

Published as a conference paper at ICLR 2026
where ls IR. - (CouexCu, The Convolution Theorem gives
FDl k(w) [))
w WERP,
where k(w) is the Fourier transform of a spatial kernel in the classical convolution setting. In spectral neural methods, however, do not constrain the operator arise from any finite-support spatial kernel. Instead, we define the spectral kernel directly by
k(w) H(w).
(4)
where H (w) the learnable frequency response of the operator. This viewpoint treats H as the primary paramctrisation, enabling general global and assolution-invaiian operators beyond those that correspond discrete spatial kernels. In practice, images are sampled on a discrete grid. Let the spatial domain be discretised using N1 N1..... ND samples along each axis, with pixel spacings A1...., Ap. The corresponding DFT frequeney sets are given by
kid
]]. d= D. Na 2 1
2 2nT
No
(5)
The full frequency grid is the Cartesian product Ω = 9t ... x Qd, containing N = Ni ... ND discrete frequencies. The DFT samples 5 defined frequencies Wn Ω.
Resolution invariance. We formalise resolution invariance by defining the operator via contin- uous spectral symbol that is independent of the sampling grid. Let
Ho TR +
Coue XCi
(6)
be a continuous function parameterised by Given a discretisation (N, with Fourier grid ON the discretised operator is obtained via sampling:
T (w,)) (w... (WN))
E ENNA.
term the operator resolution-invarian if depends only on the underlying physics of the layer, not the discretisation (N, A). Changing resolution then simply corresponds to resampling the same continuous function He on a new grid. GFNet (Rao et al. 2021 and FNO Li al. 2021) parameterise f directly on the discrete FFT grid: GFNet learns complex mask size N, and FNO learns fixed number of low-frequency coefficients. Since these coefficients correspond to specific frequency indices, changing resolution alters the operator itself. Thus, such models do not define a true inoolutton-ivvaiiann convolution operator.
Continuous spectral operators. A principled alternative defines the operator in the continu- Fourier domain and evaluates it whatever discrete grid the data provides, yielding truly resolution-invarian parameterisation. Two families appear in the literature:
Unstructured continuous operators learn / (w) general continuous function, c.g. via an MLP (Grabinski et al. 2024 Kabri et al., 2023). ). These are maximally flexible but typi- cally isotropic and parameter-innfffiienn as they offer little inductive bias for orientation or cross-channel structure.
Structured continuous operators impose additional structure through basis expansions or separability assumptions. Examples include SNO (Fanaskov & Oseledets 2024) and S4ND (Nguyen et al.) 2022), whose axis-aligned constructions couple nearby frequencies and improve parameter efficiency. However, their separability limits the capture of oriented or anisotropic patterns along general directions in frequency space. We provide the frequency- domain form of S4ND in Appendix C.2
Since natural images contain oriented structures (edges, textures, oscillations) that correspond to directional features in frequency space, axis-aligned or tensor-product constructions are insufficient. We therefore introduce SONIC: Spectral Oriented Neural Invariant Convolutions, Structured Con- tinuous Spectral Operator that parameterises Ho (w) as superposition of directional modes, yield- ing resolution-invariann parameter-efficien filters inherently adapted to the anisotropic structure of natural signals.
3

---

Published as a conference paper at ICLR 2026
3 METHOD
Overview, Starting from linear time-invariant (LTT) systems, we derive compact spectral repre- sentation for D-dimensional signals that models shift-invariant operators through shared low-rank structure: oriented spectral transfer functions are applied at each frequency and mixed across chan- nels by learned matrices B and C.
3.1 FORMULATION
To make the connection to state-space models precise, consider the continuous-time LTI system:
(t) Ax(t) + Bu(t),
yt) Cx(t),
(8)
with zero initial condition. Its impulse response is obtained by setting u(I) - 8(1):
((t) CeAB B,
t t O.
The output equals the convolution the input with the impulse response:
(K (() Jo Ce B u(l r) dr.
(10)
Taking the Laplace transform of the impulse response (derivations provided in Appendix C
(s) [(K())}(s) C(sl - A)
(11)
We use this resolvent form as modelling template: replacing the scalar Laplace variable s with the D-dimensional spatial frequency yields an analytic spectral parameterisation that inherits the smooth behaviour of resolvent filters.
Let E X.-N our method is the transfer function (ww,,, which defines the frequency response of single mode. For each mode m I,..., M we set
XND and y E XxXXNxx-.MM, with DFT 5 FD [a] and y
FD' Central to
Tm (w) 18m (w-Um) am + Tm [I - Um U J
(12)
where each mode is parameterised by: the unit orientation Vm RRD;; scale 8m > (spectral selectivity); damping Re(am) < O and oscillation Im(am); and transverse penalty 2 O (off- axis decay). The denominator replicates the resolvent structure an LTI system, substituting the Laplace variable with the oriented frequency component sm (w-Um) and adding transverse decay for anisotropic filtering. Each mode additionally carries learnable Butterworth bandwidth by to suppress aliasing near the Nyquist limit (Appendix Ap.
Rather than learning unconstrained response (ww) for every frequency, SONIC factorises the spectral operator through M shared modes with entrywise form:
M
Hk,o(w) ΣCem Ckm Tm (w) Bmc-
(13)
where B CMC and C CKXM, Given this factorised spectral response, the frequency-wise filtering applied to the input DFT
(w) Σ (w) To(w,,
A == K, w E a
(14)
where (w) is the frequency response of the (c-+k) k) channel filter. This low-rank decomposition enables expressive yet parameter-efficien filtering. The spatial output is added learnable skip projection and passed through pointwise nonlinearity:
( = ( + W²x
(15)
Multiple SONIC blocks can be stacked for desth-wise expressivity. Although SONIC is not state- space model, restricting its orientations to coordinate axes recovers the multidimensional SSM form (Appendix 2

---

Published as a conference paper at ICLR 2026
3.2 INTUITION
Each SONIC mode a compact, analytic spectral filter with a few interpretable parameters (illus- trated in Figure ??). A unit vector vm the mode's preferred direction; any decomposes into along- and across-orientation components:
Wljm- == w Dy
Wim == (I Unn 2m w.
The mode emphasises slowly varying content along Om and suppresses energy across it, producing spatially elongated kernel sensitive to oriented structure. The scale 8m controls spectral selectivity: small values give broad, smoothing response while large values narrow the passhand fine along- axis detail. The complex coefficient Oyn governs dynamics-its real part controls damping (stability) and its imaginary part introduces oscillatory structure, letting the mode capture repetitive patterns. The transverse penalty 2 0 suppresses off-axis leakage, sharpening directional selectivity. The modes form a shared dictionary of directional behaviours; mixing matrices B and C let each channel pair draw a unique superposition, inducing parameter sharing across both frequencies and channels.
SONIC Block
Linear
(1)
SONIC
Norm
ReL.U
(f++1)
x
T
i
T
(())
(w) A SONIC block: FFT, spectral modulation by fi, inverse FFT, normalization, and residual fusion.
Spectral Mixing
CM XC
C A
Ti
T1
H1
T2
BC
ÜlK
T
T's
Ts
Mode decomposition
(b) Low-rank spectral mixing: fee,o(w) = Em. Okm Tea (w) Brcs with and C mixing channels into and out of A shared modes.
Mode Parametrisation
Base kemee
Orientation
Scale 81
Real coeff.
Imag, coelf.
Transverse
1 Qq.
1 11
(c) Each mode Tm. is controlled by orientation, scale, complex coefticients, and transverse decay. Shown: parameter sweep for Ti.
Figure 2: SONIC overview: (a) Residual Block, (b) Spectral Mixing, and (c) Learnable Spectral Modes.

---

Published as a conference paper at ICLR 2026
Resolution Invariance Crucially, all of these filters are parameterized directly in the continuous spectral domain. This means their definition does not depend on the size or sampling rate of the image: defining filters continuous functions of decouples them from any particular grid size or sampling rate; the same response formula is evaluated on whatever DFT grid the data induces, yielding resolution-invarian filter. This distinguishes
approach from spatial-domain kernels, whose size and shape are tied to fixed grid. We made some minor adjustments to ensure resolu- tion invariance: To make the directional parameters resolution invariant, we express directions in physical units and normalize:
im
Da = diag( A AD),
DI' Um
(16)
This formulation provides flexibility with respect to resolution, which can also be exploited dur- ing training as proposed in Nguuyen et al. (2022). Beyond training efficiency, resolution-awan pa- rameterisation is particularly relevant in domains where resolution dependence is intrinsic, such as medical imaging, remote sensing, and microscopy.
Computation.
The number learnable real scalars is:
2KM + 2MC + (A+D)M++ +
Chh C
B
ar",aim, 9, EBD
Crucially, this count scales with M(C+K) and is independent of spatial resolution. In contrast, methods that parameterise filters on the discrete FFT grid (e.g. GFNet) require O(N₂ x ... x Np) parameters per layer, which grows rapidly with resolution and dimensionality. 'Tii distinction plains why SONIC's parameter ad advantage is modest on low-resolution 2D benchmarks but becomes pronounced 3D volumetric settings.
The FFT has per-transform cost ON log N) for single (complex) channel. The spectral for- ward pass performs one DFT per input channel and one inverse DFT per output channel, plus ((M(CKK) complex multiplications per frequency. The forward pass consists of one DFT per input channel and one inverse DFT per output channel, with cost
OCCN log N) and O(KN Nog N),
wise where N Th Na is the total number of spatial points. In addition, frequency wise multiplica- tions incur cost of
{M(C+K)N)
since each of the M modes couples inputs and outputs across all frequencies. The total complexity is therefore
((C+K)) N + M(C+K)N).
For comparison, a standard spatial convolution with kernel size d x d has cost
OCCKNd))
SONIC is thus particularly attractive for large receptive fields (where d is large even global), since the cost remains manageable and the parameter count remains compact.
EMPIRICAL VALIDATION
SynthShape. We introduce SynthShape, 64x64 synthetic segmentation benchmark that mea- sures robustness to rescaling, rotation, translation, distortion, and Gaussian noise under 5-fold cross- validation. We also introduce HalliGalli, a spatial-reasoning task requiring long-range dependency modelling: central patch must be classified according to whether exactly two matching shapes appear in the four distant corners; the centre carries no class signal. Purely local models fail since no local receptive field can capture the relevant structure. SONIC solves HalliGalli and remains bust under noise (a=0.1), demonstrating that its oriented spectral modes effectively capture global structure without accumulating noise. Implementation details are in Appendix A.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of ConvNet, VIT, S4ND, NIFF, GFNet, and SonieNet performance on SynthShape under geometric variations (left), and qualitative examples from SynthShape and HalliGalli-SRT (right).
Experiment
ConvNet
S4ND
SonicNet
Parameter count (M)
0.153
0.471
0.186
0.042
1,218
0.073
GMACs
0.156
0.012
0.023
0.041
0.139
0.006
SynthShape
Distortion
2.0
0.95
0.83
0.84
0.95
0.55
0.95
4.0
0.94
0.81
0.83
0.95
0.54
0.94
6.0
0.93
0.81
0.82
0.94
0.53
0.94
Gaussian (c))
0.67
X
0.3
Rescaling
0.65
0.99
0.82
0.93
1.00
0.79
0.98
1.50
0.59
0.64
0.32
0.33
0.93
HalliGalli
Rotation (*)
15
0.71
0.64
0.68
0.76
0.42
0.80
30
0.28
0.33
0.50
0.30
0.29
0.24
0.27
0.31
0.44
0.29
0.28
0.23
Translation (%)
10
0.92
0.81
0.76
0.92
0.48
0.93
20
0.96
0.82
0.76
0.97
0.67
0.95
30
0.92
0.80
0.77
0.92
0.52
0.92
Combined
10
0.80
0.74
0.47
0.81
0.35
0.87
20
0.62
0.60
0.30
0.63
0.30
0.78
30
0.43
0.44
0.24
0.43
0.26
0.55
HalliGalli
0.42
0.33
0.62
1.00
0,71
1,00
HalliGalli 0.1)
0.33
0.33
0.49
0.56
0.37
0.86
Validation accuracy on the training task.
3D medical image segmentation. Following Isensee et al. (2024), we benchmark on KiTS (kid- ney/tumour) and ACDC (cardiac) with identical 5-fold splits, preprocessing, augmentations, and postprocessing, so that observed differences are attributable solely to the network backbone. Table shows that SonicNet achieves segmentation quality on par with nnU-Net ResEnc (DSC 88.55 88.98 on KiTS; 92.02 vs. 91.40 on ACDC), while using only 2.59 M parameters-an order of magni- tude fewer than any competing method. The large parameter gap reflects both SONIC'S dimension- independent mode count and the directional regularity of anatomical structures (organ boundaries, vessels), which orientation-avee -aaare spectral modes capture efficiently. SonicNet also surpasses all lit- erature Transformer-basse
architectures (SwinUNETRV2, nnFormer, Co'Tr) and the Mamba-base U-Mamba Bot in both DSC and NSD, demonstrating that continuous spectral operators transfer effectively to volumetric medical data.
Table 3D medical image segmentation results (5-fold CV; mean across folds). Columns report DSC and NSD (at 2 mm). "RT" is runtime (hours) and "VRAM" is peak memory (GB). Literature results are shown in gray as reported by Isensee et 2024).
KITS
ACDC
Method
DSC
NSD
DSC
NSD
Params (M)
RT (h)
VRAM (GB)
nnU-Net ResEne L
88.98
85.74
91.40
96.21
31.12
34
23
SonicNet (Ours)
88.55
81.19
92.02
96.07
2.59
67
61.37
nnU-Nei ResEne I.
88.17
85.93
91.69
95.11
31.12
36
36.60
MedNeXt k5
87.74
85.67
92.62
96.09
55.00
233
18.00
STU-Net
85.84
83.02
89.34
95.12
440.30
51
26.50
SwinUNETRV2
84.14
80.11
86.24
95.15
72.80
15
13.40
nnFormer
75.85
69.43
81.55
95.83
150.0
5.70
CoTr
84.59
80.92
88.02
93.74
41.27
18
8.20
U-Mamba Bot
86.22
83.27
89.13
95.40
64.00
24
12.40

---

Published as a conference paper at ICLR 2026
External validation. To assess generalisability beyond the training cohort, evaluate SONIC trained on PI-CAI (Saha et al. 2022) against nnU-Net on two held-out external datasets: Prostate (Adams et al. 2022) and PROMIS (Ahmed et al.) 2017). As shown in Table EL SonicNet im- proves detection across nearly every metric on both datasets while using approximately 12x fewer parameters than nnU-Net. The gains especially pronounced the challenging PROMIS co- hort, suggesting that SONIC's compact spectral representation generalises more robustly to out-of- distribution data. Qualitative results in Appendix B
Table 3: External validation performance on Prostate and PROMIS. SonicNet achieves im-
proved detection performance with substantially fewer parameters.
Metric
nnU-Net
SonicNet
TRAINABLE PARAMETERS (M/MB)
31.20/342.0
2.59/28.4*
AUROC
0.814
0.841
Prostate 158
AP
0.533
0.548
F1 Score
0.632
0.649
Sensitivity
0.475
0.495
Precision
0.941
0.943
TP/FP/EN (%)
0.30/0.02/0.34
0.32/0.02/0.32
AUROC
0.646
0.687
PROMIS
AP
0.195
0.258
Fl Score
0.185
0.223
Sensitivity
0.103
0.127
Precision
0.912
0.907
TP/EP/EN (%)
0.05/0.01/0.47
0.07/0.01/0447
Approximately 12x fewer parameters than nnU-Net.
ImageNet classification. Table evaluates SONIC on ImageNet-IK in two settings. First, as drop-in replacement for 3 convolutions in ResNet-50, where it outperforms GFNet and S4ND un- der identical conditions. Second, as SonieNet 4-stage hierarchical backbone designed natively around spectral operators, where SONIC handles spatial mixing and 1 Ix convolutions handle chan- nel mixing. This avoids bottleneck projections and batch normalisation placement inherited from optimising convolution cost. SonicNet-S achieves 78.8% Top-1 accuracy with only 1.9 GFLOPs- less than half the compute of ResNet-50 (4.1 GFLOPs) while surpassing its accuracy by 2.7 points. The parameter gap relative medical imaging narrower because natural images lack consis- tent directional structure. Crucially, SonicNet exhibits smaller relative degradation under resolution shifts (Figure 3. confirming the resolution invariance from Section 3 Note that DeiT-S cannot be evaluated at non-training resolutions due to its fixed positional embeddings.
Table 4: ImageNet--IK classification. Published baselines, ResNet-50 with drop-in spectral operators, and SonicNet variants. All evaluated at 224 224.
Relative Top-1 degradation %%)
Complexity
Accuracy (%)
10
Model
Params (M)
GFLOPs
Top-1
Top-5
ResNet-50
ResNet-50
25.6
4.1
76.1
92.9
20
DeiT-S/16f
22.1
4.6
79.9
95.0
GFNet-S
25.0
4.5
80.0
94.9
Global Filter
NIFF!
18.6
79.7
94.8
30
SonieNet-S
ResNet-50 + GFNet
15.7
4.6
75.1
ResNet-50 S4ND
16.7
4.6
57.5
80.3
128
160
192
224
ResNet-50 SONIC
15.2
3.5
76.7
92.9
Input resolution (pixels)
SonicNet-T (Ours)
5.7
0.9
75.8
SonicNet-S (Ours)
15.0
1.9
78.8
94.5
Figure 3: Relative performance degradation under resolution changes on ImageNet.
"Published reulls
8

---

Published as a conference paper at ICLR 2026
Learned filter structure. Figure 4 (left) shows randomly sampled spatial kernels obtained by tranforrnan the learned spectral filters from each stage of the ImageNet backbone. Early stages learn compact, oriented filters, capturing edges and texture at diverse orientations. With increasing depth, filters progressively broaden: Stage 2 exhibits bipolar lobes sensitive to medium- range structure, Stage 3 develops multi-scale directionally selective patterns, and by Stage 4 fil- ters span nearly the full spatial extent. This hierarchy emerges without supervision on filter shape, confirming that the continuous spectral parameterisation provides an effective inductive bias for multi-scale directional representations. To quantify effective spatial reach we report the normalised 95%-energy radius Rgs = r9s/(5 ( H2 - + W?). where r95 is the smallest disk radius r satisfying JIxIS> S |x() dx 2 0.95 f [ dx, h F is the spatial kernel corresponding to the learned spectral filter, and H x are the spatial dimensions of the feature map. Thus Ros = means the kernel's energy spans the full spatial extent (Figure right).
Stage 1
0.8
Stagu 2
Stage
0.2
Stage
Figure 4: (Left) Learned SONIC spatial kernels (4 per stage, ImageNet backbone) via inverse FFT of the spectral filters. Early stages learn compact Gabor-like filters; deeper stages develop broad, globally reaching patterns. (Right) Normalised Ros reach across all 128 modes per stage. Median reach grows monotonically with depth, approaching full spatial extent (dashed line) by Stage 4.
Compute and memory overhead. Figure 5 compares a single SONIC block against 3x3 con- volutional block and VIT self-attention block (patch size 4x4, matching the spatial resolution of feature maps). At 224 x 224, SONIC is 1.23x slower and uses 1.18x more memory than convo- lution while self-attention grows quadratically. At higher resolutions the gap narrows further: by 512x 512, SONIC is only 1.09x slower than convolution. The overhead remains constant relative feature map size. Runtime and memory scale linearly in both C' and M (Figure 7 Appendix B). making SONIC practical for high-resolution inputs where global context is essential and one does want pay the quadratic cost of attention.
Runtime per block (ms)
10a
VIT
Peak GPU memory (MB)
Cony block
SONIC block
10
10a
VIT block
10a
10
Cony block
SONIC
320
384
448
512
330
384
448
512
Resolution H
W (pixels)
Resolution H W (pixels)
Figure 5: (Left) Runtime per block (log scale) and (Right) peak GPU memory (log scale) for VIT. convolutional, and SONIC blocks across spatial resolutions. SONIC scales favourably with resolu- tion due to its O(N log N) FFT complexity, whereas self-attention grows quadratically.

---

Published as a conference paper at ICLR 2026
5 DISCUSSION
We introduced SONIC, spectral factorisation framework that serves as theoretically grounded, parameter-effcien alternative spatial convolution blocks. By employing low-rank, orientation- aware operators in the frequency domain, SONIC provides global receptive fields with principled inductive bias for long-range, structured interactions. Empirically. SONIC demonstrated superior SynthShape, solved strict long-range dependencies in
robustness to geometric transformations HalliGalli within a single block, and matched or exceeded sate-oofthheaas 3D medical segmentation (KiTS, ACDC) with 10% of the parameters of nnU-Net and McdNeXt.
Important limitations remain. Nonlinearities must be applied in the spatial domain, forcing repeated FFT/IFFT operations shared by most spectral architectures. We also observed occasional initiali- sation instabilities when identical spatial dimensions correspond to different physical scales across datasets, issue that warrants more robust initialisation scheme. Finally, the global nature of frequency-domain representations can limit the capture of very fine local structure, suggesting that hybrid spectral-spatial architectures merit further investigation. In summary, SONIC offers new building block that complements existing paradigms through its long-range receptive field, param- eter efficiency, and eeeettaton-awareness while future work should focus on improving computa- tional efficiency and exploring hybrid designs.
ACKNOWLEDGMENTS
The authors would like to acknowledge the Research High Performance Computing (RHPC) facility of the Netherlands Cancer Institute (NKI). this paper, we used large language models to refine wording and improve the clarity of information transfer. All conceptual ideas, discussion of related work, and factual content were developed manually by the authors; the models were employed solely for assistance with presentation.
REFERENCES
Lisa C. Adams, Marcus R. Makowski, Günther Engel, Maximilian Rattunde, Felix Busch, Patrick Asbach, Stefan M. Niehues, Shankeeth Vinayahalingam, Bram van Ginneken, Geert Litjens, and Keno K. Bressem. Prostate158 an axpert-annotated 3t mri dataset and algorithm for prostate cancer detection. Computers in Biology and Medicine, 148: 2022. ISSN 0010-4825. doi: https://do.orr//00 1016/0 cotp.j.oooooommeee2222... 105817. URL htttt:://// //www. sci sciencedi rect www.oo//ccccooo sci wwc.iinnee/aco atte//rtiicii httpeeceeeaecacceee//////eaa//// 002553000008222220000
Hashim U Ahmed, Ahmed El-Shater Bosaily, Louise C Brown, Rhian Gabe, Richard Kaplan, Mahesh K Parmar, Yolanda Collaco-Moraes, Katie Ward, Richard G Hindley, Alex Freeman, Alex P Kirkham, Robert Oldroyd, Chris Parker, and Mark Emberton. Diagnostic accuracy of multi-parametrio mri and trus biopsy in prostate cancer (promis): paired validating con- 822, 2017. ISSN 0140-6736. doi: https:
firmatory study. The Lancet, 10tp:o/rr/o...010000004000066663 sctianee/arrticll//pi///00016612
URL hhtpp:: http://wwwwwceeeeeereeee....ee.e com/
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? CoRR, abs/1 105117775. 2018. URL http:://axxi..rrrreeeece// org/ab///rr//b//////////
18C5.
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. ae-nets: Double attention networks, 2018. http:/ttt////aaatee /ttp///rwww...r////////11/11
Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scal- ing up your kernels to 31x31: Revisiting large kernel design cnns, 2022. URL https: // //aaxii..rg/abs//////.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at
raaii scale, 2021. URL https: www.wwiww....aaaaaaaaa aba/////b/////// /201 0. 11929
10

---

Published as a conference paper at ICLR 2026
V. Fanaskov and L. Oseledets. Spectral neural operators, 2024. URL https://arxiv.cr abss 2tt//2205.11533
Julia Grabinski, Janis Keuper, and Margret Keuper. As large as it gets: Learning infinitely large filters via neural implicit functions in the fourier domain, 2024. BRRLhtts://arxiv.orr
abs/ abs/2301 2307. 10001
Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier- Hein, and Paul F. Jaeger. nnu-net revisited: call for rigorous validation in 3d medical image segmentation, 2024. URL /ttt::://aaxxivoo///aaaaa///
09556
Samira Kabri, Tim Roith, Daniel Tenbrinck, and Martin Burger. Resolution-invariant image clas- sification based on fourier neural operators, 2023. URL /KK_itttsas//axxxx.rr//////////p
01227
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436, 2015.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, 2021.
org abs http://www...o//a///.10.00...0 08895
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard S. Zemel. Understanding the effective receptive field in deep convolutional neural networks. CoRR, abs/1701.04128, 2017. URL http://
arxiv.org/abs//1700 11tp://aaas/111..01111 04128
Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen A. Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals using state spaces, 2022. URL https: http://axivvv.ooo .rg. abs/ 22122221122 102.00000 06583
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification, 2021. ht//:/ttsss///aav...rr//////////
00645
Oren Rippel, Jasper Snoek, and Ryan Adams. Spectral representations for convolutional neural networks, 2015. URL/htps:: //ats::::://arivv///// org www./r/.////oooooo./o.oo//..eera wttpaaawwwwvv.aa//a/aaaaaaa////a
03767
Anindo Saha, Jasper Jonathan Twilt, Joeran Sander Bosma, Bram van Ginneken, Derya Yakar, Mattijs Elschot, Jeroen Veltman, Jurgen Fütterer, Maarten de Rooij, and Henkjan Huisman. The pi-cai challenge: Public training and development dataset, June 2022. URL https:
(1t.:i..rg/10.522//eeeee... 2021///eoooo.... 5624726
Xiaolong Wang. Ross Girshick, Abhinav Gupta, and Kaiming Hc. Non-local neural networks, 2018.
URL / 1ttp://...or//aa//111...7...
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:/5/1.07/222 2015.

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
We constrain 8m > O, Tmn > O, dom > O, and enforce Re(am) < 0 so that the spatial response
function decays rather than grows. Direction vectors 4m are initialised evenly spaced in (0, x) for 2-D, or via a Fibonacci lattice on the sphere for 3-D, with optional jitter.
All parameters are learned end-to-end by backpropagation. A convenient reparameterisation that enforces the constraints is:
8m exp(Om), Tm =
am = softllu((mm) O' Im = -m..
with free variables Jm, Fiy Om, pm E R. Direction vectors are parameterised via spherical angles Pm) converted to unit vectors. The mixing matrices B and C are complex-valued and learned without constraints.
Implementation notes. (i) We standardize each input sample to zero mean and unit variance over all channels and spatial dimensions, with a small noise floor for numerical stability. (ii) We use real-FFT (FFFTT/rFFT) along the spatial dimensions; consequently we enforce Hermitian con- sistency by zeroing the imaginary part at DC and, for even-sized dimensions, at the Nyquist bin. (iii) Direction vectors are rescaled by D and renormalised to unit length before use, ensuring invariance to pixel spacing. (iv) Optional mode dropout applied to the filtered modes as a regu- lariser. (v) Transfer function magnitudes are soft-clamped via tanh saturation to prevent resonance peaks from destabilising training.
Butterworth bandlimiting. To suppress aliasing near the Nyquist limit, we multiply the de- nominator of Tm by a Butterworth factor (1 + ([| ([]=)///wmmm))) where We,m bm Whyg and bm = =obb) E (0, 1) is a learnable per-mode bandwidth. We use order n=4 in all experiments. This allows each mode to adapt how much of the spectrum it utilises.
A.1 SYNTHSHAPE
The dataset consists of a random number of geometric primitives (circle, square, triangle, cross, star) random positions and scales within the image, while preventing overlaps through collision checks. Each object is assigned a randomly perturbed base colour, ensuring that models cannot exploit a trivial mapping between RGB values and semantic classes. The ground-truth segmentation mask assigns a unique class label to each shape type, with background indexed as class 0.
Models. All models an embedding width of 1=
ConvNet: A lightweight stack of convolutional layers (default L = 4), each followed by group normalisation and GELU activations. A 1 x 1 convolution projects the final feature map to the number of classes. The patch size set to to give the model a fair opportunity to capture broader context, rather than learning solely from small local receptive fields.
VIT: A Vision Transformer consisting of a patch embedding layer, sinusoidal positional encodings (interpolated if image resolution differs), and stack of transformer blocks with multi-head self-attention and MLP layers. The output features are reshaped and upsampled to the original spatial resolution, followed by 1 convolution for classification.
For SonicNet we use a depth of 4 stacked SonicBlocks, each consisting of GroupNorm, GELU, and residual spectral convolutional mapping. The final stage applies GroupNorm, GELU, and a 3x3 convolution to project features to class logits.
SonicNet:
GFNet: Each block replaces local convolutions by learned complex-vale valued mask applied in the Fourier domain. The overall encoder head structure is kept identical to the ConvNet.
NIFF:
Each block learns continuous frequency response via small MLP that maps frequency coordinates to complex filter values. These filters are applied depthwise in the Fourier domain and wrapped in the same normalisation, residual, and head structure as ConvNet.
S4ND: A state-space baseline where the convolutional backbone is replaced by stacked S4ND layers operating directly the H x W grid. Each block applies a 2D structured state-space update to the feature map and is embedded in the same residual and segmenta- tion head pattern the other models.
12

---

Published as a conference paper at ICLR 2026
Training. All models were trained using the AdamW optimizer with learning rate 10 and weight decay 10 +, for 1000 epochs and batch size 32. A one-cycle learning rate schedule was applied. To liverse-frequene class weights were computed dynamically from
account for class imbalance, large synthetic batch and used the cross-entropy loss. The final training objective combined cross-entropy with the multi-class Dice loss in equal weighting.
Evaluation. Model robustness was assessed by applying five geometric transformations at infer- ence: rescaling, rotation, translation, distortion, and Gaussian noise. Each transformation was ap- plied with three levels of severity. Rescaling resized the full image before resampling it back to 64 64, introducing interpolation artefacts. Translation shifted the imput by a fixed percentage of image width/height, potentially moving parts of objects out of frame. Distortion was implemented via bicubic upsampling of a low-resolution displacement field. Rotation was performed around the image centre, and Gaussian noise was added per pixel channel.
A.2 IMAGENET
We evaluate two settings on ImageNet-IKK
Drop-in replacement (ResNet-50 + SONIC), We take a standard ResNet-50 and replace each 3x3 convolution in the bottleneck blocks with SONIC layer, keeping the 1x1 projections and skip connections unchanged. This isolates the effect of the spectral operator within an existing convolutional template.
Native backbone (SonicNet). We design a 4-stage hierarchical backbone natively around spectral operators. Each stage consists of SonicBlocks: GroupNorm + SONIC (spatial mixing) + Lay- erScale + residual, followed by an optional I 1x] 1 conv MLP for channel mixing. We provide two contigurations: SonicNet-T (tiny: depths [2, 2, 6, 2). dims [64, 128, 256, 512], ~5.7M params) and SonicNet-S (small: depths [3, 3, 9, 3), same dims, ~ 155 params).
Both settings use Adam W with cosine learning rate decay. Full architectural details and the PyTorch implementation are provided in the repository.
A.3 MEDICAL IMAGING BENCHMARK
Setup. Following the recommendations of Isensee et al. (2024), we minimize confounding factors and keep the experiment as plain possible. We retain the baseline nnU-Net preprocessing and postprocessing and change only the network backbone: the original U-Net is replaced by a stack of SONIC Blocks ("SonicNet").. The first block lifts the input from C' to K channels; the remaining
D 1 blocks keep K channels. We apply GroupNorm and GELU before a final 3x3 convolution produce Plllassss output channels. For this experiment,
used four stacked SonieBlocks (i.e., depth of 4). We employed stochastic gradient descent with an initial learning rate of 10-2 and a weight decay of 10-5, Training was performed with mini-batch size of two for total of 1000 epochs, each consisting of 250 iterations. For inference, we used the checkpoint corresponding to the highest validation performance during training.
A.4 QUALITATIVE COMPARISON OF THE EXTERNAL VALIDATION
Ground Figure 6: Qualitative comparison prostate cancer detection methods. The figure shows representative cases from the Prostate 158 (left) and PROMIS (right) datasets, with ground truth lesions (red) and model predictions (cyan) overlaid on T2-weighted MRI slices (confidence 2 0.5).
13

---

Published as a conference paper at ICLR 2026
B PRACTICAL IMPLEMENTATION
Role of and M. The parameters and M play complementary roles in shaping the behaviour of SONIC block. The number of modes M determines the spectral diversity of the operator; in contrast, the channel width K controls the capacity with which these shared modes are mixed across feature channels. The ratio between K and M therefore reflects the balance between channel-mixing capacity and spectral richness. Understanding this trade-off helps guide architectural choices across different model sizes.
Practical scalability of SONIC block. To validate that the SONIC block exhibits the intended linear scaling behavior, we empirically benchmark its runtime and memory usage across a range of channel dimensions and mode counts. We confirm this behavior by measuring wall-clock runtime and peak memory under controlled synthetic settings, sweeping and M independently while keep- ing spatial resolution fixed. Across all tested configurations, runtime increases straight line with respect both variables, and memory usage follows the same linear trend.
Effect of SONIC hyperparameters on runtime and memory Runtime vs C (M-12, H- 2224)
Runtime vs M (C=64, H H2W=
(a)
(b)
Runtime (ms)
70
Runtime (ms)
52
60
50
40
48
100
120
10
20
30
Channels
Modes M
Memory vs (M-12,
Memory vs M (C-64, H-W-224)
Peak memory (MB)
15000
(c)
Peak memory (MB)
8000
(d)
12500
7800
10000
7600
7500
7400
5000
40
60
80
100
120
10
20
30
Channels C
Modes M
Figure 7: Runtime and memory SONIC when varying channels C and modes at fixed resolu- tion.
C SUPPORTING PROOFS
C.1 CONVOLUTION THEOREM FOR THE D-DIMENSIONAL FOURIER TRANSFORM
Let the convolution of two functions on RD be defined by
( U g))) [ f(T))((x -
x x P.
Then, for w E RRD the D-dimensional Fourier transform satisfies
F(f * [(+g)w)= = fo. (1 - T) dr
iw.x dx
-fo JRD f(T) - dudT
-(1pr f(T) ((w) iwt dr )r) g(u) iwu du
14

---

Published as a conference paper at ICLR 2026
Hence,
C.2 CONNECTION TO =
* -
KERNELS
Consider the linear time-invariane state-space model
z(t) Ax(t) - Bu(t),
s(t) Ca(t),
(17)
with zzt) C utt) E C y(t) E CP, and system matrices A E Caxn, B E Caxm C E CPx" Assume a zero initial state x=(0 = and a strictly proper output.
The corresponding impulse response (or kernel) is
X((I)
Ce* B,
I 120 0.
(18)
By definition, the transfer function is the Laplace transform of the impulse response:
H(s) Lo- e KC(t) dt Jo e CeA dt.
(19)
Pulling out C' and B gives
H(s)= - (f,2 c(A-sT)e dt B.
(20)
For Re(s) sufficiently large, the integral converges to
J c(A- dl (sl - A)
(21)
Hence the transfer function is
H(s) = C(sl - A)
(22)
We show that our general Fourier domain formulation reduces to the Laplace resolvent parameterisation of S4ND when orientations are restricted to the coordinate axes.
SONIC with Restricted Modes.
Recall frequency response factorisation
M Ha. (w) = E
Ckm Tm (w) Buos
(23)
with mode response
T (w) ism (w (w-Um. Om + Ti (1 Um Um ww]
(24)
Suppose V... ed, the d-th standard basis vector. Then
(I - Um wm = Σ wyet
wmmm = Wd,
jzd
so that
[1(1 - Um w| = Σ w/
fok
In this case,
(w) = ism We am + Tm Ejea
(25)
We discard the transverse penalty Tm O, then
1
1
T,, (wa) is am
Sm
iw'
8m.
8m
where the absorption is into the learned parameters (am/ 8m / in A, and B or C absorb 1/8m). Thus
(wa) = [C (iwrl -)- ke H(s) = C(sl A)-
15


Reviews:
[{"review_id": "cP5rAJ65cS", "reviewer": "Reviewer_KhJS", "summary": "This paper introduces SONIC (Spectral Oriented Neural Invariant Convolutions), a method that learns compact, directional filters directly in the Fourier domain, being different from CNNs and ViTs. SONIC factorizes multi-channel frequency responses using a small set of shared, oriented components, which are mixed across channels via learned matrices. This design provides a global receptive field, is inherently resolution-invariant, and maintains high parameter efficiency. Empirical validation on synthetic and medical imaging benchmarks demonstrates that SONIC matches or exceeds the accuracy of established models like nnU-Net while using fewer parameters.", "questions": "As mentioned in the \"weaknesses\" section, the paper can be further improved in several aspects. My main concerns are the comparisons with the existing solutions in the frequency domain and the relatively weak experimental study.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper overall is easy to follow.", "weaknesses": "- My main concern is the unclear comparisons with the existing solutions. In Section 2 (line 149), the authors mentioned some limitations of previous solutions like GFNet and FNO. However, there is no further study to show how the proposed method overcomes these limitations and why these limitations are important. Besides, there is also no direct comparison with these methods in the experiments. For example, there are two limitations of GFNet mentioned: \"the FFT grid is tied to the input resolution, and the number of learnable parameters scales with the discretisation of the frequency spectrum\". But as shown in Fig. 6 of GFNet, the model can be directly adapted to other resolutions with finetuning and works well on detection tasks with images of varying resolutions. The parameters involved in frequency-domain operations also account for only a small proportion of the model’s total parameters. Therefore, I am still concerned about the significance of the contribution in the paper.\n\n- The experimental study presented in the paper is a bit weak for publishing at a top-tier conference like ICLR. To show the contribution of the proposed solution, it would be better to either show significant improvements over previous methods with solid experiments (e.g., on ImageNet and directly comparing with GFNet, FNO, etc.) or present the unique property of the new solution that may have a large potential impact. The current experimental study in the paper is not very convincing to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hXyEK4qJZZ", "reviewer": "Reviewer_1MAG", "summary": "The paper introduces Spectral Oriented Neural Invariant Convolutions (SONIC), a novel neural building block which implements learnable spectral filters directly in the Fourier domain. The method is motivated by the shortcomings of current fully-connected, convolutional and attention based feature extraction blocks and derives a novel frequency domain feature motivated by State Space models (like MAMBA).  This allows the implementation of low-rank, orientation-aware global operators in the frequency domain which is suitable for modelling long-range relationships in the data (like large spatial kernels),  while remaining highly parameter-efficient and modestly computational expensive (compared to large spatial kernels). \n\nThe Experimental evaluation on a provided toy dataset SynthShape demonstrates the theoretically derived properties regarding robustness against distortions, noise and geometric transformations in comparison to CNNs and Transformers. A second experiments shows a prove of concept on real medical (MRI) data.", "questions": "Q1: many of the existing Fourier-Domain methods are quite sensitive to the normalization of the coefficients. Did the authors investigate this for their method?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, very well motivated and easy to follow. The presented idea is novel and presents an interesting concept, bringing core elements of state-space system into the the Frequency domain. \n\nThe fact that the learnable filters are formulated in a continuous parameterization is particularly noteworthy. This actually opens the door to solve many sampling related problems in current network designs (aliasing causing low robustness, limitation to fixed input sizes, low robustness against geometric transformations ...).\n\nThe second main contribution of the paper is the low-rank formulation of frequency domain filters. Prior approaches often suffer from large amounts of necessary learnable parameters of crude forced reduction of the same. Transferring the concepts of state-space systems into the frequency domain presents a novel approach to engage these problems.   \n\nThe authors provide a very detailed and honest limitation section - this is very much appreciated!", "weaknesses": "There are several aspects in which the paper could be improved:\n\n1) the paper mentions several previous approaches of Fourier-domain feature extraction (page 3 bottom), but does not compere to these methods in the experiments or in terms of computational complexity\n\n2) the authors missed to discuss and to compare to [1] - another Fourier-domain approach of efficient large kernel implementations. \n\n3) the experiments comparing to CNNs do not show the used kernel size (also not in the appendix). Here it would be good to compare not only to 3x3 klernels, but also to some spatial large kernel implementations like [2] \n\n4) the proposed SONIC block is purely linear - this makes it necessary to conduct two expensive Fourier-Transformation between blocks, in order to be able to apply localized non-linear transformations. This massive practical drawback is inherent to all existing frequency space methods and thus remains unsolved\n\n5) the authors discuss the possibility to combine their method with other (spatial) kernels. This indeed appears to be a very promising approach which would not be very difficult to implement - why did the authors not pursue this?  \n\n\n[1] Grabinski, Julia, Janis Keuper, and Margret Keuper. \"As large as it gets-studying infinitely large convolutions via neural implicit frequency filters.\" Transactions on Machine Learning Research 2024 (2024): 1-42.\n\n[2] Ding, Xiaohan, et al. \"Scaling up your kernels to 31x31: Revisiting large kernel design in cnns.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVeWn6960M", "reviewer": "Reviewer_72sp", "summary": "This paper introduces SONIC (Spectral Oriented Neural Invariant Convolutions), a novel spectral framework that replaces spatial convolutions with a compact set of orientation-aware transfer functions learned directly in the Fourier domain. By modeling filters as continuous frequency functions, SONIC achieves global receptive fields, resolution invariance, and linear parameter scaling with channels. Experiments on synthetic geometric segmentation and 3D prostate cancer MRI detection show that SONIC delivers stronger robustness and comparable or superior accuracy to CNN and Transformer baselines while using far fewer parameters.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a spectral framework for multidimensional signals that offers global receptive fields, complete convolutional capability, and built-in resolution invariance. It provides a lightweight and flexible foundation for building scalable, adaptable vision models.\n2. The paper presents comprehensive empirical validation across both synthetic and real-world settings.", "weaknesses": "1. The innovations mentioned in the abstract and contributions are mainly about unifying existing convolution kernels, spectral filtering, and state-space kernels under one spectral framework. However, the idea of parameterizing operators in the frequency or linear domain already exists in models such as S4ND, GFNet, FNO, and Mamba. The so-called directional modes only add a few interpretable parameters (e.g., direction, scale, damping) to the frequency response function, but in essence, it is still a functional representation of a frequency-domain convolution kernel.\n2. The SynthShape dataset seems to be self-generated to verify the effectiveness of the proposed method (it is not a public dataset, so the persuasiveness of the results is questionable). In addition, there are no ablation experiments—for example, there is no comparison between using spectral factorization and directly learnable spectrum, nor any analysis of the effects of hyperparameters such as parameter count or directional constraints.\n3. The experiments are limited to a toy dataset (SynthShape) and a single medical imaging application. There are no results on standard 2D benchmarks (such as CIFAR-10/100, ImageNet, or ADE20K) to test general visual performance and scalability.\n4. The readability of the paper is not very good, and many formulas are not clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KEuoRxOxIF", "reviewer": "Reviewer_7wpe", "summary": "The paper proposes SONIC (Spectral Oriented Neural Invariant Convolutions), a spectral-domain alternative to spatial convolutional networks (CNNs) and attention mechanisms. SONIC aims to combine the global receptive field propertly of transformers and spectral models, while enjoying strong spatial inductive bias and parameter efficiency that CNNs have. \n\nSONIC learns a set of spectral filters in the Fourier domain, each parametrized by six parameters modeling the amplitude, orientation, and oscillatory nature of the spectral transfer function. Learned matrices B and C factorize these filters and give a representation that is in theory resolution invariant, interpretable and also parameter efficient (parameters scale linearly with channels).", "questions": "Q1) could you devise an experiment to directly test resolution invariance on real data?\n\nQ2) It is hard for me to undestand the visualization in Fig 1. What are the axes? Please summarize the core observations from the Figure. Zooming in (so that white space is minimized) would imporve reeadability.\n\nQ3) Same with Fig2. What are the main observations  from this figure?\n\nQ4) Can you add some form of timings in Table 2? how does Sonic compare to the spatial baselines wrt that? ie what is the actual overhead of all the FFT/iFFTs?\n\nQ5) How do different values of modes and channels affect performance?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* The filters are highly parameter efficient and defined as continuous spectral functions, therefore have a number of desireable properties like resolution invariance and interpretability. \n* The papers contains a concise background section and  clear text presenting the method.", "weaknesses": "W1) Limited Experimental Validation: The experimental validation of the proposed method is limited. SynthShape appears to serve mainly as a proof-of-concept for demonstrating desired properties, but lacks realism. On PROMIS/Prostate158, the comparisons are only against basic baselines, and the observed gains over a simple UNet are relatively small. \n\nMoreover, there are no results on standard computer vision benchmarks (e.g., CIFAR, ImageNet, ADE20K). I do not expect state-of-the-art result in such generic cases, it is however important to demonstrate the broader applicability of SONIC to more generic tasks such as image classification or semantic segmentation.\n\nW2) Lack of Empirical Support for Resolution Invariance: A central claim of the paper is that the method is resolution-invariant. While this may hold in theory, there is no clear experimental evidence to support it. A direct empirical evaluation across varying resolutions would greatly strengthen this claim.\n\nW3) Undiscussed Compute and Memory Overhead: The method requires FFT/IFFT operations at every layer, which likely introduces non-negligible memory and computational costs. These costs are not discussed beyond theoretical costs.\n\nW4) No Hyperparameter ablations: The impact of key hyperparameters (eg K or M) is not explored. An ablation study would help clarify how these parameters affect model performance and stability.\n\nW5) Missing Discussion of Related Work on Hybrid Architectures\nThere is extensive prior work on combining global receptive fields with efficient CNN-style architectures in hybrid models. Classic examples include Non-Local Networks (Wang et al., CVPR 2018) and A²-Nets (Chen et al., NeurIPS 2018).\nGiven the motivation outlined in the introduction (\"With the proposed method, which enables global receptive fields using significantly fewer parameters, we aim to narrow this conceptual gap\"), a discussion of these and similar works is important to situate SONIC in the broader landscape and clarify its novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)