Infinite-Dimensional Feature Interaction
Authors: Chenhui Xu
Fuxun Yu
Maoliang Li
Zihao Zheng
Zirui Xu
Jinjun Xiong
Xiang Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: deep learning
kernel method
feature interaction
Primary area: deep_learning_architectures
TL;DR: We introduce InfiNet, a modern architecture using kernel to explore the infinite-dimensional feature interaction.

Abstract: The past neural network design has largely focused on feature \textit{representation space} dimension and its capacity scaling (e.g., width, depth), but overlooked the feature \textit{interaction space} scaling. 
 Recent advancements have shown shifted focus towards element-wise multiplication to facilitate higher-dimensional feature interaction space for better information transformation. Despite this progress, multiplications predominantly capture low-order interactions, thus remaining confined to a finite-dimensional interaction space. To transcend this limitation, classic kernel methods emerge as a promising solution to engage features in an infinite-dimensional space. We introduce InfiNet, a model architecture that enables feature interaction within an infinite-dimensional space created by RBF kernel. Our experiments reveal that InfiNet achieves new state-of-the-art, owing to its capability to leverage infinite-dimensional interactions, significantly enhancing model performance.

Infnitte-Dimensional Feature Interaction
Chenhui Xu!
Fuxun Yu
Zihao Zheng Jinjun Xiong Xiang 14.
Maoliang Li
Zirui Xu
GGerrg Mason University
University Buffalo
Microsoft (innn,,,injjmm0ouuffaa edu, xiang. chen@pku. edu. en
*Peking University
Abstract
The past neural network design has largely focused on feature representation space dimension and its capacity scaling (c.g., width, depth), but overlooked the feature interaction space scaling. Recent advancements have shown shifted focus towards element-wise multiplication to facilitate ligher-dimensional feature interaction space for better information transformation. Despite this progress, multiplications predominantly capture low-order interactions, thus remaining confined to a finite- dimensional interaction space. To transcend this limitation, classic kernel methods emerge as a promising solution to engage features in an infiniteddimensional space. We introduce InfiNet, model architecture that enables feature interaction within an infinite- -iineneiodnn space created by RBF kernel. Our experiments reveal that InfiNet achieves new state-of-the-ard owing to capability to leverage infinite- dimensional interactions, significantly enhancing model performance.
Introduction
In the past decade, deep neural network architecture design has experienced several major paradigm shifts regarding the feature representation learning. As shown in Fig. [a). the early stage of neural network design dominant by flat stream architectures in the form of weight-feature interaction (e.g., Wx), like multi-layer perceptron (MLP), convolution neural networks (CNN), ResNet, etc. These models usually adopt linear superposition (e.g.. W.x @ W in the feature representation space. Therefore, the feature representation space scaling limited increase model channel width and depth [4 19I. Nevertheless, this scaling approach has witnessed model development from the very small-scale MLPs or LeNet(20) to the recent huge ConvNext V2 [41) With the ultra-scaled parameter amounts, computing complexity, and model size, the return of investment on model performance by further scaling feature dimensions has largely plateaued [0]]
Despite the plateau feature representation space, recent sporadic architecture design works [29 shed light another potential dimension of scaling: feature interaction space. Specifically, as shown in Fig. ()), these neural network designs generally demonstrate feature-feature interaction (e.g..
Wix s W,x). As mathematical example, the self-attention mechanism in Transformers [8] can be formulated x (x & (x²) @ fx('),, which is also element- wise multiplication between processed input feature themselves. Characterized by element-wise interaction, these designs offer complementary feature correlation capabilities in addition simple linear superposition. Such feature interactions have become the essential mechanisms of mainstream state-of-the-art neural architectures. For example, it's implemented in SENet with squeeze and excitation [17 non-local network with transposed multiplication [39], vision transformers with self-attention
38 gated
aggregation 222 35 46), and quadratic neurons
45
"Corresponding Author.
'The symbol denotes the Direct Sum vector spaces. This corresponds structures such as channel expansion/bootlenecki in commonly used neural networks. While the @ is elementwise multiplication.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

y
y
W
X
W
K (W,, W&X)
Wax + WyX
X1
X
X2
Z/z
+0W2
(MN
Xn
XT1
W
X1
Xz
X
W
W
W
X1
X2
W
X2
Wax
X1
X2
Xyn
(a) Feature Representation Space
(b) Feature Interaction Space
(c) oo-D Feature Interaction Space
Figure 1: (a) Traditional feature representation without interaction [1,4], (b) Recent work with
finite feature interaction 35
45
(c) Our method: Kernel-enabled infinite feature interaction.
Although these models greatly improve the performance of -tate-of-the-arts, as mentioned above, these works provide diversified explanations that neglect the underlying shared design of element- wise feature multiplication operation (29). and thus may fail to reveal the fundamental source of improvement. To provide both explainability and quantifiability. in this paper, propose unified theoretical perspective to rethink the feature interaction scaling, i.e., the dimensionality of feature inferaction space. For example, shown in Fig Tb), by employing the 8, element- wise multiplication, an implicit interactive quadratic space Q span (4f. T1122 In-1Inn x with degrees freedom nnn + 1)/2 is constructed from the original representative vector space y span (x1, 22,- ... I.,)) with degrees of freedom [29]. Such space dimensionality scaling is the key to improving feature representation quality and end-task, as we will show later.
From the unified feature dimensionality perspective, new opportunity emerges in neural architecture design, that is scale to Sefinite-dimessionnl feature interactions than former methodologies (e.g.. from n n + 1)/2 to lim
[n-+k k-1)! However, scaling feature interaction space dimensionality (n-1))l 11)!!
from architectural enhancements (e.g.. quadratic, self attenniono and recursive gates) comes with linear scaling cost w.r.t. interaction order k. which hinders the infinite dimensionality increase
22
Thus, there is an open question:
How can we efficiently extend interactions to an infinite-dimensione space?
Inspired by traditional machine learning, we propose approach that introduces kernel methods for feature interaction in neural networks. We define a set of feature interactions between features Wax and Wix with kernel function (Wa,, x. Wix) instead of the element-wise multiplication. As shown in Fig Fig c), the kernel method transforms the feature to an ultra-high dimensional space by an implicit mapping (--). From there, the feature interaction space defined by the inner product on the Reproducing Kernel Hilbert Space (RKHS) [1 H. constituent with the kernel function K ((,,)) )). The RKHS can greatly expand the interaction space at very little cost, enabling infinite dimensions.
We then propose InfiNet, a novel family of neural networks that generate high-quality feature represen- tations. Specifically, introduce the Radial Basis Function (RBF) kernel [32] as replacement of the common @ or @ operations. With the infinite series expansion in RBF kernel, it enables theoretical "I...""""
provable dimensionality approximation span my = Van.: /ny!..mu!! while with as low-overhead as feature
evaluating an exponential function. In this way, InfiNet enables efficient interaction space scaling upon finite set of branches in the model architecture, thus achieving better complexity performance tradeoffs than prior sate-of-thee-rs
Contributions. We make the following contributions:
We unify the perspectives of recent feature interactive works and identify novel direction of neural network performance scaling: the feature interaction space dimensionality.
We propose method to expand the feature interaction space infinite dimension with RBF kernel, that can effectively model the complex implicit correlations features.
We propose InfiNet, a novel series of neural networks that explore the neural interaction from infinite-dine dimenniodmmniionall space, and achieve state-of-the-an performance.

---

Extensive ablation studies verify that the shift from finite feature interaction space to infinite one is key factor learn better representations and therefore improving model performance. Meanwhile, large-scale experiments on ImageNet classification, MS COCO detection and ADE20K segmentation also demonstrate InfiNet design's 8s effectiveness, which consistently outperforms the state-of-the-an and finite-order interaction networks [25 35
flat stream networks 14 26
2 Related Work
Interactions in Neural Networks. Interaction in neural networks has undergone long period of change. During the evolution from AlexNet [19] to ResNet [[4], the model has for long followed the principle of summing the weighted pixels at each position in the layer-by-layer feature iteration. And then, as the potential of the attention mechanism was realized, model development is towards an element-wise multiplication way. This trend is punctuated by the emergence of models such as Non-Local [39]. Transformer [38), and ViT [T]] Recent insights suggest that the crux of the attention mechanism lies in high-order information interactions, rather than the mechanism of "Attention" itself 4 35 43 This revelation has spurred the development of innovative neural network architectures. For example, HorNet (35), QuadraNet [44] and MogaNet (22 examines high-order models from the perspective of spatial interactions through pnltiilicatioonineeraaaee architecture design. However, limited by the support of existing deep learning platforms such as Pytorch [33], few attempts have been made to extend the model's 's feature interaction to an ultra-high dimensional situation through the kernel method for more potential.
Kernel Methods in Neural Networks. The fundamentals of kernel methods are well-studied in traditional ML domains like support vector machines (36) but they are less used in neural architec- tures. Early extensions to deep learning through the kernelized perceptron [7] have improved the performance but mainly in shallow neural networks. Recent advancements include the development of Convolutional Kernel Networks (CKN) [30 which merge CNNs' robust feature learning with kernel stability. This approach offers theoretical foundation for deep learning's application in struc- tured data. Additionally, the introduction of Kervolutional Neural Networks [6] replaces traditional convolution in CNNs with kernel-based operations enhance feature extraction without excessive computational costs. The combination of Gaussian processes with neural networks to create Deep Kernel Learning 40 adjusts model complexity based on data while maintaining Bayesian inference. The neural tangent kernel (NTK) [18] framework establishes direct connection between infinitely wide neural networks initialization and kernel methods. Specifically, NTK shows that as the width grows, the network's 's training dynamics can be described by a kernel function, linking the neural net- work's behavior to that of kernel methods. Random features 34 provide an efficient approximation to the feature mappings used in kernel methods. By random projections, one can approximate the inner product defined by kernel function, making feasible to apply kernel methods B3
Despite these innovations, a significant limitation remains: Although these methods expand the feature representation space, they fail scale up the feature interaction space, limiting the network to aggregate information in a superposition manner. As a result, these methods also fall short of potential feature interactions and thus are incapable of handling complex data correlations and functionalities.
3 From Feature Representation Space to Interaction Space
We start with considering the transformation of feature representation of a normal chape-preserving 2-layer perceptron block. Given an input x = (2,, ... Ix) with n-dimension. We denote the first layer transformation as z g(x) o (W,x) (we omit the bias term for simplicity), and the second layer transformation as hzz)) = Wzz. Therefore the whole block is a mapping hog R +> R + IR from feature space I2. to a middle feature representation space R" and eventually to a output space R". Expanding the width of the neural networks, for example with coefficient 2, is going to expand the feature representation space to 2h The core idea of this dimensional expansion that implicit associations in features in low dimensions will be expressed explicitly when projected into a high-dimensional space. Model architecture (i.e. convolution, multi-branch (37) is an additive superposition of pixels, essentially no different from the perceptron in terms of representation space.
3

---

3.1 Feature Interaction Space
Although flat stream neural networks are defined by linear transformations and activations, modern network design also introduces multiplications in the network structure or neurons. They are called attentions from an interpretable point of view, that is the degree of interest of one position relation to another; from more abstract point of view, they are called interactions in spatial contexts. But invariably, these are realized with element- wise multiplication. We conduct the following definition.
Definition (Feature Interaction) A Feature interaction refers to transformations between features with the same or different positions defined by element-wiss multiplication.
For example, Star Operation [29 a basic 2-order feature interaction, defined as (Wax) (Wix), where W.. We R". This is simple element-win multiplication fusion of two linear transforma- tions of the ordinary input x. We write the expansion of such multiplication operation:
y = (Wax) (Wxx)
WoiT:
WhjJz)
Σ aiyTya iSj
where 0i.j = WaiWhi + Way Whi, if i + j, and Ciii Whi Then we vectorize a and 2;Ryy
[11, 01,2 02222 Ap-1,n 1, Gn E (=n++))/2
(2)
= 212,,
Fn-1Ins Zn
X can therefore define basis of space. Thus the output of the current layer can be rewritten as:
y a1jz, Ax.
From the independence of the pixel level, we know that each term in x is linearly independent, this indicates every dimension in X is individual dimension. Given set of basis vectors X. define sxan(x) as the feature interaction space. In this way. the generation of the next layer of features is constituted by a linear superposition Ax the feature interaction space span(x) like in Eq.
3.2 Dimension of Feature Interaction Space
Now, we consider the number of dimensions of feature interaction space. Given k-1 multiplication operations, we first define the k-order feature interaction space as follows:
Definition (Feature Interaction Space) A k-order Feature Interaction Space S refers the span of monomial basis { .. Σ d. k, N) defined by k-order Feature Interaction.
An element-wiss multiplication generates feature interaction space of n(n + 1)/2 dimension, which is the number of elements of an upper triangular matrix. In general, considering the symmetry of the interactions and elements, for k-order interaction on the n-dimensional feature, the dimension of the corresponding feature interaction space is:
(n kn+k-10 1)! (n 1)!A!
dim((*))
The next layer of feature generation based on the feature interaction space greatly expands the spatial dimensions which the features are mapped compared the original model based only on the feature representation space, i.c., it is possible to explore the feature's 's non-linearity in high-dimensional space. worth noting that this process introduces terms like æ2 an interaction that cannot be captured by traditional plane networks in feature representation space.
For example, consider the Self-Attention in the Transformers tains two element wise multiplications, so it explores a 3-order feature interaction space. This is due to the fact that: (1) in the first stage, in the query-key dot-product attention map com- putation All(x) = Q KT = Wox x W explores the feature interaction space A = span (a?: 0132, ... En - 2) (2) In the second stage, the multiplication between the atten- tion map and the value y o (At((x)) V (x)t(x))) W,x explores the feature interaction space S A 8 R = span (x. ff x 13233, ... In | 22, ai),, which has (n + 2) + 1)n/6 dimensions in line with (5. Thereby we explain, from the perspective of feature interaction space, why the transformer family of models has, so far, generally outperformed recurrent neural networks and convolutional neural networks in various domains.
38]
The Self-Attention con-

---

Expanding Interaction Space to Infinite Dimension
This element-wist multiplication-basse bbaed interaction, since the construction of each order of the feature interaction space is based on multiplication operator, leads to a problem in that feature interaction space expansion is still difficult. This is due the fact that these interaction operators is explicit mapping, which tends to have quadratic or higher complexity w.r.t the input length (e.g. self- attention (38) or lengthy recursive designs (e.g. HorNet (35) and linear complexity w.r.t interaction order. But the problem with building this mapping explicitly is: (1) The complexity of mapping itself. The computational overhead associated with defining a set of explicit nonlinear mappings is non-negligible. A mapping from C' channels to C" channels means a computation complexity of O(CC"). (2) The complexity of interaction. The complexity of the inner product used to interact with the two sets of features increases dramatically to O(C") when the dimension is raised. Considering C² >> C, these two complexities will largely increase the computational overhead of the networks.
We would like to obtain a method that can expand the dimension of feature interaction space as much as possible in O(1) time. Fortunately, the machine learning community has already given a method for increasing the dimension of a feature defined on the inner product: kernel methods.
4.1 Expanding Interaction Space with Reproducing Kernel
The nature of kernel methods is that they are substitutions for inner product operations. This requires combining element-wise multiplication and summation to define a set of inner products within the network. For this purpose, we rewrite the form of element-wiss multiplication in Eq. on two groups of features, which a common design in literature architecture (e.g. multi-head self-attention (38)
It will therefore be superposition of multiple interaction in the format:
y Σ x WxX = (Wax, Wbx) 1=1
(6)
where W.x [ Wa2X, Wacx], C N is the number of branches. Thereby we generalize the form of the feature interaction from element-wise multiplication inner product which is a multi-branch paradigm. At this point, we have W.x E IRC, while it is generated from feature representation space Rd. In order to extend the interaction space, we need further project W, W, and Wb to a high-dimensional space. An obvious way to do this is to construct implicit mapping (-) to a high-dimensional space, so we can compute (W(W,,) x)), , & (Wix)) for interaction.
By Mercer's Theorm 31 taking a continuous symmetric positive semi-definite function K K(s, f), there is an orthonormal basis (o:(-)), 0, 1, 0o, consisting of eigenfunctions of function K K(,,)) such that the corresponding sequence of eigenvalues {};} is non-negative. These means:
(s,t)) = Σ (s)
where Vi * j, Vs and t, (5)(s)))()))))) = 0 since. Then we construct Hilbert space H with the orthonarmal basis (VAidi()) Consider vector / = (2, ) the space H. then we have:
Σ f(ddi()).
(8)
Thus for a vector -s) in space H:
K(s,.) = = x((()))(()) = Σ Virollo) (VTi01(s), -s).
(9)
==1
Therefore, for the Hilbert space H. we can define the reprodcuing kernel by:
00
(K(s, K(t, = Σ VA Σ (s)o,().
(10)
Implicit Mapping to High- Biigh-Diennsooall RKHS. Let ((8) K(.,.), then we have (0(s), x()) = (s, The Hilbert Space H is known as the Reproducing Kernel Hilbert Space (RKHS) corre- sponding to kernel function K( (..)). Note that the ((-) is therefore defined on the RKHS H. which
5

---

CIFAR Accuracy/Epoch
Accuracy
256. 1x1, 32
256, 1x
0.9
3x 32
32, 3x3,
100
0.7
32, 1x11, 256
32, 1x
Epoch
0.5
DemoBlock
RELU
20
40
60
Element-wise Addition
Element-wise Multiplication
RBF Kernel
Figure 2: Comparison of simple representation, finite interaction, and -iniiie-admeniionall interaction. The ? circle in DemoBlock chosen from element-wing Add, element-wise Mul. or RBF kernel.
can be diiente-dimensionall given specific kernel K ((..)). The mapping Φ(-) does not have to have an explicit expression since we can get the result of ($)s), (()) by computing (s, t). This means that we can achieve an extension of the dimensions for the feature interaction space by simply replacing the inner product (Wax, Wb)) used in the interaction with kernel K KWW,, Wbx).
4.2 Infnnite-Dimensionall Feature Interaction with RBF Kernel
To maximize the dimension of the feature interaction space, we consider Radial Basis Function (RBF)
Kernel Kar(s, t) =exp (- I|s
(11)) with an infiniteddimensional RKHS, given the fact that:
2
t)
cxp
|s t11
= Σ
exp
|s| + l|t12
(11)
j!
j=0
||sl1²
1
exp (-2 [|tl1))
1 m
exp
(12)
ns!
Vnn! no!
1= no +22 +mx=)
= =()nr(s), (ror(t)),
(13)
where (x) EEz0 Eze exp (-2 1x
1 Vn, I..
span JEN Span
n==j /ny 1..!
Infinite-dimensional Feature Interaction Space Observing the RKHS of such RBF kernel, /. We note that each of its dimensions is a j-order interaction within the feature given the fact one of the bases of this RKHS is:
29 "0 Se n== /nnl..n!!
(1. In. 27 21
F2.
(14)
which contains an all-order monomial among all elements of the feature x. This means that we get an dnfinitedimmensional Hilbert space for the superposition of interaction information through such an RBF kernel, and most importantly, each dimension of this space is defined by a feature interaction. Therefore, get infiniteddimensional feature interaction space.
4.3 Demo Case Performance of Models Different Feature Space
In order to compare networks that utilize the summing superposition of information mappings on the feature representation space, finite feature interaction space, and space, we design a demo network with 8 DemoBlock, as shown Fig, 2 DemoBlock has a two two-group design, the only difference among models in different spaces is the interaction method the end of the block (add for simple representation, multiplication for finite interaction, and RBF kernel for infinite dimensional interaction), the demo models are trained CIFAR10 and Imag-
interaction
As shown in Fig. 2 with the procedure of transferring from the simple feature representation space to finite feature interaction and eventually to infinie-ddimnniion interaction, the performance

---

Stage
Stage
Stage
Stage
Linear
11
W
XC
8
X2C
16 x4C
W
32
H
X
32
W
XBC
Linear
LayerNorm
HxW x3
Gom monni plnp
Gaannn Tomeenn Rung
Gowmamm
Image
WILM
Blocks
Infi-
Infi- Blocks
Blocks
Infi-
Infi- Blocks
dw-Conv
dw-CConv
dw-Conv
Linear
Lineur
LayerNorm
(a) InfiNet Architecture
(b) InfiBlock
Figure 3: Overview of InfiNet. (a) Four-stage hierarchical InñNNet design. (b) InfiBlock Design
on CIFARIO of the networks is growing. This is done throughout the training process, implying the superiority of feature iteration a high-dimensional interaction space. The right side of Fig. shows the Class Activation Mapping 447 of three different demo nets on Tiny-ImageNer. From this, we can see that the network of feature interaction better reflects the pixel-level correlation within the image.
Method
5.1 InfiBlock: Infnite-Dimensionee Spatial Feature Interaction
InfiBlock. In this section, we present InfiBlock, the basic block build the high high-performant InfiNet architecture to achieve InfiBlock starts with a LayerNorm layer and subsequently transforms the feature into separate representations of the two groups through two different linear layers. Then InfiBlock utilizes depth convolution with an expansion coefficient of and a ReLU activation to obtain a feature branch vector of length an RBF kernel for feature interaction. At the same time, we retain a pathway containing only one depth-wo wise convolution for summing superposition over the feature representation space to ensure that the linear connections between features are not neglected and overfitted. This followed by residual connection with the original input values after passing through a two-layer MLP. Starting with a input X E R #WC Infi- Block can be formulated as:
spatial feature interaction. As presented in Fig. 3 b).
each group. The feature branch vectors on both groups are then fed into
X = LayerNorm(X")
(15)
1Z Z,,, | (o (x'' (Convy ([CConvoX''Waa (X'W))), Conv. x MLP(LayerNorm (Kr Zz. 2%) 2()) + x
(16)
(17)
where Kros -,) = exp( - ER is an RBF kernel with out any
TR. #WC+r expands the number of channels of feature to times, where Wab preserve the shape of the feature. The discretization of the feature is performed in depth-width convolution. LayerNorm(.) is layer normalization (2 and MLP(.) is activated by GELU [15).
and || |13 is squared I-2 norm. Conva, (X''.....W....
Specifically, following the ConvNeXt (26), we select 7 x 7 depth-wise convolution obtain larger receptive field. We set the expansion coefficient r as practice. This means:
(18) where 7 a (DW-Conv... (')))) HwC. Therefore, the RBF kernel performs feature interaction on multiple (7) branches of convolution filter and outputs a result still in R nWC
2. 2
[2 Z 2 Z/,, 4/7171,
5.2 Model Architectures
As shown in Fig. 3 a), InfiNet uses the widely adopted 4-stage hierarchical architecture as in ResNet [14, ConvNeXt [26] and Swin Transformer (25). We stack InfiBlocks into each stage.
7

---

Table ImageNet classification results. We compare our models with state-of -o-e---------rr art models with comparable parameters, the Top-1 accuracy is reported on the ImageNet- validation set.
Interact. Params FLOPs
Topl
Interact. Params FLOPs
Top1
Model
Orders
(M)
(G)
Acc.(%)
Model
Orders (M)
(G)
Acc.(%)
ConvNeXI-T226
no
29
4.5
82.1
ConyNeXt-B 26
no
89
15.4
24
30
5.0
82.5
SLaK BAK-BB24 2
no
85
17.1
84.0
Conv2Former-
27
4.4
83.1
Conv2Former-h
16
90
15.9
84.4
UniFormer-S 21
22
3.6
82.9
CoAtNet-2
75
15.7
84.1
CoAINet-0(88
25
4.2
82.7
FocalNect-
89
15,4
83.9
FocalNet-
82.1
Swin-B
89
15.4
83.5
125
4.5
81.3
HorNet-B
2-5
87
15.6
84.3
MogaNet-I.
4
83
15.9
83.5
HorNet- T
2-5
22
82.8
InfiNet-B
82
12.8
84.5
MogaNet-SS2
25
5.0
83.5
InfiNet-L
116.8
19.1
84.8
InfiNet-T
0C
23
3.2
83.4
ImageNer-21X Pretrained Models Fine-tuned @3384²
ConvNeXt-S
no
50
83.1
SLaK-S
no
83.8
ConvNeXt-LL 26
198
101
87.5
Conv2Former-9
16
50
8.7
84.1
CoAtNet-3* g
3
168
107
87.6
UniFormer-Bi
21
50
8.3
83.9
FocalNet-L.
46
CoAINet-1
42
8.4
83.3
Swin-L'
FocalNet-S[A4
50
8.7
83.5
HorNet- [35
2-5
202
102
87.7
Swin-S
50
8.7
83.0
MogaNet-XL. 22
4
181
102
87.8
HorNet-S 35
2-5
50
8.8
84.0
InfiNet-L.
116.8
60
87.8
MogaNet-HI 22
44
9.9
84.3
ConvNeXt-XL 26
350
179
87.8
InfiNet-S
48
7.2
84.0
InfiNet-XI.
00
255.8
126
88.2
We set the number of channels in each stage [C,2C,4C,8C] following common practice. We build a family of InfiNets that InfiNe--T/SB/L/XL with model variants hyper-parametert C' (64, 96, 128, 128, 192), number of blocks in each stages {2, 18,2) for InfiNet-TTS/BB and (3,3, 27, 3} for InfiNet-L/XL. A down-sampling with x 2 convolution with stride = is used to connect each stage. STEM (26 is used to connect the input of InfiNet Stage 1.
Experiments
We perform a series of experiments to validate the efficacy of InfiNets. The primary results on ImageNet [9] are showcased and contrasted with several architectures. Furthermore, models were
evaluated on downstream ADE20K [48] semantic segmentation and COCO (33 object detection. ImageNet- IK experiments conducted 4xNvidia A A100 GPUs and ImageNet-21K 16x.
6.1 ImageNet Classification
Setups. We conduct image classification experiments on ImageNet- [ which contains 1.28 million training samples belonging to 1000 classes and 50K samples for validation. We train the InfiNet-T/S/BLL models for 300 epochs with AdamW (2 optimizer. We use the cosine learning rate scheduler 27 with 20 warmup epochs and the basic learning rate is set as 4 x 10-3, The training resolution is set as 224 224. further evaluate the InfiNet's scalability, we train the InfiNet-L/XL. models on 14M-sample ImageNet-22K dataset for 90 epochs and then fine-tune on ImageNet-IK IK at 384 x 384 resolution for 30 epuchs following [26.. More details can be found in Appendix A.
Results. We present our ImageNet experiment results and comparison with basclines in Table | Our models achieve competitive performance with sfate-of-the-art, It is worth noting that InfiNet has about 20% fewer FLOP than the baseline models with similar parameter scales, but still achieves great performance. demonstrates the effectiveness of our proposed generation of features in cimittedimensionall interaction spaces. Experiments on isotropic models can be found in Table
Diffference between FLOPs and FLOPS: FLOPs (floting point of operations). the number of floating point operations, used to measure complexity of an algorithm/modde. This number gets smaller the better. The FLOPs different from FLOPS (tloating point per second), which a measure of hardware performance. Following other beepleeming architecture works, we use FLOPs to measure the computing demands. Since our model gets smaller FLOPs, method is more efficient.

---

Table Object detection and semantic segmentation results on MS COCO and ADE20K.
Object Detection with Cascade Mask R- R-CNN.R ix
Semantic Segmentation with UperNet 160K
Model
Aphox
Apmask
Params
FLOPs
m
mloU
Params
FLOPs
ConvNeXt-TT
26
50.4
43.7
86M
741G
46.0
46.7
60M
939G
Swin-T 25
50.4
43.7
86M
745G
44.5
45.8
60M
945G
HorNet-T 35
51.7
44.8
80M
730G
48.1
48.9
52M
926G
InfiNet-T
51.9
46.2
77M
724G
46.7
47.4
50M
924G
ConvNeXt-S
51.9
45.0
108M
827G
48.7
49.6
82M
1027G
Swin-S 25
51.8
44.7
107M
838G
47.6
49.5
81M
1038G
HorNet- 35
52.7
45.6
107M
830G
49.2
49.8
81M
1030G
InfiNet-S
52.8
46.4
98M
802G
49.4
49.9
78M
1002G
ConvNNcXt-B
52.7
45.6
146M
964G
49.1
49.9
122M
1170G
Swin-B 26]
51.9
45.0
145M
982G
48.1
49.7
121M
1188G
HorNet-B 65
53.3
46.1
144M
969G
50.0
50.5
121M
1174G
InfiNet-B
53.7
47.3
126M
906G
50.2
50.9
105M
IIIG
ConwNeXI-L
54.8
47.6
255M
1354G
53.2
53.7
235M
2458G
Swin-L 25
53.9
46.7
253M
1382G
52.1
53.5
234M
2468G
HorNet-L. 35
55.4
48.0
251M
1363G
54.1
54.5
232M
2473G
InfiNet-XLL
56.3
48.9
273M
1454G
54.6
55.2
253M
2544G
Table 3: More Results on isotropic models and different kind of Reproducing Kernel
(a) Isotropic Models
(b) Ablation Study
Interact. Params FLOPs
Top1
Interact. Params FLOPs Topl
Model
Orders (M)
(G)
Acc.(%)
Model
Orders
(M)
(G)
Acc.(%)
ConsNeXl-SSiso.)
22
4.3
79.7
InfiNet-Q
23
3.2
81.6
Conv2Former(iso)
4.3
81.2
InfiNet-
2
82.1
De2
79.8
InfiNet-2-polyno.
4
23
3.2
82.3
HorNet-S(isn.)
2-5
80.6
InfiNet-3-polyno.
6
23
3.2
82.5
InfiNet-S(iso.)
oc
4.3
81.4
InfiNet-T
8
23
3.2
83.4
6.2
MS COCO Detection
Setups. We evaluate models for object detection tasks widely used MS COCO 23 benchmark. the experiments, BafiNet-T/S/b/X serves as the backbone network within Cascade Mask RCNN (5] We use AdamW as the optimizer and a batch size of 16 and adhere the xx schedule, following ConvNeXt [26] and Swin (25). We resize the input so that the longer side is at most 1333 and the short side is at most 800. We initialize the backbone model with ImageNet- 1K pre-trained weights for T/S/B models and ImageNet-22K pre-trained weights for XL model.
Results. As shown in Table 2 InfiNets comprehensively beat the non-interactive model Con- vteXt (66. and space-limited interactive Swin 25 and HorNet (35) under the same cascade Mask RCNN framework in box AP and mask AP. This means that for such dense prediction tasks, spatial interaction of features in a high-dimension space is crucial. The InfiNet series model obtain0.9 ~ 1.5 box AP and 1.3 ~ 2.5 mask AP gain compared with non-interactive ConNeXt.
6.3 ADE20 Segmentation
Setups. We evaluate our models for the semantic segmentation task on widely used ADE20K 48
benchmark covering 150 semantic categories 25K images, in which 20K are used for training. We use UperNet 42 for as the basic framework and adopt InfiNets as the backbone model. Training AdamW optimizer with learning rate x 10 and batch size 16.
details follow the Swin (25)
The right half of Table lists the mloU and corresponding model size and FLOPs for different configurations. Our models beat most of the baseline in the segmentation task. The results show that as the model size increases, the performance gap between InfiNet and other baselines getting larger, illustrating the scalability of InfiNet segmentation.
Results.

---

6.4 Ablation Study
We use the additive operator, Hadamard product, quadratic polynomial kernel and cubic polynomial kernel, and RBF kernel in the kernel methods section of InfiNett respectively, tiny size model, to verify the effect of the gradual expansion of the order of the interaction space up infinite dimensions on the performance of the model. As in Table b), we can see that the performance of the model is gradually improving as the order of the model interaction space increases up to infinite dimensions.
Conclusion
As conclusion, in this paper, propose that one of the key points of success of today's 's element- wise multiplication--ast models is that they explore a high-dimensional feature interaction space through feature interactions. And the RBF kernel can greatly expand this interaction space into an infinite dimensional feature interaction space. Based on this observation, we propose InfiNet, a perforrrfome neural network that explores a modern model structure, which has achieved state-of state-of-the-aar the-art results on several visual tasks.
feature interactions while using
Limitations
Although we propose the use of kernel methods for kernel methods we have tried so far are the RBF kernel function and some polynomial kernels of finite dimension. Substitutions utilizing variety of kernel, including Laplace kernels, exponential kernels, a learnable kernel, etc., can be considered in subsequent studies. Our model has only been performed some basic computer vision tasks, and validation in language and other modalities still requires some effort. The training of our model is only performed in the supervised learning paradigm, and more training and validation on self-supervised tasks still require effort. In addition, to avoid additional hyperparameter tuning, may have deprived us of the possibility of exploring the optimal InfiNet, but due to the high cost of training the model, we will leave the impact of this hyperparameter on the InfiNet as follow-up work.
feature interaction, the only
fixed the parameter in the RBF kernel to 1. This
Broader Social Impact
InfiNet is state-of-the-an vision neural network architecture. The advancements in computer vision neural network architectures hold significant potential for positive societal impact, particularly in enhancing healthcare diagnostics, improving security systems, and advancing autonomous trans- portation. However, it is crucial to address potential negative implications such as privacy concerns, algorithmic biases, and job displacement. Ensuring ethical development and deployment involves implementing strict data protection measures, promoting fairness and inclusivity in algorithm design, and supporting workforce retraining programs. By proactively managing these challenges, we can maximize the benefits of computer vision technologies while minimizing their risks. Engaging with diverse stakeholders will be essential to ensure these technologies are used responsibly and equitably.
Acknowledgement
We sincerely thank the reviewers for their valuable suggestions during the review stage.
10

---

References
[1] N. Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society,
68(3)3337-444, 1950.
[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
[3] F. Bach. Exploring large feature spaces with hierarchical multiple kemnel learnin. Advances in neural
information processing systems, 21, 2008.
[4] B. Bai, J. Liang, G. Zhang, H. Li, K. Bai, and F. Wang. Why attentions may not be interpretable? In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 25-34, 2021.
[5] Z. Cai and N. Vasconcelos. Cascade r-can:
Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154 0144-616.2226 2018
[6] C. Chen, M. Seuret, M. Liwicki. J. Hennebert, and R. Ingold. Kervolutional neural networks. arXiv preprint arXiv: 1119994439555 2019.
[7] Y. Cho and L. K. Saul. Kerne methods for deep Iearning. In Advances in neural information processing systems, pages 142--50,222 2009.
[8] 7. Dai, H. Liu, Q. Le, and M. Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems, 34:3965-3977, 2021.
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and puttern recognition, pages 248-255, leee, 2009.
[10] E. Dohmatob, Y. Feng. P. Yang, F. Charton, and J. Kempe. A tale of tails: Model collapse as a change of scaling laws. International Conference on Machine Learning, 2024.
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner. M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scalc. In International Conference on Learning Representations, 2020.
[12] F.-L. Fan, H.-C. Dong, Z. Wu, L. Ruan, T. Zeng, Y. Cui, and J.-X. Liao. One neuron saved is neuron earned: On parametric efficiency of quadratic networks. arXiv preprint arXiv:2303.063 2023.
[13] A. Hashemi, H. Schaeffer, R. Shi, U. Topcu, G. Tran, and R. Ward. Generalization bounds for sparse random feature expansions. Applied and Compulational Harmonic Analysis, 62:310-330. 2023.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[15] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[16] Q. Hou, C.-Z. Lu. M.-M. Cheng. and J. Feng. Conv 2former: A simple transformer-style convnet for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[17] J. Hu, L. Shen, and G. Sun. auueezz-and-exiiiiee computer vision and pattern recognition, pages 7132-7 7141, 2018.
networks. In Proceedings of the IEEE conference on
[18] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[20] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition.
Proceedings of the IEEE, 86 1):2278 2324, 1998.
[21] K. Li, Wang. G. Peng, Song, Y. Liu, H. Li, and Y. Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Conference on Learning Representations, 2022.
[22] S. Li, Z. Wang, Z. Liu, C. Tan, H. Lin, D. Wu, Z. Chen, J. Zheng, and S. Z. Li. Moganet: Efficient multi-order gated aggregation network. International Conference on Learning Representation, 2024.
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L.. Zütnick. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th Furopean Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13,, pages 740-755. Springer, 2014.
[24] S. Liu, T. Chen, X. Chen, X. Chen, Q. Xiao, B. Wu, T. Kärkkäinen, M. Pechenizkiy, D. C. Mocanu, and
Z. Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 1 using sparsity. In The Eleventh International Conference on Learning Representations, 2023.
[25] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
11

---

[26] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-1 1196, 2022. [27] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[28] Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on
Learning Representations, 2019.
[29] X. Ma, X. Dai, Y. Bai, Y. Wang, and Y. Fu. Rewrite the stars, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024.
[30] J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid. Convolutional kernel networks. In Advances in neural information processing systems, pages 2627 2635, 2014.
[31] Mereer. Xvi. functions of positive and negative type, and their connection the theory of integral equations. Philosophical transactions of the royal society of London. Series A, containing papers of a mathematical or physical character, 209 -458):415- 1133:4455146,,, 1909.
[32] M. T. Musavi, W. Ahmed, K. H. Chan, K. B. Faris, and D. M. Hummels. On the training of radial basis function classifiers. Neural networks, 5(4):595-603, 1992.
[33] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradhury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein,
L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[34] A. Rahimi and B. Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[35] Y. Rao, W. Zhao, Y. Tang. J. Zhou, S. N. Lim, and J. Lu. Hornet: Efficient high- spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems, 35:10353- 10366. 2022.
[36] B. Schölkopf, A. Smola, and K.-R. Müller. Nonlinear component analysis as kernel cigenvalue problem. Neural computation, 10(5):1299-1311) 1998.
[37] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1-9, 2015.
[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[39] X. Wang. R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[40] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 770-378. 2016.
[41] S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages (6133-16142, 2023.
[42] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for seene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418 434, 2018.
[43] C. Xu, X. Wang, Yu, Xiong, and Chen. Quadranet v2: Efficient and sustainable training of high-order neural networks with quadratic adaptation. arXiv preprint 2rXiv:2-05.03 2024.
[44] C. Xu, F. Yu, Z. Xu, C. Liu, J. Xiong, and X. Chen. Quadranet: Improving high-order neural interaction efficiency with hardware-awe aware quadratic neural networks. The 29th Asia and South Pacific Design Automation Conference, 2024.
[45] Z. Xu, F. Yu, J. Xiong, and X. Chen. Quadralib: A performant quadratic neural network library for architecture optimization and design exploration. Proceedings of Machine Learning and Systems, 4:503- 514, 2022.
[46] J. Yang, Li, X. Dai, and J. Gao. Focal modulation networks. Advances in Neural Information Processing Systems, 35:4203-4217, 2022.
[47] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921 2211, 2016.
[48] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through 2de20k dataset. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 633-641, 2017.
12

---

A Appendix
A.1 Training Details
The training details for ImageNet experiments are shown in Table and Table 5
Table 4: Training details for ImageNet- 1K experiments
Configuration
InifNet-T/S/B/L.
Input resolution
224
Epochs
300
Batch size
192/128/64/64
Optimizer
Adam
Adam (31,
0.9, 0.999
Learning rate
0.004
Learning rate decay
Cosine
Weight decay
0.05
Warmup epochs
20
Label smoothing
0.1
Stochastic Depth
Y
Rand Augment
9/0.5
Repeated Augment
Y
Erasing prob.
0.25
Colorlitter
N
Gradient Clipping
Y
EMA decay
Y
Table 5: Training details for ImageNet-2IK experiments
Configuration
IN-21K PT
IN-IK FT
XL
XL
Input resolution
224
384"
Epochs
90
30
Batch size
256
64
Optimizer
Adam W
Adam W
Adam W (81,32)
0.9, 0.999
0.9, 0.999
Learning rate
4 x 10-3
5 10-5
Learning rate decay
Cosine
Cosine
Weight decay
0.1
0.00001
Warmup epochs
5
0
Label smoothing
0.2
0.1
0.2
Rand Augment
9/0.5
9/0.5
Repeated Augment
N
N
Erasing prob.
0.25
0.25
Gradient Clipping
5
EMA decay
N
Y
13

---

Figure 4: Visualization Comparison (1) Feature Representation Space model, (2) Finite Feature Feature Interaction model
Interaction Space model, (3)
A.2 More Visualization Comparison
In Fig. we provide more Class Activation Mapping to illustrate the interpretability of interaction models. dnfinit--dimensiona Interaction models capture more totality of objects within category, illustrates the importance of this anfinit-ddimensione interaction.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We attribute the success of modern models to feature interaction and claim the kernel methods can expand this feature interaction space. And propose the InfiNet. They are included in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
After the Conclusion
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: In Section B and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 6 and Appendix A.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
All datasets we use in this paper are third-pane third-party open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /ttee//ggudddeeeedSuuuummbibbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section G and Appendix
A.I A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Training for multiple times is too expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper is with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: After the Conclusion
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and baselines are cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "WXzNtT793m", "reviewer": "Reviewer_Tii5", "summary": "The paper shifts the focus from traditional neural network design, which emphasizes feature representation space scaling, to feature interaction space scaling. It introduces a new model architecture, InfiNet, that enables feature interaction within an infinite-dimensional space using the RBF kernel, leading to state-of-the-art results. The paper also discusses the limitations of current models in capturing low-order interactions and proposes the use of classic kernel methods to engage features in an infinite-dimensional space.", "questions": "- How does InfiNet compare to other models in terms of training time and resource consumption?\n\n- Can the kernel methods used in InfiNet be applied to other types of neural network architectures beyond those discussed?\n\n- Can the authors quantify the increased dimensionality of the kernel methods over simpler operations (sum, product). If the authors take the simplest architecture for imagenet and look at the representations generated  by means of using different kernels, can they quantify what is the actual increase in the intrinsic dimensionality of the representation upon training? It is not fully clear to me that the increase in performance is due to an increase in dimensionality.\n\n- the author mention the possibility of exploiting a learnable kernel in place of RBF. Could the author explain and discuss the ratio behind using RBF in place of others? Is it solely driven by the computational complexity. Would the results be different with a different kernel?", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The idea of the paper is simple, novel and well exposed. \n\n- The paper introduces InfiNet, a model architecture that leverages infinite-dimensional feature interactions using RBF kernels, which enhances model performance of traditional models.\n\n- InfiNet achieves new state-of-the-art performance in various tasks, demonstrating the effectiveness of infinite-dimensional interactions.\n\n- The paper includes extensive experiments on datasets like ImageNet and MS COCO, showing the scalability and efficiency of InfiNet.", "weaknesses": "- the paper builds on the simple use of kernel methods. The novelty of the methods is minimal, in the end it is an RBF kernel.\n\n- the performance improvement of Infinet over other models is mostly marginal and no errors have been displayed.\n\n- the paper doesn't really have theoretical novelty", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer Tii5 and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[Weakness 1]**: The paper builds on the simple use of kernel methods. The novelty of the method is minimal, in the end it is an RBF kernel.\n\n**[Re: W1]**: Our novelty lies in the fact that we perform feature interaction in neural networks by means of kernel methods, which is not trivial. We provide an in-depth discussion of the reasons behind our proposed approach, and we point out that the dimensionality of a hidden space generated through feature interactions is an important part of the performance of a model.\n\n\n>**[Weakness 2]**: The performance improvement of Infinet over other models is mostly marginal and no errors have been displayed.\n\n**[Re: W2]**: \n\n**i. Our main contribution is a new perspective of model design instead of a visual system that can beat all the other models.** We design the experiments for the purpose of verifying the effectiveness of our design perspective over models from plain feature superposition and interaction. We follow the architecture and training configuration of widely used models of Swin Transformer for fair comparison. Our goal is to provide a new perspective of model design, but not a SOTA visual recognition system.\n\n**ii. Our performance on ImageNet-1K can be further improved with more refined configuration tuning.** \nWe can manually refine the various hyper-parameters of the training to obtain stronger performance on the ImageNet validation set. We did not do this, and this is because the performance improvement gained in this way is essentially an overfitting of the dataset. Therefore, there is still substantial room to further improve the performance on ImageNet-1K. We think many techniques including architecture optimization, and training methods would further improve the model performance.\n\nThe reason we do not give the error bar is that model training is expensive and it is difficult for us to schedule enough resources to perform multiple rounds of ImageNet-1K training and ImageNet-21K pre-training. For the same reason, this (no error bar is shown) is very common in the deep learning architecture community.\n \n\n>**[Weakness 3]**: The paper doesn't really have theoretical novelty\n\n**[Re: W3]**: We agree that an in-depth theoretical analysis of the proposed method on the interaction space perspective is helpful to better understand our model. But to be honest, currently, we could not find a good theorem to prove rigorously such interaction mechanisms in deep networks, since theoretically analyzing a complex system like InfiNet is very difficult. However, we still have some intuitive and empirical analysis to show the methodology and philosophy behind our feature interaction perspective. We hope such analysis can help readers to better understand our motivation and provide some guidance to design better architectures in future research.\n\n>**[Question 1]**: How does InfiNet compare to other models in terms of training time and resource consumption?\n\n**[Re: Q1]**: In terms of training time. We take an example of InfiNet-T level models on ImageNet-1K training with the same configuration on 4 $\\times$ A100.40GB based on our test.\n\nModel| Training Time(min/epoch)|\n|---|---|\n|ConvNeXt|22|\n|Swin|27|\n|HorNet|31|\n|MogaNet|43|\n|InfiNet|30|\n\nAs the model increases, the model width gradually increases and the FFN layer will occupy more computational load. The difference in training time for these models will be smaller.\n\nWe notice a large amount of data reuse in InfiNet in the multi-branch structure before the kernel. Therefore, the memory access bandwidth is the limiting factor in our model training. We substantially optimized the computation time by using fully equivalent grouped convolutions instead of computing depth-width convolutions in a round-robin fashion. This reduces the training time per epoch from 54 minutes to 30 minutes versus a substantial reduction through cyclic depth-width convolution.\n\nBecause of the lack of innate computational optimization, is the main reason for the current slowness compared to ConvNeXt and Swin, but our model training is still faster than most mainstream High-Order networks. The model suffers from load imbalance during computation. However, these problems are solvable and optimizable, and this is the goal of our subsequent work to further optimize the model computation.\n\n\n>**[Question 2]**: Can the kernel methods used in InfiNet be applied to other types of neural network architectures beyond those discussed?\n\n**[Re: Q2]**: Yes! In fact, element-wise multiplication in any network architecture can be attempted to be replaced by the kernel method, under consideration of the convergence of the model. This includes state-space models (aka Mamba) (Gu, Albert, et al. 2023), HorNet (Rao et al. 2022), gated convolution (Dauphin, et al. 2017), and so on. Other than that, we are trying to use the kernel method on xLSTM (Beck M. et al. 2024).\n\n>**[Question 3]**: Can the authors quantify the increased dimensionality...\n\n**[Re: Q3]**: We give a quantify ablation study on Sec 6.4 and Table 3(b).  The intrinsic dimensionality can be calculated with Eq.5 with n=HWK^2 and k = interaction order in Table 3(b), where H/W is the height and width of the model and K is the convolution kernel size. In Table 3(b), we can see that as the interaction order increases, the intrinsic dimensionality increases and the performance improves. Thus, we can see a positive correlation trend that performance increases with dimensionality.\n\n>**[Question 4]**: About the kernel selection.\n\n**[Re: Q4]**: The reason we use the RBF kernel in our model is: (1) The RBF kernel is the simplest kernel function that can realize infinite-dimensional feature interaction. （2） The empirical results. We show the result of using a linear kernel and monomial kernels in Table 3(b).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jeOnTzlXsd", "reviewer": "Reviewer_Jbm2", "summary": "The authors present a new architecture for computer vision applications that models high-order interactions between features. The architecture is similar to an attention block, but introduces an RBF Kernel layer that captures interactions of order higher than two. The resulting method has strong empirical performance across image classification tasks.", "questions": "- What are some examples of features and interactions that help learning and that the new model can learn?\n- Is it possible to analyze or visualize what interactions the model learned?", "limitations": "I do not see any ethical and societal implications of the work that need to be discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of the paper is very interesitng and novel.\n- The empirical results show promising performance across involve multiple tasks against sophisticated methods", "weaknesses": "- The presentation of the method seems overly complex in some places. For example, providing a clearer explanation of each new layer (perhaps in pseudocode) would help. While the Infiniblock definition is clear, the reader needs to go back to the previous section to understand the input/output shaped of the RBF layer, which takes work, and can be made simpler. Making clearer the intuition behind high-order interactions would be helpful as well. Showing examples of what the model learns would be helpful to make things concrete.\n- The empirical performance is reasonably similar to those of previous methods, hence the empirical improvement is not that large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer Jbm2 and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[Weakness 1]**: The presentation of the method seems overly complex in some places. For example, providing a clearer explanation of each new layer (perhaps in pseudocode) would help. While the Infiniblock definition is clear, the reader needs to go back to the previous section to understand the input/output shape of the RBF layer, which takes work, and can be made simpler. Making clearer the intuition behind high-order interactions would be helpful as well. Showing examples of what the model learns would be helpful to make things concrete.\n\n**[Response to W1]**: \n\n(1) We adopt the diagrammatic and layer-wise formulaic representations commonly used in articles in the field of deep learning architecture. We will add input/output shapes for each layer in subsequent versions of Fig.3 to make it easier to understand. We have open-sourced our code, which will further help in the understanding of the model we design. However, due to the NeurIPS rebuttal policy, we cannot provide you with a direct link. Instead, as per NeurIPS rules, we have sent an anonymous link to AC.\n\n(2) The intuition for high-order interactions is that new research suggests that high-order interactions are widespread in biological systems, neuroscience, and physical-social systems[1]. We believe that such high-order interactions are clearly present in neural network features as well. The idea of interaction-inspired designs like HorNet and MogaNet, but their models are limited to exploring simple interactions, whereas our model can explore a larger interaction space.\n\n（3）We show some cases in Figure 4 in the Appendix that demonstrate the difference in the Class Activation Mapping feature regions learned in the feature representation space, the finite simple interaction space and the infinite-dimensional interaction space.\n\n>**[Weakness 2]**: The empirical performance is reasonably similar to those of previous methods, hence the empirical improvement is not that large.\n\n**[Response to W2]**: \n\n**i. Our main contribution is a new perspective of model design instead of a visual system that can beat all the other models.** We design the experiments on the purpose of verifying the effectiveness of our design perspective over models from plain feature superposition and interaction. We follow the architecture and training configuration of widely used models Swin Transformer for fair comparison. Our goal is to provide a new perspective of model design, but not a SOTA visual recognition system.\n\n**ii. Our performance on ImageNet-1K can be further improved with more refined configuration tuning.** \nWe can manually refine the various hyper-parameters of the training to obtain stronger performance on the ImageNet validation set. We did not do this, and this is because the performance improvement gained in this way is essentially an overfitting of the dataset. Therefore, there is still substantial room to further improve the performance on ImageNet-1K. We think many techniques including architecture optimization, and training methods would further improve the model performance.\n\n\n>**[Question 1]**: What are some examples of features and interactions that help learning and that the new model can learn?\n\n**[Response to Q1]**: We give some examples in Figure 4. We can observe that the region responded to by the model that employs kernel methods better fits its actual class. This shows that our method is more effective in extracting effective features, which is largely due to the fact that the interaction of the target's features allows the target to be learned as a whole, thus enhancing the model to a certain extent.\n\n\n>**[Question 2]**: Is it possible to analyze or visualize what interactions the model learned?\n\n**[Response to Q2]**: We believe that the commonly used methods such as GradCAM, LRP, etc. can be directly used for the visualization and analysis of InfiNet. However, it is worth noting that due to the multi-branch structure of InfiNet's features and the introduction of mixing of two branches of activation maps by kernel methods, it is possible that relevance scores are not successfully propagated back to the inputs, resulting in only providing the partial information of relevance. \n\nWe think that an interpretable visualization of InfiNet‘s interaction is a very complex task that still requires a great deal of effort to achieve. We give some basic ideas for possible visualization of interactions. (1) We can try to anchor a region of interest and obtain a heat map of the regions that are co-interacting with it by using the gradient method. (2) We can obtain the coefficients of the interactions in the region corresponding to the activations by Taylor expansion of the activations at each level of the kernel function, and construct a statistical map.\n\n\n\n\n\n[1] Battiston F, Amico E, Barrat A, et al. The physics of higher-order interactions in complex systems[J]. Nature Physics, 2021, 17(10): 1093-1098.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ZVWY9Akpu", "reviewer": "Reviewer_USts", "summary": "This paper studies placing a kernel function inside of a neural network architecture to facilitate interaction of features/dimensional expansion. They consider deep convolutional networks with parallel pathway features $x$ and $x'$ and a kernel function computed with both pathways' features as inputs $k(x,x')$. Standard kernel mathematics is used to explain feature expansion. The main novel results are empirical performance of these \"InfiNet\" architectures, which are shown to perform well in a number of computer vision tests.", "questions": "* There is a tension between dimensional expansion, which leads to expressivity in networks, and generalization, which is typically better in low-dimensional settings. Can you discuss this?\n* When queries and keys in a transformer are computed using a multilayer network with nonlinearities (rather than a single linear layer, as you've considered), aren't the effective order of interactions higher?\n* You claim that the kernel map applied to inputs with $C$ channels takes constant $O(1)$ time (section 4 intro). Wouldn't evaluating the kernel still take $O(C)$ i.e. linear time?\n* Can you please include the matrix/tensor shapes and layer sizes explicitly in section 5.1? They could be put into the appendix. It is unclear how many kernel evaluations are performed and on what shape input.\n\nMinor points:\n* Sentence lines 45-47 is confusing and should be reworded. Also, the combinatorial expression with the limit is unexplained, not obvious, and doesn't seem to contribute anything here. I suggest removing it.\n* Line 61, the expression for span of a certain space is unclear. The main point seems to be that this is an infinite-dimensional function space. Does using this math really add anything?\n* Line 61: \"as low-overhead as exponential operations\" is unclear. Do you mean \"evaluating an exponential function\"?\n* Line 91: \"Kernel Method\" -> \"Kernel Methods\" typo\n* Line 106: \"isotropic\" here is unclear to me, suggest removing\n* Line 110: \"medium\" for the intermediate layer connotes different size, suggest changing to \"intermediate\" or \"middle\"\n* Line 130: Without saying it, are you assuming that the image inputs span the pixel space vector?\n* Line 149: \"two element-wise multiplication\" typo -> \"multiplications\"\n* Notation $W_a \\mathbf{x}$ is confusing: In equation (1) this seems to output a scalar. Is that the same in Eqn (6)? What are the shapes of the W matrices?\n* What is a \"feature branch\"? Unclear throughout.\n* You say the input is passed through \"STEM\" and refer people to the ConvNeXT paper https://arxiv.org/pdf/2201.03545. There is more than one \"stem\" in that paper. Can you be explicit about what you did?", "limitations": "I would strongly prefer that the limitations be included in the main text during the discussion. With movement of some of the standard math, there would be space.\n\nThe authors only consider the squared exponential kernel with bandwidth parameter equal to 1. Other kernels might work better. In particular, the effective dimensionality of the RKHS (related to the kernel decay rates) would be higher with a \"less smooth\" kernel like the exponential/Laplace kernel.\n\nThe results are likely not reproducible unless the authors release their code.\n\nThe results are also limited only to supervised vision tasks, rather than other modalities or unsupervised settings.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of unifying different orders of interaction embodied in various neural network architectures, including Transformers is appealing and probably important. The accuracy of the InfiNet experiments is impressive, with a moderate reduction in FLOPs. The paper is easy to read and well-organized, although suggestions are given for how it could be improved.", "weaknesses": "My main concerns with the paper are a lack of context for the approach as well as missing important explanations. I also think a good amount of the math that's included could be considered \"filler\" material that could go into the appendix, since it doesn't represent new results. (I am referring to sections 4.1 and 4.2, most of which can be found in most textbooks which cover kernel methods.)\n\n* Notation which is commonly used in the paper $\\oplus$, $\\otimes$, * is not explained. You should _explicitly_ define it somewhere, at least in the appendix (and refer people there). In particular, people may be confused by * for elementwise/Hadamard multiplication, since in convnet literature this is often the convolution operator. You call this the \"Star Operation\" in line 124, but I think it is just elementwise multiplication.\n* The authors seem to have missed the vast literature on the connections between random features, neural networks at init, and kernel methods. (CKNs are mentioned but without any discussion of the topics I mention here.) In particular, one way that you could approximate the InfiNet architecture would be to take the two feature streams and pass them each into the same wide, random network/layer and compute the dot product of features at the next level. That would only approximate the kernel function in the InfiNet architecture, and is likely less efficient, but it provides a way to perform dimensionality expansion with a more traditional layer. The authors should discuss these connections.\n* Different order of interactions have been studied in random feature and kernel settings already. In random features, interaction order is connected to the sparsity of weights, see e.g. https://arxiv.org/abs/2103.03191 and https://arxiv.org/abs/1909.02603. In kernels, this were referred to as additive kernels https://arxiv.org/abs/1602.00287, also studied in multiple kernel learning https://arxiv.org/abs/0809.1493 (these are just some examples among a larger literature).\n* The authors do not seem to want to release their code. They have said \"Yes\" on Question 5, stating that the code and data are open, but there is no link or indication in the text that the code is available or will be when the paper is published. That seems deceptive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer USts and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[Weakness 1]**: Notation is not explained...\n\n**[Re: W1]**: We'll add a detailed explanation of notation in a subsequent release. The symbol $\\oplus$ denotes the Direct Sum of vector spaces. This corresponds to structures such as channel expansion/bottleneck in commonly used neural networks. While the $\\otimes$ is elementwise multiplication. Some of the expressions in our text use Star Operation (*). This is intended to be aligned with reference [1], which is essentially elementwise multiplication. We realize that we have overlooked the consistency of notation in the paper, and we will fix this in a subsequent version.\n\n>**[Weakness 2]**: About literature on the connections between random features, neural networks at init, and kernel methods.\n\n**[Re: W2]**: Thank you for your advice. We will add the discussion on the connections between random features, neural networks and kernel methods. \n\nThe NTK framework establishes a direct connection between infinitely wide neural networks at initialization and kernel methods. Specifically, it shows that as the width of the network grows, the network's training dynamics can be described by a kernel function (the NTK), linking the neural network's behavior to that of kernel methods. Random features provide an efficient way to approximate the feature mappings used in kernel methods. By using random projections, one can approximate the inner product defined by a kernel function, making it feasible to apply kernel methods. Our contribution is that we look at the relationship between kernel methods and neural networks in a different light. Instead of examining the dynamics of neural networks from the standpoint of the kernel, we consider how kernel methods can benefit neural networks from the perspective of expanding the interaction space.\n\nWe appreciate your advice on using random features to approximate InfiNet. We recognize that this is something we haven't thought about in the past, and we think it's a worthy discussion for future work. Meanwhile, we believe that our work will be an important milestone in the development of this field.\n\n>**[Weakness 3]**: Different orders of interactions have been studied in random feature and kernel settings already. ...\n\n**[Re: W3]**: We appreciate your references to research on different orders of interaction in the fields of random features and kernel methods. We will include a section to discuss them. However, we believe there is still a gap between the research in these areas and the research on interaction in NN design. Therefore it is difficult to derive a methodology from these studies that can be directly applied to NN design. Our work looks at interactions in the design of modern NN architectures in the hope of finding a new design consideration.\n\n\n>**[Weakness 4]**: About the code open source.\n\n**[Re: W4]**: We release the code recently. However, due to the NeurIPS rebuttal policy, we cannot provide you with a direct link. Instead, as per NeurIPS rules, we have sent an anonymous link to AC.\n\n>**[Question 1]**: Tension between dimensional expansion and generalization.\n\n**[Re: Q1]**: In neural network research, the prevailing view is that such a tension does not exist. This is due to the presence of the double descent phenomenon which challenges the traditional bias-variance trade-off perspective. Double Descent suggests a positive correlation between dimensional expansion and generalization when over-parametrized[2].\n\n>**[Question 2]**: About effective order of interactions of transformers\n\n**[Re: Q2]**: An interesting question, and we think the answer is yes. There have been studies of neural network width/depth equivalence in the past, and we think the order of interaction can be added to that. A quantitative study of this issue is more complex and requires more effort.\n\n>**[Question 3]**: About the claim of constant time.\n\n**[Re: Q3]**: Thanks for pointing out that we ignored the complexity of the kernel itself here. If we consider the complexity of the kernel itself, it's supposed to be linear time. \n\n>**[Question 4]**: About including the matrix/tensor shapes.\n\n**[Re: Q4]**: We will further refine the description of the tensor shape in camera-ready. As in Fig.3(b), each rounded rectangular box does not change the tensor shape. And parallel boxes will create independent copies. The two branches of the kernel will each be r (7 in our implementation) times the size of the tensor.\n\n**[Re: Minor Points]**: Due to character limitations, we combine responses for minor points. (1)We have noticed that the description of the paragraph in lines 45-49 may not be clear, and we will reorganize the description. (2)(3) we will remove the redundant expression and clarify that the in kernel method, the computation overhead is the same as evaluating an exponential function. (4)(6)(8)We will fix the typo. (5)Isotropic means constant shape. (7) Yes, we consider it as a basic assumption in computer vision. (9) No, W in Eq.1 is an n-d vector, and in Eq.6 is a matrix. We will change some notation to make it more clear.  (10) feature branch means a multi-branch structure like in ResNeXt. (11) STEM refers to the same patchify stem (4*4 non-overlapping convolution).\n\n**[Re: Limitations]**: We will tweak some paragraphs of the article to discuss limitations. We will try more kernels to enhance the described approach. The initial code is open source now. Our work on language is in progress but still needs engineering effort. The performance in unsupervised settings is still unknown, but we are confident that InfiNet has a strong representation.\n\n[1] Ma, et al. \"Rewrite the Stars.\" CVPR 2024\n\n[2] Nakkiran, et al. \"Deep double descent: Where bigger models and more data hurt.\" Journal of Statistical Mechanics: Theory and Experiment 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wNS1YkuPLA", "reviewer": "Reviewer_u52R", "summary": "This work proposes a novel approach for enhancing neural network performance by scaling feature interaction spaces to infinite dimensions using kernel methods. Recent advancements have introduced feature interaction spaces, but these are often limited to finite dimensions, primarily through element-wise multiplications. To overcome these limitations, the authors propose InfiNet, a model architecture leveraging the Radial Basis Function (RBF) kernel to enable infinite-dimensional feature interactions. Finally, the authors provide several empirical results on standard vision tasks.", "questions": "* what is meant in line 47 + 48? the current formulation is very cryptic. What exactly is linear in $k$?\n\n\n* figure 2: why does the addition and multiplication interactions reach the same accuracy on cifar10? Isn't that basically MLP vs self-attention? I would presume self attention to perform better.", "limitations": "* The paper provides empirical results only on vision tasks. However, a major selling point of this paper is generalizing approaches like self attention in terms of feature-feature interactions. Therefore, comparisons with transformers on language tasks should be performed. \n\n* No theoretical analysis is provided proving that the proposed method leads to better generalization.\n\n* The method appears to have on average lower FLOPs than competing methods, while at the same time only marginally outperforming (or even performing worse than) competing methods on imageNet.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work provides an interesting generalization of feature-feature interactions via kernels. For the best of my knowledge, this is a novel idea that appears to perform well in practice. However, I am not overly familiar with the current state of the field of deep learning for computer vision. It further provides several larger-scale experiments and interesting ablations.", "weaknesses": "* there is no theoretical justification that increasing the dimension of the feature-feature interaction space will lead to better generalization. The paper does a good job analysing this question with ablations. However, this remains an open theoretical question.\n\n* I understand that the motivation for this work comes from applications in computer vision. However, since a major focus in this paper is on comparing the proposed approach to self attention, it would be interesting to not only test this method on images, but also on language. \n\n* the method is reported to have lower FLOPs on average than competing methods. Why is that? Is that a major drawback of this method?\n\n* performance improvement on ImageNet is only marginally. In many cases the proposed method even performs worse than competing methods.\n\n* paragraph starting in line 148: this is on over-claim and has to be removed or rigorously proved. It is not clear how a higher order of $k$ implies better generalization or training. Unless shown in this paper or referenced from another paper, this has to be removed.\n\nMinor: \n\n* line 28: more context for formulating self attention that way has to be provided. It is explained in more detail only at the end of section 3. \n\n* caption of figure 2: there is '?'. Moreover, a description of the presented images should be included. What is shown in Figure 2 on the right hand side? This is only explained in the main text,not the caption. This needs to be changed.\n\n* figure 2, first image on the left: hard to read -- text overlaps with drawing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer u52R and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[Weakness 1]**: There is no theoretical justification that increasing the dimension of the feature-feature interaction space will lead to better generalization. The paper does a good job analysing this question with ablations. However, this remains an open theoretical question.\n\n**[Re: W1]**: We agree that an in-depth theoretical analysis of the proposed method from the interaction space perspective would enhance our understanding of the model's generalization ability. However, we must acknowledge that we have not yet found a rigorous theorem to fully prove such interaction mechanisms in deep networks. Theoretical analysis of complex systems like InfiNet presents significant challenges. To our knowledge, no existing theorem comprehensively analyzes the generalization ability of advanced neural architectures, such as those incorporating self-attention or gated convolution. Nonetheless, we offer some intuitive and empirical analyses to illustrate the methodology and philosophy underpinning our feature interaction perspective. We hope that these analyses will help readers better understand our motivations and provide guidance for designing improved architectures in future research work.\n\n>**[Weakness 2]**: About testing on language.\n\n**[Re: W2]**: Computer vision models are generally similar to encoder language models. So, we are testing a BERT-like design with InfiNet's architecture on language. InfiNet works on the Encoder model, and it still needs engineering effort. In a GPT-like decoder-only architecture, like most vision models, InfiNet cannot be directly applied due to the limitations of auto-regressive modeling. However, our idea of utilizing kernel methods can be applied in network architecture design, and we are trying to combine the kernel method with xLSTM to construct a new language model.\n\n>**[Weakness 3]**: the method is reported to have lower FLOPs on average than competing methods. Why is that? Is that a major drawback of this method?\n\n**[Re: W3]**: The FLOPs refers to the count of floating point operations in the deep learning architecture area. It represents the amount of computation required by the model. This is a metric where smaller values are better, and our model significantly outperforms others in this regard. Is not a drawback instead an advantage of this method.\n\n>**[Weakness 4]**: performance improvement on ImageNet is only marginally. In many cases the proposed method even performs worse than competing methods.\n\n**[Re: W4]**:\n**i. Our main contribution is a new perspective of model design instead of a visual system that can beat all the other models.** We design the experiments for the purpose of verifying the effectiveness of our design perspective over models from plain feature superposition and interaction. We follow the architecture and training configuration of widely used models Swin Transformer for fair comparison. Our goal is to provide a new perspective of model design, but not a SOTA visual recognition system.\n\n**ii. Our performance on ImageNet-1K can be further improved with more refined configuration tuning.** \nWe can manually refine the various hyper-parameters of the training to obtain stronger performance on the ImageNet validation set. We did not do this, and this is because the performance improvement gained in this way is essentially an overfitting of the dataset. Therefore, there is still substantial room to further improve the performance on ImageNet-1K. We think many techniques including architecture optimization, and training methods would further improve the model performance.\n\n>**[Weakness 5]**: paragraph starting in line 148: this is on over-claim and has to be removed or rigorously proved. It is not clear how a higher order of  implies better generalization or training. Unless shown in this paper or referenced from another paper, this has to be removed.\n\n**[Re: W5]**: This is an association with Weakness 1. We will modify the wording here. We will state our view of this in the form of a conjecture/hypothesis instead of an explanation. Some relevant support is given in [1], but a complete theory of such a complex system still needs more effort.\n\n**[Response to Minor Points]**: (1) We will provide more context to make the formulating more self-contained in Section 1. (2)(3) The \"?\" here corresponds to the one in the figure and represents a replaceable operation. We will change the caption and figure for readability. \n\n>**[Question 1]**: what is meant in line 47 + 48? the current formulation is very cryptic. What exactly is linear in k?\n\n**[Re: Q1]**: This refers to the fact that if the high-order interaction is realized by changing the design of the model architecture, the computational cost will increase linearly with the size of the order k that one wants to achieve, because of the need for elementwise multiplication of k groups of features.\n\n>**[Question 2]**: figure 2: why does the addition and multiplication interactions reach the same accuracy on cifar10? Isn't that basically MLP vs self-attention? I would presume self attention to perform better.\n\n**[Re: Q2]**: This is due to the small size of the CIFAR10 dataset. In fact, Transformer is considered to show some advantages over convolution only on large-scale datasets[3].\n\n**[Response to Limitation]**: See response to W2, W1 and W3\n\n[1] Wu, Yongtao, et al. \"Extrapolation and spectral bias of neural nets with hadamard product: a polynomial net study.\" NeurIPS 2022\n\n[2] Rao, Yongming, et al. \"HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions.\" NeurIPS 2022\n\n[3] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Area Chair and Reviewers,

We appreciate reviewers' precious time and valuable advice. We are happy that most of reviewers acknowledged our novel idea (Tii5, Jbm2, USts, u52R) and experiments (Tii5, Jbm2, USts, u52R).

At the same time, we note the concerns and suggestions of the reviewers on our work. We provide detailed answers to all the questions raised by the reviewers in the following individual responses. We hope these responses address your questions and concerns well. If you still have questions and concerns, we appreciate you discussing them further with us!

Since NeurIPS 2024 does not allow for the submission of revised papers at the rebuttal stage, we are committed to incorporating changes based on the comments of the reviewers in subsequent revised release. 

We would like to thank the reviewers again for their valuable comments on our paper and for their time in reviewing and discussing it! 

Best Regards,

Authors

Meta-review: This paper introduces and investigates an architectural component for infinite-dimensional feature interactions based on an RBF kernel. Extensive experiments show strong performance across a wide range of computer vision tasks. Although some of the reviewers noted a lack of theoretical development or justification, they found the method to be novel and the results promising. Regarding novelty, one relevant reference that did not emerge in the discussion is Choromanski, Krzysztof, et al. "Rethinking attention with performers." arXiv preprint arXiv:2009.14794 (2020), which also investigates infinite-dimensional feature interactions, albeit with a different focus and perspective. The authors should be sure to discuss this paper in their next revision. In any case, the present work will be of significant interest to the community and I recommend acceptance.
Decision: Accept (poster)