Multilinear Operator Networks
Authors: Yixin Cheng
Grigorios Chrysos
Markos Georgopoulos
Volkan Cevher
Venue: ICLR 2024 poster (2024)
Keywords: Polynomial Networks
Image recognition
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce a network based solely on multilinear operations.

Abstract: Despite the remarkable capabilities of deep neural networks in image recognition, the dependence on activation functions remains a largely unexplored area and has yet to be eliminated. On the other hand, Polynomial Networks is a class of models that does not require activation functions, but have yet to perform on par with modern architectures. In this work, we aim close this gap and propose MONet, which relies *solely* on multilinear operators. The core layer of MONet, called Mu-Layer, captures multiplicative interactions of the elements of the input token. MONet captures high-degree interactions of the input elements and we demonstrate the efficacy of our approach on a series of image recognition and scientific computing benchmarks. The proposed model outperforms prior polynomial networks and performs on par with modern architectures. We believe that MONet can inspire further research on models that use entirely multilinear operations.

Published as a conference paper at ICLR 2024
MULTILINEAR OPERATOR NETWORKS
Yixin Cheng', Grigorios G. Chrysos? Markos Georgopoulos, Volkan Cevher
LIONS École Polytechnique Fédérale de Lausanne Unniversity of Wisconsin-Madisse
ABSTRACT
Despite the remarkable capabilities of deep neural networks in image recognition, the dependence on activation functions remains a largely unexplored area and has yet to be eliminated. On the other hand, Polynomial Networks is class of models that does not require activation functions, but have yet to perform on par with mod- ern architectures. relies solely on multilinear operators. The core layer of MONet, called Mu-Layer, captures multiplicative interactions of the elements of the input token. MONet captures high-degree interactions of the input elements and we demonstrate the efficacy of our approach on series of image recognition and scientific computing benchmarks. The proposed model outperforms prior polynomial networks and performs on par with modern architectures. We believe that MONet can inspire further research models that entirely multilinear operations. The source code is available at [ONei
this work, we aim close this gap and propose MONet, which
INTRODUCTION
Image recognition has long served as a crucial benchmark for evaluating architecture designs, includ- ing the seminal ResNet (He et al. 2016) and MLP-Mixer (Tolstikhin et al. [021). As architectures applied to new applications, there are additional requirements for the architecture design. For instance, encryption is key requirements in safety-critical applications (Caruana al. 2015). Concretely, the Leveled Fully Homomorphic Eneryption (LFHE) (Brakerski et al. 2014), can provide a high level of security for sensitive information. The core limitation of FHE (and especially LFHE) is that they support only addition and multiplication as operations. That means that traditional neural networks cannot fit into such privacy constraints owing their dependence on elementwise activation functions, making developments in MLP-Mixe and similar models invalid for many real-world applications. Therefore, new designs that can satisfy those constraints and still achieve high accuracy on image recognition are required.
core advantage of Polynomial Nets (PNs), that express the output as high-degree interactions between the input elements (Ivakhnenko) 1971, (Shin & Ghosh, 1991; Chrysos al. 2020), is that they can satisfy constraints, such as encryption or interpretability (Dubey al. 2022). However, a major drawback of PNs far is that they fall short of the performance of modern architectures standard machine learning benchmarks, such as image recognition. This is precisely the gap we aim to close in this work.
We introduce a class of PNs, dubbed Multilinear Operator Network (MONet), which is based solely multilinear operations[. The core layer captures multiplicative interactions within the token elements The multiplicative interaction is captured using two parallel branches, each of which assumes a different rank enable different information to flow. By composing sequentially such layers, the model captures high-degree interactions between the input elements and can predict the target signal, e.g.. class in the case of image recognition.
Concretely, our contributions can be summarized as:
'The terminology multilinear operations arises from the multilinear algebra. Concretely, we follow the terminology of the seminal paper of Kolda & Bader (009).
"Consistent with the recent literature of MLP-based models and transformers (Dosovitskiy et al. 2021 we consider sequences of tokens as inputs. In the case of images, token refers to (vectorized) patch of the input image.

---

Published as a conference paper at ICLR 2024
We propose Mu-Layer, module which uses purely multilinear operations to capture multiplicative interactions. We showcase how this module can serve as a plug- in replacement to standard MLP.
We introduce MONet, which captures high-degree interactions between the input elements. To our knowledge, this is the first network that obtains strong performance on challenging benchmarks.
We conduct thorough evaluation of the proposed model across standard image recognition benchmarks to show the efficiency and effectiveness of our method. MONet significantly improves the performance of the prior art on polynomial networks, while it is on par with other recent strong peroonpprfonmiggggg architectures.
The source code is available at MONel We hope the code can enable further improvement of models relying on linear projections.
2 RELATED WORK
We present brief overview of the most closely related categories of MLP-based and polynomial- based architectures from the vast literature of deep learning architectures. For detailed overview, the interested reader can find further information on dedicated surveys image recognition (Lu & Weng 2007: Plested & Gedeon 2022 Peng et al. 2022).
MLP models: The resurgence of MLP models in image recognition is attempt to reduce the computational overhead of transformers (Vaswani et al. 2017). MLP-based models rely on linear layers, instead of convolutional layers or the self self-attention block. The MLP-Mixer (Tolstikhin et al., 202T) is among the first networks that demonstrate a high accuracy on challenging benchmarks. The MI.-Mixer uses tokens (i.e., vectorized patches of the image) and captures both inter- and intra-token correlations. Follow-up works improve upon the simple idea of token-mixing MLP structure (Tourron et al.) [002 Liu al. 2021). Concretely, ViP (Hou al. (022), cycleMLP (Chen iet al. 2022), S2-MLPv2 Yu et al. (2022a) design strategies to improve feature aggregation across spatial positions. Our work differs from MLP- models, is inspired by the idea of capturing high-order interactions using polynomial expansions.
Polynomial models: Polynomial expansions establish connections between input variables and learnable coefficients through addition and multiplication. Polynomial Nets (PNs) express the output variable (e.g., class) as a high-degree expansion of the input elements (e-g., pixels of image) using learnable parameters. Even though extracting such polynomial features is not a new idea (Shin &
Ghosh 1991 Li 2003 Dauphin et al.) 2017). it has been largely sidelined from architecture design.
The last few years PNs have demonstrated promising performance in standard benchmarks in various vision tasks including image recognition (Chrysos et al., (002)). In particular, PNs augmented with activation functions, which are referred as hybrid models in this work, can achieve state-of tet-thh-nr art performance (Hu et al.) [2018] Li et al.] 2019 Yin et al.) 2020; Babiloni et al. [002]] Yang et al.) 2022) al. 202 The work of Chrysos
Georgopoulos et al.] 2020)] Chrysos al. 2021 Georgopoulos et al. (2022) introduces a taxonomy for a single block of PNs based on the degree of interactions captured. This taxonomy allows the comparison of various approaches based on specific degree of interactions. Building upon this work, researchers have explored ways to modify the degree or type of interactions captured to improve the performance. For example, Babiloni al. (202T) reduce computational cost of the popular non-local block (Wang et al. 20188) by capturing exactly the same third degree interactions, while Chrysos et al. (2022) investigate modifications the degree of expansion.
Arguably, the works most closely related to ours are Chrysos al. 2020 2023). In II-Nets (Chrysos et al. (020), three different models are instantiated based on separate manifestations of three tensor decompositions. That is, the paper establishes a link between a concrete tensor decomposition and its assumptions to a concrete architecture that can be implemented in standard frameworks. The authors evaluate those different architectures and notice that they perform well, but do not surpass standard networks, such as ResNet. Chrysos et al. (2023) improve upon the II-Nets by introducing carefully designed (implicit and explicit) regularization techniques, which reduce the overfitting of high-degree polynomial expansions.
2

---

Published as a conference paper at ICLR 2024
Fully-Connecedd
Connecioo
Average Pooling
Mu Layer
Poly Block
N (Poly Block)
Pyramid Patch Embedding with Linear Projection
MoNett Architecture
The architecture of the proposed Mu-Layer (on the left) and MONet (on the right). In the left figure, the grey box represents layer normalization. The color solid line boxes represent channel projection in different dimensions, all projection operations linear. The box denotes elementwise (Hadamard) product. The red dash box represents the spatial aggregation module.
Figure I:
Contrary the aforementioned PNs, that is comparable to modern networks. To achieve that, we are inspired by the modern setup of considering the imput as a sequence of tokens. The token-based input widely used across range of domains and modalities that last few years. The token- input also departs from the design of previous PNs that utilize convolutional layers instead of simple matrix multiplications, Arguably, our choice results in a weaker inductive bias as pointed out in the related works of MLP-based models. This can be particularly useful domains outside of images, e.g., in the ODE experiments.
motivated introduce new architecture using PN
3 METHOD
Let us now introduce MONet, which expresses the output as polynomial expansion of the input using multilinear operations. Firstly, we introduce the core layer, called Mu-Layer, in Section 3.1 Then, in Section 3.2, we design the whole architecture with schematic visualized Fig.
Tokens:
Following the conventions of recent MLP-based models, we consider an image sequence of tokens. practice, each token patch of the image. We denote the sequence of tokens as X E Rdxx, where d N is the length of a token and N is the number of tokens. As such, our method below assumes an input sequence X provided. MLP-based models capture linear interactions between the elements of token. That operation would be denoted rX, where E Roxd a learnable matrix.
3.1 MU-LAYER
Our goal is capture rich interactions among the input elements of token. We achieve that using multiplicative interactions. Specifically, we use two branches, and each captures a set of linear interactions inside the token. An elementwise product is then used to capture the multiplicative interactions between the branches. Notation-wise, the output is expressed as (AX) x (AX), where denotes the elementwise product and A, A are learnable matrices.
We perform the following three modifications to the aforementioned block. Firstly, we add a shortcut connection capture the first-degree interactions. Secondly, decompose A into two matrices as
A = BD. This rank factorization of A enables us to capture low-rank representations in this branch. Thirdly, add one matrix C capture the linear interactions of the whole expression. Then, the complete layer is expressed as follows:
Y C ((A) (BDX) AX),
(1)
3

---

Published as a conference paper at ICLR 2024
Table 1: Configurations of different MONet models. We present four models: two variants with different parameter-size (Tiny, Small) and two variants with different hidden-size. The original model use the same hidden-size across the architecture. The Multi- model adopts a different hidden sizes across the architecture. The number in stages represents the number of blocks that each stage includes, while the hidden size list numbers correspond to each stage's hidden size. We provide further details along with schematic in Appendix B high-resolution image classification benchmarks, c.g., on ImageNetIK.
We only use Multi-stage models for
Specification
Thy
Small
Multi-stage-Tin Tiay
Multi-stage- stage-Small
Numbers Blocks Embedding Method
32
45
32 Pyramid Patch Embeddinng 12, 4.8. 12,11
32 Pyramid Patch Embedding 128, 192,256,384
Pyramid Patch Embodding Pyramid Patch Embedding
Hidden size
192
320
Stuges Expansion Ratio
4,,,12,14
3
3
Shrinkage Ratio
4
8
Parameters FLOPs
14.0M
56.8M
10.3M
32.9M
3.6
11.2
2.8
6.8
where A E R B E R Roxm D E Rixd are learnable parameters and * symbolizes an elementwise product. The block has two hyperparameters we can choose, i.e., the rank m E N and the value E N. In practice, we utilize a shrinkage ratio zht >> 1 to encourage different information flow in the branches. Prop.[] the proof of which is in Appendix [A] indicates the interactions captured. The schematic of this layer, called Mu-Layer, is depicted in Appendix
Proposition 1. The Mu- captures multiplicative interactions between elements of each token.
The Poly-Block, which is the core operation of MONet architecture, stacks Mu-Layers sequentially. Concretely, we connect two Mu-Layers sequentially and add a a layer normalization (Ba) et al. 2016) between O Additionally, we add shortcut connection to skip the first block. The two Mu-Layers are similar except for two differences: (a) a spatial shift operation (Yu et al. 2022b) is added to the first block, (b) the hidden dimension is m in the first Mu-Layer and in the second, where e is an expansion factor. The Poly-Block is illustrated in Fig.
Poly-Block:
3.2 NETWORK ARCHITECTURE
Our final architecture is composed of N Poly-Blocks. Each block captures up to 4thh degree interac- tions, which results in the final architecture capturing up to 4.V interactions, with N 10 in practice") Beyond the Poly-Block, one critical detail in our final architecture is the patch embedding.
Patch embeddings: In deep learning architectures, the choice of patch size plays crucial role in capturing fine-grained features. Smaller patches prove more effective in this regard. However, a trade-off arises, as reducing the patch size leads to an increase in floating-poins operations, which scales approximately quadratically. To overcome this challenge, we introduce a novel scheme called pyramid patch embedding. This scheme operates by considering embeddings at smaller scales and subsequently extracting new patch embeddings on top. By adopting this approach, we can leverage the finer details of features without introducing additional parameters or computational complexity. Our experiments validate the effectiveness of incorporating multi-level patches, enhancing the overall performance. For the ImageNetIK K dataset, we employ a two-level pyramid scheme, although is worth noting that higher-resolution images may benefit from additional levels. We present the schematic and further details of this approach in Appendix D
4 EXPERIMENTS
In this section, we conduct thorough experimental validation of MONet. We conduct experiments on large-scale image classification in Section 4.1 fine-grained and small-scale image classification
We also conduct experiments with batch normalization and obtain similar performance; results and analysis on Scction 4.1
A theoretical proof on the degree of expansion required to tackle specific task is interesting problem that is left as future work.
4

---

Published as a conference paper at ICLR 2024
in Section In addition, exhibit unique advantage of models without activation functions to learn dynamic systems in scientific computing in (ection[4.3) Lastly, we validate the robustness of our model to diverse perturbations in Section 4. We summarize four configurations of the proposed MONet in Table T with different versions of MONet. We present a schematic for our configuration in Appendix B Due the limited space, we conduct additional experiments in the Appendix. Concretely, we experiment on semantic segmentation in Appendix H while we add
additional ablations and experimental details on Appendices K to M
4.1 IMAGENETIK IK CLASSIFICATION
ImageNetIK, which the standard benchmark for image classification, contains 1. 1.2M images with 1,000 categories annotated. We consider a host of baseline models to compare the performance of MONet. Concretely, we include strong- performing polynomial model Chrysoos et al. 2020 2023).
MLP-like models (Tolstikhin et al.] [021 Touvron et al., [022 Yu et al. [022b)). models based on
vanilla Transformer /Vaswani et al. 2017 Touvron et al., 2021) and several classic convolutional
networks (Bello et al. 2019 Chen et al. 2018b Simonyan & Zisserman 2015 He et al.
2016).
Training Setup: We train our model using AdamW optimizer (Loshchilov Hutter 2019). We use batch size of 448 per GPU to fully leverage the memory capacity of the GPU, We use a linear warmup and cosine decay schedule learning rate, while the initial learning rate is le-4, linear increase to le-3 in 10 epochs and then gradually drops to le-5 in 300 epochs. We use label smoothing (Szegedy et al. 2016). standard data augmentation strategies, such as Cut-Mix (Yun et al. 2019), Mix-up (Zhang et al. 2018) and auto-augment (Cubuk et al. 22119), which used in simi lar methods (Tolstikhin
et al. 2021 Trockman & Kolter 2023 Touvron et al. 2022). Our data augmentation recipe follows
the used in MIP-Mixer (Tolstikhin et al.) 2021) We do not use any external data for training. We train our model using native Py PoTch training on 4 NVIDIA A GPUs.
We exhibit the performance of the compared networks in Table 2 Notice that our smaller model achieves a 10% improvement over previous PNs. Our larger model further strengthens this perfor- mance gain and manages for the first time to close the gap between PNs and be on par with other recent models.
4.2 ADDITIONAL BENCHMARKS IN IMAGE RECOGNITION
Beyond ImageNetIK, we experiment with a number of additional benchmarks to further assess MONet. We use the standard datasets of CIFARIO (Krizhevsky et al. (009). SVHN (Netzer et al.) 2011) and Tiny ImageNetI (Le & Yang 2015) for image recognition. A fine-grained classification experiment on Oxford Flower102 (Nilsback & Zisserman 2008) is conducted. Beyond the different distributions, those datasets offer insights into the performance of the proposed model in datasets with limited dat= The results in Table 3 compare the proposed method with other strong-performing methods. MONet outperforms the compared methods with the second strongest being the R-PollNets. MLP-models usually do not perform well in small datasets as indicated in Tolstikhin et al. (2021) due their weaker inductive bias. Notably, MONet still performs better than other CNN-based polynomial models.
4.3 POLY NEURAL ODE SOLVER
An additional advantage of our method is the ability to model functions that have a polynomial form (e.g, in ordinary differential equations) in an interpretable manner (Fronk & Petzold 2023). neural ODE solver is a computational technique that models dynamic systems relying on ODEs using neural networks. Chen al. (2018a); Dupont al. (2019) illustrate the potential of neural ODEs to capture the behavior of complex systems that evolve over time. However, an evident limitation is that those models remain black-box which means the learned equation remains unknown after training. On the
'e note that we report results of le-nets without activation functions from Chrysos et al. 2020 for a fair comparison. The models with activation functions are reported as 'hubrid'. 'Previous MLP-based models have demonstrated weaker performance than based models in such datasets. The original Sa MLP only reports two large models on these datasets. We use open source S MLMLP code design two models with 12- 14 M parameters for a fair comparison. Those models, noted as MLM..--Wdd-SS (hidden size 512, depth 8) and ML (hidden size 384, depth 12) are used for the comparisons.
5

---

Published as a conference paper at ICLR 2024
Table 2: ImageNetIK classification accuracy (%) on the validation for different models. In the Activation Function column, G denotes Gelu, R denotes Relu, T denotes Tanh, and P denotes RPReLU. The models with are special polynomial models with activation functions. We mark model with up to 15M params in red color, models with 15-40M parameters with green color and model with over 40M parameters with blue Beyond our models with Layer Normalization, we conduct an experiment with Batch Normalization and notice that the performance is similar (i.e., the model Multi-stage HONet-T-BatchNond below).
Accuracy
Extra Data
Top-1 ( (6)
Top-5(%)
FLOP (B)
Params (M)
Activation Attention
CNN-bused
[H ResNet-18
2016
69.7
89.0
1.8
X
ResNer-50
[Te et
2016
77.2
92.9
4.1
25.0
R
X
Net
Chen at a.
2018b
77.0
93.5
31.3
AA ResNet (Ballo
2019
79.1
94.6
23.8
61.6
VGG-16 Simonyan & Zissemann
2015
71.5
92.7
15.5
138.0
R
X
Transformer
DeiT-S/16
Touvron 2021
81.2
5.0
BT16
Disuvissiyy a. 2021
77.9
55.5
86.0
G
MI.P-bused
BiMLP-S
1.21
BiMLP-B Xu cl 2022
72.7
1.21
ResMLP Togymin m.
2022
76.6
3.0
15.0
Hire-ML.PTThny
2022
79.7
2.1
CyclcMLP-T Chen al 2022
81.3
4.4
28.8
ResMLP-24 Touvvon et 2022
79
MLP-Mixer-B/10 Toisiikin Kin et
2021
X
76.4
11.6
59.0
MLP-Mixer-L/16 [Tostakaa Cl
2021
71.8
44.6
207.0
S'MLP-Wide (Yu ct
80.0
94,8
14.0
71.0
5*MLP-Deep ot 20226
X
80.7
95.4
10.5
51.0
FF (Melas Kymtaz 2021
74.9
7.21
59.0
Polynomial-base
NetN (Chrysos
2020
65.2
85.9
1.9
12.3
None
X
Hybrid TT-Nes (C (Chrysos et 2020
70.7
89.5
1.9
11.9
R,T
X
PDC-comp (Chrysos
89,9
1.3
R.T
X
PDC (Chryso et 212-
71.0
89.9
1.6
10.7
R,T
R-PolyNee (Chrysos
2023
70.2
89.3
1.9
None
D-PolyNets (Cinsso al
2023
70.0
89.4
1.9
11.3
None
Multi-stage NIONeI- 1-Lyce---Laeoooo (Ours)
77.0
93.4
2.8
10.3
None
Multi-stage MKNet-T-Batchhil (Ours)
77.7
93.8
2.8
None
Multi-stage Layee-S- (Ours)
81.3
95.5
6.8
32.9
None
X
Table 3: Experimental validation of different polynomial architectures smaller datasets. The best are marked in bold. D-PolyNets perform favorably to most of the other baseline polynomial networks and the rest baselines. MONet outperforms all the baselines and all polynomial networks in two of the four datasets, and shares the best accuracy with R-PolyNets another dataset.
Model
CIFARIO
SVHIN
Oxford Flower
Tiny Imagenet
Resnet18
97.3
61.3
MLP-Mixer
90.6
96.8
80.2
45.6
Res-MLP
92.3
97.1
83.2
58.9
MPML--Dep-
92.6
96.7
93.0
59.3
S MMLP-Wide-S
92.0
96.6
91.5
52.7
Hybrid Pi-Nets
94.4
88.9
61.1
I1-Nets
90.7
96.1
82.6
50.2
PDC
90.9
96.0
88.5
45.2
R-PolyNets
94.5
97.6
94.9
61.5
D-PolyNets
94.7
97.5
94.1
61.8
MONet-T
94.8
97.6
95.0
61.5
contrary, the proposed model can recover the equations behind the dynamic system and explicitly restore the symbolic representation.

---

Published as a conference paper at ICLR 2024
10
Thee Date for Thie Payt
Mou OOE for Mou aa ODD for
500 Ued Lo
opoches
Figure 2: The training loss change with epochs trained([.eft) The ground truth and model predicted trajectory. (Right) Our model achieves low loss in 20 epochs and successfully predicts real trajectory.
Table 4: Robustness on ImageNet-C (Hendrycks Dietterich, 2019). The corruption error (and mean corruption error (mCH)) is used as the robustness metric, where lower score indicates better performance. The best performance per category is highlighted in bold. Notice that the proposed method is robust to range of corruptions. In fact, in many categories, such as 'Weather' and 'Digital', the proposed model outperforms the compared models in all designated corruptions.
Noise
Blur
Weather
Digital
Network
mCR(J)
Gauss Shot Impulse
Defocus Glass Motion Zoom Snow
Frost
Fog Bright Contrast
Flastic Pixel JPFG
ResNet50
76.7
79.8
81.6
82.6
74.7
88.6
78
79.9
77.8
74.8
66.1
56.6
71.4
84.7
76,9
76,8
DaiT
54.6
46.3
47.7
46,4
61.6
71.9
57.9
71.9
49.9
46.2
46
44,9
42.3
66.6
59.1
60.4
Swin
62.0
52.2
53.7
53.6
67.9
78.6
64.1
75.3
52.8
51.3
48.1
45.1
75.7
76.3
79.1
MLP-Mixer
78.8
80.9
82.6
84.2
86.9
92.1
79.1
93.6
67.4
64.6
90.5
72.7
92.2
ResMLP
66.0
57.6
58.2
57.8
72.6
83.2
67.9
76.5
61.4
57.8
63.8
53.9
52.1
78.3
72.9
75.3
gMLP
64
52.1
53.2
52.5
73.1
77.6
64.6
79.9
77.7
78.8
54.3
55.3
43.6
70.6
58.6
67.5
CycleMLP
53.7
42,1
43.4
43.2
61.5
76.7
56.0
66.4
$1.5
47.2
50.8
41.2
39.5
72.3
57.5
56.1
HireMLP
51.9
52.4
55.3
55.3
60.3
71.6
59.6
57.8
$4.3
52.5
43.5
29.1
42.2
54.9
49.7
40.2
R-PolyNets
73.8
84.0
84.4
88.0
81.9
83.6
75.0
77.9
71.8
72.2
69.4
45.7
67.4
65.8
74.8
65.4
MONet-S 49.7
51.3
52.3
53.0
57.8
72.1
55.0
60.6
50.4
46.0
42.0
38.1
53.7
47.7
39.0
We provide an experiment of a polynomial neural ODE approximating the Lotka- Volterra ODE model. This model captures the dynamics of predator-pree population within biological system. dy B.rr,, de -oy yzz. Given a = 1.56, B = 1.12, 3.10, = 1.21, the ground truth equation are presented below:
The equations representing the Lotka-Volteraa Volterra formula are in the form of de
dæ
dy = --133.1p 3.y - 1.21.r
1.56x - 1.12xy
dt
dt
We generate N = 100 discrete points between time and 10 for training. The training loss with epochs and predicted trajectory shown Fig.
Notice that our model can directly recover the right hand side of the Voltera formula. Indeed, the learned model our case recovers the following formula:
dr
dy
dt = 1.56x 1.12001zy.
- 3.10y - 1.21001xr
dt
Examining the results, it is evident that our Poly Neural ODE achieves high accuracy in recovering the ground truth formula and demonstrates rapid convergence. more comprehensive comparison with NeuralODE and Augmented NeuralODE is conducted in Appendix G
4.4 ROBUSTNESS
We further conduct experiments on ImageNet-C (Hendrycks Dietterich, 2019) to analyze the robustness of our model. ImageNet-C contains 75 types of corruptions, which makes it a good testbed for our evaluation. We follow the setting of CycleMLP (Chen et al.) 2022) and compare MONet with existing models using the corruption enror as the metric (lower value indicates better performance). Table 4 illustrates that our models exhibits the strongest ability among recent models. Notice that in important categories, such the 'weather' or the 'digital' category, MONet outperforms the
7

---

Published as a conference paper at ICLR 2024
Table 5: The influence of the different module in Block. Based on depth 16, hidden size 384 model on ImageNetl00. We incorporate spatial shift since this operation is multilinear and brings a performance gain. However, the primary performance improvement comes from the Mu-Layer we proposed as shown in the following table. Notice all models mentioned below are without activation functions.
Block Type
Layer
Layer 2
Paras, (M)
Top-1 Acc( %
Foly-Block
Mu-
Mu-Layer
1.70
82.94%
Poly-Block*
Mu-layer
Mu-Layer
1.70
81.50%
Mix Block
Mu-LLyer-SSatial-Shie
MLP Layer
1.26
67.61%
Linear Block
MLP Layer+Spatial-Shi
MLP Layer
1.21
55.11%
Table 6: The influence of the patch embedding approach on a model with depth 16, hidden size 384 trained on ImageNet100. Notice that the pyramid patch embedding can improve the performance with a minor difference in the parameters and maintain small FLOPs when using small patch.
Method
Patch size
Top-1 Acc(%)
Paras (M)
FLOPs
Patch Embedding
83.08
27.77
28.1 18.14
Patch Embedding
14
79.54
27.94
7.07
Pyramid Patch Embedding
7
82.04
28.02
7.23
Pyramid Patch Embedding
14,7
82.94
28.54
7.28
compared methods in all types of corruption. Those categories can indeed be corruptions met in realistic scenaria, making us more confident on the robustness proposed model in such cases.
4.5 ABLATION STUDY
We conduct an ablation study using ImageNet subset of ImageNetIK with 100 classes selected from the original ones. Previous studies have reported that ImageNet100 serves as a representative subset of ImageNet1 (Yan et al.] 2021 Yu al., 2022a Douillard et al., 2022). Therefore, ImageNet100 enables us to make efficient use of computational resources while still obtaining representative results for the self-evaluation of the model. In every experiment below, we assess one module or hyperparameter, while we assume that the rest remain the same as in the final model.
Module ablation: In this study, we investigate the influence of various modules on the final network performance by removing and reassembling modules within the block. Please note that the following blocks do not contain any activation functions. In Table we report the result using a hidden size of 384 and depth 16 on the ImageNet100. We observe that spatial shift contributes to the performance improvement in our model but it is not crucial. The network still achieves high accuracy even in the absence of this module. At the same time, the removal of the Mu-Layer from the model leads to significant drop in performance. The results validate that the spatial shift module alone cannot replace the proposed multilinear operations.
Patch Embedding: In this study, we validate the effectiveness of our proposed pyramid patch embedding. We replace the input embedding layer of model with depth 16 and hidden size 384 on the ImageNet100 dataset. As depicted in Table 6 the model utilizing multi-level embedding achieves an Multi-stage performance with Top-1 accuracy of 82.94%. The model utilizing single-level embedding achieves a Top-1 accuracy of 81.78% which is slightly lower than the multi-level embedding approach. For normal patch embedding, the computational and parameter overhead increases quadratically with decrease in patch size. The result highlights that our approach significantly improves performance while adding only a small number parameters and computational overhead. We conduct the same study for MLP-Mixer in Appendix D with similar outcomes, indicating the efficacy of our core Poly-Block independently of the patch embedding module.
Hidden size: We evaluate the impact different hidden size m. We fix the depth of the network to 16 and vary the hidden size. We observe sharp decrease when MLP-Mixer uses small hidden size under 128. The results in Table / validate that our proposed method more robust in hidden size change compared to the normal MI. layer.
8

---

Published as a conference paper at ICLR 2024
Table 7: The influence of the hidden size, m. models result are based on depth 16 model with different hidden size, the number in the second and their row indicates Top-1 Accuracy($ on ImageNet100. The MLP-Mixer shows sharp decline when hidden size change from 128 to 96.
96
128
192
256
384
512
MLP-Mixer
48.3%
70.05%
72.39%
76.88%
18.24%
77.93%
MONet
67.5%
75.82%
79.3%
81.94%
82.94%
82.24%
Table 8: The influence of the depth, N. The model is based on depth 16, hidden size 384 and is trained ImageNet100.
Table 9: The influence of the shrinkage ratio, T. The model is based on depth 16, hidden size m 384, trained ImageNet100.
N
Top-1(%)
Top-5(%)
Paras (M)
Top-5(%) Top-1(%)
Paras (M)
44.25
71.18
2.96
83.18
94.86
52.98
70.58
78.30
6.37
2
83.88
95.54
36.46
78.30
93.20
11.49
82.94
95.04
28.54
12
80.03
94.12
21.72
8
82.28
95.12
24.06
82.94
95.04
28.54
16
82.34
94.62
21.99
The backbone of our proposed model, MONet, consists of N blocks. We evaluate the influence of the number of blocks (depth N) on Top-1 accuracy and parameter numbers, using model with a fixed hidden size and shrinkage ratio. We observe that for our network, the depth holds more significance role than the hidden size. Our experiments result in Table @ validate our theoretical conjecture, as increasing depth yields a more pronounced improvement in performance.
Depth:
Shrinkage ratio: As mentioned in Section 3. the shrinkage ratio is defined as T. The results of different shrinkage ratios based on hidden size of m 384 and a depth of 16 in our model are presented in Table We observe that: i) A larger shrinkage ratio effectively reduces the number of parameters while having a relatively minor impact on performance; ii) a lower effective rank the weights is sufficient for performance gain.
CONCLUSION
In this work, introduce model, called MONet that expresses the output polynomial expansion of the input elements. MONet leverages solely linear and multilinear operations, which avoids the requirement for activation functions. At the core of our model lies the Mu-Layer, which captures multiplicative interactions inside the token elements (i.e. input). Through comprehensive evalua- tion, we demonstrate that MONet surpasses recent polynomial networks, showcasing performance levels outperforms modern transformers models across range of challenging benchmarks in image recognition. We anticipate that our work will further encourage the community reconsider the role of activation functions and actively explore alternative classes of functions that do not require them. Lastly, we encourage the community to extend our illustration of the polynomial ODE solver in order to tackle scientific applications with PNs.
Limitation: A theoretical characterization of the polynomial expansions that can be expressed with MONet remains elusive. In our future work, we will conduct further theoretical analysis of our model. We believe that such an analysis would further shed light on the inductive bias of the block and its potential outside of image recognition.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
Throughout this study, we exclusively utilize publicly accessible benchmarks, guaranteeing that other researchers can replicate our experiments. Additionally, we provide comprehensive information about the hyperparameters employed in our study and strive to offer in-depth explanations of all the techniques employed. Our plan to make the source code of our model open source once our work gets accepted.
ACKNOWLEDGMENTS
We are thankful Giorgos Bouritsas and the ICLR reviewers for their feedback and constructive comments. We are also thankful to Colby Fronk for his help in the NeuralODE symbolic representa- tion restoration. We thank Zulip for their project organization tool. This work was supported by the Hasler Foundation Program: Hasler Responsible AI (project number 21043). Research was spon- sored by the Army Research Office and was accomplished under Grant Number 1911NF-24-1-00 This work supported by the Swiss National Science Foundation (SNSF) under grant number 200021 1050111.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXivv1607.064500, 2016.
Francesca Babiloni, Ioannis Marras, Filippos Kokkinos, Jiankang Deng, Grigorios Chrysos, and Stefanos Zafeiriou. Poly-nl: Linear complexity non-local layers with polynomials. In International Conference on Computer Vision (ICCV), 2021.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp- 3286-3295, 2019.
Zvika Brakerski, Craig Gentry, and Vinod Vaikuntanathan, (leveled) fully homomorphic encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT), 6(3):1-36, 2014.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pp.
1721-1730, 2015.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems (NeurIPS), 31, 2018a.
Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo. Cyclemlp: mip-like architecture for dense prediction. International Conference on Learning Representations (ICLR), 2022.
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A 2-nets: Double attention networks. In Advances in neural information processing systems (NeurIPS), volume 31, 2018b.
Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng, and Stefanos Zafeiriou. P-nets: Deep polynomial neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7325-7333 2020.
Grigorios G Chrysos, Markos Georgopoulos, and Yannis Panagakis. Conditional generation us- ing polynomial expansions. In Advances in neural information processing systems (NeurIPS), volume 34, pp. 28390 2021.
httpss:/zul11 wp.
10

---

Published as a conference paper at ICLR 2024
Grigorios G Chrysos. Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks. In European Conference Computer Vision (ECCV), pp. 692-716. Springer, 2022.
Grigorios G Chrysos, Bohan Wang, Jiankang Deng, and Volkan Cevher. Regularization of polynomial networks for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning (ICML), pp. 933-941, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Abhimanyu Dubey, Filip Radenovic, and Dhruv Mahajan. Scalable interpretability via polynomials. Advances in neural information processing systems (NeurIPS), 2022.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in neural information processing systems (NeurIPS), 32, 2019.
Colby Fronk and Linda Petzold. Interpretable polynomial neural ordinary differential equations.
Chaos; An Interdisciplinary Journal of Nonlinear Science, 33(4), 2023.
Markos Georgopoulos, Grigorios G Chrysos, Maja Pantic, and Yannis Panagakis. Multilinear latent conditioning for generating unseen attribute combinations. In International Conference on Machine Learning (ICML). pp. 3442-3451, 2020.
Markos Georgopoulos, James Oldfield, Mihalis A Nicolaou, Yannis Panagakis, and Maja Pantic. Mit- igating demographic bias in facial datasets with style-based multi-atrribute transfer. International Journal of Computer Vision (IJCV), 129(7):2288-22) 2307, 2021.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 249-256, 2010.
Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision mlp via hierarchical rearrangement. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 826-83868 836, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing heman-level performance imagenet classification. In International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. In International Conference on Learning Representations (ICIR), 2019.
11

---

Published as a conference paper at ICLR 2024
Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: permutable mlp-like architecture for visual recognition. IEEE Transactions on Patter Analysis and Machine Intelligence, 45(1):1328-1334, 2022.
Jie Hu, Shen, and Gang Sun. Squeeze-and-excitttioon networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp- 7132-7141, 2018.
Alexey Grigorevich Ivakhnenko. Polynomial theory of complex systems. IEEE trunsactions on Systems, Man, and Cybernetics, 1(4):364-378, 1971.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. Neural networks: Tricks of the trade, pp. 9-50. Springer, 2002.
Chien-Kuo Li. A sigma-pi-sigma neural network (spsnn). Neural Processing Letters, 17:1-19, 2003.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Conference on Computer Vision and Pottern Recognition (CVPR), pp. 510-519, 2019.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan. and Serge Belongie. Feature pyramid networks for object detection. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2117-2125, 2017.
Hanxiao Liu, Zihang Dai, David So, and Quoe V Le. Pay attention to mips. Advances in neural information processing systems (NeurIPS), 34:9204-9215, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference Learning Representations (ICLR), 2019.
Dengsheng Lu and Qihao Weng. A survey image classification methods and techniques for improving classification performance. International journal of Remote sensing, 28(5):823-870, 2007.
James Martens et al. Deep learning via hessian-free optimization. In International Conference on Machine Learning (ICML), volume 27, pp. 735-742, 2010.
Luke Melas Delaa-Kyriazzi. Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet. arXiv preprint arXiv:2 105. (0l.:22200 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. Advances in neural information processing systems (NeurIPS), 2011.
Maria-Flena Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics and Image Processing. doi: 10.1 10.100/CCVGIP2000....
pp. 722-729, 2008.
Luzhou Peng, Bowen Qiang, and Jiacheng Wu. A survey: Image classification models based on convolutional neural networks. In 2022 14th International Conference on Computer Research and Development (ICCRD), pp. 291-298, IEEE, 2022.
Jo Plested and Tom Gedeon. Deep transfer learning for image classification: survey, 2022.
12

---

Published as a conference paper at ICLR 2024
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assise Intervention- 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234-241. Springer, 2015.
Yoan Shin and Joydeep Ghosh. The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation. In IJCNN-91-Seatle international joint conference on neural networks, volume 1, pp. 13-18. IEEE, 1991.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016.
Iyy O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems (NeurIPS), 34: 24261-24272, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. International Conference on Machine Learning (ICML), pp. 10347-10357, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, al. Resmlp: Feedfor- ward networks for image classification with data-efficien training. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2022.
Asher Trockman and Zico Kolter. Patches are all you need? Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances neural information processing systems (NeurIPS), volume 30, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 568-578, 2021.
Conference Computer Vision and Pattern Recognition (CVPR), pp. 7794-7803, 2018. Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 1000111771111 -20770 2021.
Yixing Xu, Xinghao Chen, and Yunhe Wang. Bimlp: Compact binary architectures for vision multi-layer perceptrons. Advances in neural information processing systems (NeurIPS), 2022.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 3014-3023, 2021.
Guandao Yang, Sagie Benaim, Varun Jampani, Kyle Genova, Jonathan Barron, Thomas Funkhouser, Bharath Hariharan, and Serge Belongie. Polynomial neural fields for subband decomposition and manipulation. Advances in neural information processing systems (NeurIPS), 35:4401-4415, 2022.
Jiancheng Yang. Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 191-195, 2021.
13

---

Published as a conference paper at ICLR 2024
Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled non-local neural networks. In European Conference on Computer Vision (ECCV), pp. 191-207. Springer, 2020.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlpv2: Improved spatial-shift mlp architecture for vision. In Winter Conference on Applications of Computer Vision (WACV), 2022a.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mip architecture for vision. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 297-306, 2022b.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. Conference Computer Vision and Pattern Recognition (CVPR), pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Pazez mixup: Beyond empirical risk minimization. International Conference on Learning Representations (ICLR), 2018.
14

---

Published as a conference paper at ICLR 2024
CONTENTS OF THE APPENDIX
The contents of the supplementary material are organized as follows:
In Appendix A and Poly-Block.
we provide technical proofs regarding the interactions learned by Mu-Layer
In Appendix B we exhibit the details of the architecture design for our model, including the design of single-stage and multi-stage architecture.
present the schematic of the Mu-Layer provide more intuitive understanding of its structure. Additionally, we provide detailed explanations of the Mu- Layer and Poly-Block components.
In Appendix C
In Appendix D elaborate the pyramid patch embedding method mentioned in Section 3.2 We provide illustrative comparisons between the conventional patch embedding approach and our method, showcasing the differences in the representation of input data between the two approaches.
In Appendix we extend our method beyond natural images to the medical domain.
In Appendix we visualize the effective receptive field of our ImageNet- 1K pre-trained
model. We also visualize related models with same input image as a comparison.
In Appendix G we follow the original evaluation experiment to further compare our method with existing NeuralODE Solver (Dupont al. 2019: Chen et al. 2018a).
In Appendix
scrutinize model further beyond image recognition. We conduct a semantic segmentation evaluation on ADE20K dataset. The result shows our proposed model also greatly surpasses previous PN models in semantic segmentation.
In Appendix we measure computation cost using FLOPs. We notice that the tool impacts the number of FLOPs and indeed we observe that this resulted in less accurate measurements of previous PN models. We utilize open-source code to reevaluate the computational costs of existing PN models. In this section, we present precise FLOPSs (Floating Point Operations) values for range of PNs, along with their corresponding performance ImageNet1 K.
In Appendix T conduct complexity analysis of proposed model.
In Appendix K we list the training setting and hyperparameter of our model used for ImageNet- training. We also conduct an error analysis using saliency maps in Section to investigate the weaknesses of our model further.
Appendix K.
In Appendix we list the training settings and hyperparameters used for the medical image classification experiment and the fine-grained classification experiment. Additionally, include the model settings table.
In Appendix M we perform experiments to examine the effects of various parameter initialization methods the final performance of our model.
15

---

Published as a conference paper at ICLR 2024
A PROOFS
In this section, we derive the proof of Prop. 1 and also include new proposition for the Poly-Block, along with the associated proof.
A.1 PROOF OF PROP.
In this paper, we use bold capital letter to represent a matrix, c.g., F, while a plain letter to represent scalar. For instance, X3,p represents the element in the jth row and the p column. The matrix multiplication of two matrices F, K results in the following j. element:
[F K 3.p- Eg Fj.g K.p-
As reminder, the imput is X E Rd where d is the length of a token and E N is the number of tokens. Thus, to prove the proposition, we need to show that products of the form x exist between elements of each token.
Eq. relies matrix multiplications and Hadamard (elementwise) product. Concretely, if we express the J. p element of (AX) (BDX), we obtain:
[(AX) * (BDX)]
Σ
A D...
1,pmm
Then, can add the additive term and the matrix C. If we express Eq. the following expression:
elementwise, we obtain
M = cus Σ
Du Σ A Xr.
(5)
j=1 r=1
That last expression indeed contains sums of products X X for every output element, which concludes our proof.
A.2 INTERACTIONS OF THE POLY-BLOCK
As we mention in the main paper, a Poly-Block comprises of two Mu-Layers, so one reasonable question is how the multiplicative interactions of the Mu-Layer extend in the Poly-Block. We prove below that up to fourth degree interactions are captured in such block. To simplify the derivation, we focus on two consecutive Mu-Layers as the Poly-Block.
Proposition 2. The Poly-Block captures up to fourth degree interactions between elements of each token.
Proof. All need to show is that products of the form X, X appear. Using the insights from Appendix A.. we expect the fourth degree interactions to appear the Hadamard product, so we will focus on the term (2) ( (B(2) D X (2), where the (2) declares the weights and input of the second Mu-Layer.
The elementwise expression for X is directly obtained from Eq. 5 To simplify the expression, we ignore the additive term, since exhibiting the fourth degree interactions enough. The q, w element of the expression is the following:
M Σ
[ (2) xx11) * (B()(()))) ( x
(
Cgg
B a. (fi (f2)
9.5
1=1,8=1
where the expression {fi} {f2} is the following:
Σ
Σ Ag. Be, Dus X Ao, Bo Dus Xe.p
7=1, wis!
Notice that the last expression indeed contains products of the form X, Xp y Xep which concludes the proof.
16

---

Published as a conference paper at ICLR 2024
B MULTI-STAGE MONET
Image
Unber Bribodiing
Poly Block
3x
Depth
Chasten
Ovenview Single Stage MONet
Image
Unee
Poly
Paich
Parh
Poly
Paic
Poly
Seage
Sageo
Slgee Humber
C x
Cs x
C.
C.
The Overview Architecture of Multi-stage NOWee
Figure 3: The Schematic of (simple) MONet and Multi-stage MONet. PPE represents our pyramid patch embedding.
In deep architectures often different number of channels hidden size is used for different blocks (He et al., 2016). Our preliminary experiments indicate that MONet is amenable to different hidden size as well. Following recent works (Chen et al. [022] Guo et al. 2022). we set 4 different hidden sizes across the network (referred to as stages) and we refer to this variant as the 'Multi-stage MONet'. This variant is mostly used for large-scale experiments, since for datasets such CIFARIO. and CIFARIO0 100 single hidden size is sufficient.
C DETAILS OF MU-LAYER AND POLY-BLOCE
B
S
D
A
S
*
f(z)
Figure 4: The Schematic of Mu-Layer. Blue boxes correspond learnable parameters. Green and red boxes denote input and output, respectively. The denotes the Hadamard product, the - denotes element-wise addition. The gray box denotes the spatial aggregation module, the dotted line represents it as an optional module. In our design the first Mu-Layer of each Poly-Block includes spatial aggregation unit, while the second Mu-Layer does not.
In Fig. 4 present the schematic of the Mu-Layer. The matrices A, B, C, and D in Fig, are described in Section 3.1 As mentioned Section (.11 the Poly-Block serves as larger unit module in our proposed model. In our implementation, the Poly-Block consists of two Mu-Layer modules. Only the first Mu-Layer in each block incorporates a spatial aggregation module, where we utilize a spatial shift operation (Yu al. 2022b).
D PYRAMID EMBEDDING
Patch Embedding, which is typically used in the input space of the model, converts an image into sequence of N non-overlapping patches and projects to dimension d. If the original image has
17

---

Published as a conference paper at ICLR 2024
1#14Convatride-11
2'2oow,stridee2
77Conv.
Embedding
Input Image
Pyramid Embedding
Input Image
Figure 5: Pyramid Patch Embedding
Table 10: Influence of the patch embedding on MONet.
Table 11: The influence of the patch embed- ding approach MLP-Mixer.
Method Parch Limbedding
Patch size
TTop.1 Aect (b)
Paras (M) FLOPS
Method
Patch size
Top-1 Auce %)
Purus (Mi FLOPS
$3,05
27.33
28.14
Puich Lmbesking
77.92
26.50
19.18
Purch Embldding
7.07
Putch Endedding
77.93
Pyramid Embedding
7.25
Perand Punch Embbdding
79.64
Pyramid Embhedding
Pyramid Parch Fanholding
14.7
70.06
(H, W) resolution, each patch X; has resolution (P, Then, we obtain S H W/p2 patches, where X; E Rxxxxd For models based on MLP structure, smaller patches can capture finer-grained features better. However, due the quadratic growth in computation caused by reducing the patch size, most models (Guo et al. 2022 Chen et al.) 2022) do not follow this design. To reduce the computational burden by smaller patches, these networks assume multiple hidden sizes across the network, see Appendix for further details. Instead, we utilize an additional method for reducing the computational cost. After performing the first patch embedding on the original image using a small patch size, we can further reduce the input size by performing 2 x 2 convolution with stride With this approach, we can achieve performance comparable to directly using a small patch size while significantly reducing computational costs and allowing to adopt more elegant and simple structure.
Inspired by FPN (Lin al. 2017). we introduce different level of patch embedding to further enhance the performance. Each level of the pyramid represents an image at a different scale, with the lower levels representing images with higher resolution and the higher levels representing images with lower resolution. We use lower resolution image patch merged with downsampled higheroresolution image patch by element-wise addition. In Section 4.5 we compare our method with normal patch embedding. The embedding method we propose not only reduces network computational complexity but also improves the performance when compared larger patch sizes.
We conduct 8 comparisons to prove the soundness of our method. We employ different patch embedding and patch sizes for the MLP-Mixer and our model. We train those models on ImageNet- 100 dataset and report Top-1 Accuracy, their parameters and computation costs (FLOPs) in Table T0 and Table
Appendix D indicates that our pyramid patch embedding improves the performance compared to normal patch embedding. By leveraging our proposed method, we can utilize small patch size to achieve better results while preventing quadratic growth of computation costs. This could be used as an ad-hoc module in other MIP models. At the same time, even with a normal patch embedding, our model still outperforms MLP-Mixer.
In the context of merging different levels of features, we initially adopt an elementwise addition approach. Additionally, explore a U-Net-style approach (Ronneberger et al.) 2015). In this approach, we concatenate the features along the channel dimension and then reduced the dimension using 1x1 depthwise convolution. However, after conducting experiments, we observe that this approach had limited improvement and was considerably less effective compared to the elementwise addition method.
18

---

Published as a conference paper at ICLR 2024
Table 12: MONet performance compared ResNet18 on the MedMNIST benchmark, with detailed numbers. The best results are marked in bold.
Dataset
MONei-T
PDC
TT-Nets
MLP-Mixer
ResMLP
S S*MLP-DS
W-P.-W-S
ResNet
AutoSklearn
Path
90.8
92.1
90.8
89.2
90.7
90.7
83.4
Derma
77.5
77.5
72.9
76.1
76.5
76.2
76.9
73.5
74.9
Oet
80.1
77.2
79.8
78.2
79.7
79.1
79.5
74.3
60.1
Pncumonia
93.4
92.5
89.1
93.1
89.1
91.9
93.1
85.4
87.8
Retina
55.7
53.6
52.2
54.5
54.7
53.7
54.0
52.4
51.5
Blond
96.7
95.5
94.5
94.7
95.3
95.8
95.2
93.0
96.1
Tissue
67.7
67.5
65.9
66.8
65.9
66.0
67.7
53.2
OrganA
93.6
93.5
92.5
90.5
91.4
92.3
92.7
93.5
76.2
OrganC
88.9
93.0
89.3
90.6
88.9
89.4
90.0
87.9
OrganS
78.5
77.9
75.0
76.9
75.7
78.4
78.2
67.2
E
MEDICAL IMAGE CLASSIFICATION
To assess the efficacy of our model beyond natural images, we conduct an experiment on the MedM- NIST challenge across ten datasets (Yang et al. (021). The dataset encompasses diverse medical domains, such chest X-rays, brain MRI scans, retinal fundus images, and more. MedMNIST serves as benchmark dataset for evaluating models in the field of medical image analysis, enabling us to evaluate the performance of our model across various medical imaging domains.
In our experiments, we train the variant of MONet-T with 14.0M parameters and 0.68G FLOPs. The results are shown Table exhhibii that our model outperforms other models in eight datasets.
F VISUALIZATION OF REPRESENTATIONS FROM LEARNED MODELS
In this section, we use visualizations to understand the difference between the learned models of MI.P-Mierr, ResMLP and CNN architectures. The effective receptive field refers the region in the input data that a neural network's 's output is influenced by, which has been proved as an effective approach understand where learned model focuses on. Specifically, effective receptive field of our model trained on ImageNetlk, i.e., the output of the last layer, and compare with pretrained MLP-Mixes ResMLP and Resnet in Fig. 6 We adopt a random image from ImageNet1 K as input and visualize the effective receptive field. We can notably observe that due to the flattening of input tokens into one-dimensional structures by MLP-Mixer and ResMLP, features they learn exhibit distinct grid effect due to the loss of 2D information. They also tend to emphasize low-level texture information to a greater extent. ResNet's effective receptive field is more discrete, encompassing both background and foreground elements to extent, focusing the global context. On the contrary, MONet focuses on the semantic parts (e.g. on the dog's 's face) which achieves balance between global and local context.
visualize the hidden unit
G NEURALODE SOLVER COMPARSION
Following the evaluation setup of the Neural-ODE network, we implement a Poly-Neural ODE based method, which can be used to solve ordinary differential equations for simulating physics problems. We conduct scientific computing experiments using simulated data, with the aim of predicting and interpreting the dynamics of physical systems
The specific task is to simulate moving the randomly generated inner sphere particles out of the annulus by pushing against and stretching the barrier. Eventually, since we mapping discrete points and not a continuous region, the flow is able to break apart the annulus to let the flow through. The initial particle states shown in Figure Fig. 7
We set the model with hidden dimension 32 and train on simulated data for 12 epoches. The revolution of the first 10 epochs shown in Fig.
We implement two models based on NeuralODE CChn et al. 201 Sa) and Augmented Nerual ODE (Dupont et al. 2019), called Poly NeruaOODE(PoyyNODE) and Poly Augmented NeuralODE(Poly
19

---

Published as a conference paper at ICLR 2024
MONet
MLPMixer
ResMLP
ResNet
Figure 6: Input imett. The ERF of model pretrained on ImageNet- BeEnNett-KK(Riigeee ((Right). We visualize the effective receptive field of MLP-Mixer, ResMLP, ResNet and MONet. Our model more focus on the dog's 's face and the outline shape of dog, while the rest MLP models focus the low-level texture feature.
Figure 7: The initial particles states. The task move the inner sphere of blue particles out of the red particles.
Figure 8: The move of particles. From left right it's feature evolution with iterations increase.
Figure 9: The predicted trajectory of particles. The lines are their history trajectory.
20

---

Published as a conference paper at ICLR 2024
ANODE). For Poly NeuralODE, higher-ordes model could achieve better accuracy with the cost of higher computation cost, the following experiment are based on minimal order 2 Poly NeuralODE.
We first compare NODE with Poly NODE, the loss plots shown in Fig. I0] We observe that with 40 epochs training, The Poly NODE approximates the functions faster and more accurately. We then
Poty New.-eeurOOOOO. Lons
Figure 10: Loss plots for NODEs and ANODEs trained 2d data. Poly NODEs easily approximate the functions and are consistently faster than NODEs.
ANODE
Poly ANODE
331
Figure 11:
Loss plots for Poly ANODES and ANODES trained on 2d data. Poly NODES easily approximate the functions and are consistently faster than NODEs.
compare ANODE with Poly ANODE. The loss plot trained 30 epoches is shown in Fig. II augment the space on which the ODE is solved, allowing the model use the additional dimensions to learn more complex functions using simpler flows. Our PolyANODE inherent its advantage while converge faster.
ANODES
H SEMANTIC SEGMENTATION
Settings. To further explore the performance of MONet in downstream task. We conduct semantic segmentation experiments on ADE20K dataset and present the final performance compared to previ- ous models. The Appendix H shows the result. Previous PNN models exhibited poor performance on downstream tasks. Our model has overcome this issue and achieved results comparable to some state-of-the-aar models.
I COMPUTATION COST COMPARED TO PREVIOUS POLYNOMIAL MODELS
FLOPs (Floating- Point Operations Per Second) is commonly used metric to measure the computa- tional cost of a model. However, tools like flops-counters and mmnv, which use PyTorch's 's module forward hooks, have significant limitations. They are accurate only if every custom module includes a corresponding flop counter.
In this paper, we adopt fvcore Flop Counter developed by Meta FAIR, which provides the first operator-level FLOP counters. This tool observes all operator calls and collects operator-level FLOP counts. For models like polynomial models that involve numerous custom operations, an operator- level counter will give more accurate results. We reproduced previous polynomial models according
fveore flops counter
21

---

Published as a conference paper at ICLR 2024
Table 13: Semantic segmentation on ADE20K val.All models use Semantic FPN head. The transformer and ResNet models result from the original paper. The polynomial networks result from the reproduced experiment.
Semantic FPN
Backbone
mloU(%)
R8 (He et 2016)
32.9
FCN (Long al. (2015)
29.3
PyT--Tiny (Wang et al. 2021
35.7
Seg-Former (Xic et al.
2021
37.4
R-PDC (Chrysos et al. 2022
20.7
R-PolyNets Chryssos et al. Multi-stage MONeI-S
2023
19.4
37.5
their open-source code and re-measure their computation cost with fvcore flops counter. This causes slight difference from the FL.OPs reported in previous PNs. For instance, operations such as custom normalization modules and Hadamard product computations were often overlooked due to former tool limitations. We adopt the number taken from original papers in Table 2 and we have corrected and updated the FLOPs report for recent polynomial models in the table below: In comparison to previous work, our model has significant performance benefits while only incurring 20% of the computational cost of the previous state-of of-the-art work. Simultaneously. our performance on ImageNet- IK surpasses the previous state-of-the-ar by 10 %.
The re-measure computational costs for polynomial models, along with their corresponding Top-1 accuracy ImageNet-IK
Table 14:
Model
FLOPs(B)
Accuracy(%)
PDC
30.78
71.0
R-PDC
39.56
R- PolyNets
29.34
70.2
DD-PolyNets
26.41
70.0
MONet-T (Ours)
3.6
76.0
MONet-S(Ours)
11.2
79.6
Multi-stage MONet-T(Ours)
2.8
77.0
Multi-stage MONet-S((Ors)
6.8
81.3
COMPLEXITY ANALYSIS
To simplify the process, conduct complexity analysis of MONet adopted the single-stage design as follows.
Pyramid Patch Embedding Layer (PPEL) crops the input image T Hxxx3 where W is the width and H the height into several non-overlapping patches of dimension P E Rxxxx3 Consider 1-level embedding, Our method conducts compression the patches with resulting in the token Wn E Rexxxe, token W, E x xc where W, is the output of the first convolution layer, W, is the output of the second convolution layer. Noted that the final number of x H which is 25% of normal patch embedding. Thus, the total number of patches are N₂ 2p 2p
2 convolution layer,
W
parameters of the Pyramid embedding is:
ParamspPEL. (3p² 1)c c(4c+ (3p² + 4c +
(8)
In summary the floating operations in PPEL is
W H
W
H 2p
FLOPsPPEL- = 3
x
4cN
p
2p
Poly Blocks(PB): Our model consists of N identical Poly Blocks. Each blocks contains Poly Layer. The first Poly (ay))s(PT,)) contains two Spatial Shift module in branches, and the
22

---

Published as a conference paper at ICLR 2024
second Poly Layers doesn't contain Spatial Shift module. We denotes that first Poly Layer four fully connected layer as W1. b1. W2, b2, W3, b3, Wa, b1 and shrinkage ratio as s, where W1 E R W2 E R W3 E Ri W4 E Rexc Therefore, the total parameter numbers of the first Poly Layers is
ParamsPL, = c x (2c + 2 3c = -2 ? +c33+
(10)
8
The second Poly Layer similar, while we have expansion rate r similar to other MLP models. We denotes that second Poly Layer four fully connected layer Wa, by, Wo, bs, W7, b7, Ws, by and shrinkage ratio as 8, where W3 E Rexre, W2 E Iexx = W3 E IR + xre Wa E Rrexx We get the total parameter numbers of the second Poly Layers is
c+ + 2re
r +
c2r r+
1 - + 1) (11)
ParamspL, cx (2re+
+ c(2r
s
8
Hence, the total number of Poly Block is
r² +
+ c(4 + 2r 2 (12)
Paramspg ParamssL, + ParamspL, 22r 2+
Besides the fully connected layer, the hadamard product bring extra computation where the flops of hadamard product of two matrices in shape x m is n x m. The overall hadamard product flops is
FLOPS = N₂c N N2 +r)
(13)
and the flops becomes
2 +r+2 +FLOPP = N2d 12 +r++ ) +N,o((1t))
=LOPspB= = N2r
(14)
Fully-connected classification layer(FCL): takes input c-dimensional vector and feedforward to average pooling layer. The output vectore in & dimension where k is the number of classes. In summary, the number of parameters of FCL is:
ParamsyeL = 1)
(15)
The flops of FCL is:
FLOPsPCL Npck
(16)
Overall architecture: We conclude that the summary of parameters of overall of architecture. The total numbers of parameters of our architecture is:
Params ParamspPEL + N x Paramspp + ParamspCL
(17)
The flops of our architecture is:
FLOPs = FLOPsPPEL +N FLOPspB + FLOPsFCL
(18)
23

---

Published as a conference paper at ICLR 2024
K IMAGENETIK K CLASSIFICATION
K.1 IMAGENETIK K. TRAINING SETTING
The Appendix K.
shows our experiment setting for training MONet on ImageNetIK dataset. Our implementation for the optimizer and data augmentation follow the popular timm library 0 We train 300 epochs in ImageNetI on GPUs, we enable exponential moving average on 200th epochs.
Table 15: ImageNet1 K Training Settings Section
ImageNet1 K Training
Setting
optimizer base learning rate
AdamW le-3
0.01 (Multi-stage
weight decay
MONet-'))
0.02 (Multi-stage
MONet-S)
batch size training epochs learning rate schedule
448
300
cosine
warmup label smoothing auto augmentation random crase cutmix mixup
0.1
/
0.1
0.5
0.5
K.2 ERROR ANALYSIS
We utilize our performrf MONet model to compute per-class accuracy rates for all 1000 classes on the validation dataset of ImageNet1 K. In Table T6] we present the top 10 least accurate and misclassified classes. Additionally, Fig. 12 we showcase the images of the most misclassified class (laptop). In Fig. [3 we show failure case of the screen. In Fig. [4] we show failure case of tennis.
Fig.
15
we show successful case of small object sunglasses.
Table 16: The top-10 10 least accurate classes and their labels
Class Name
Accuracy (%)
Class Name
Accuracy (%)
tiger Cat
20
laptop computer
24
screen
24
chiffionier.oomoode
28
sunglasses
cassette player
30
letter opener
30
malliot
32
sonjectileemisslee
32
spotlight
32
Our analysis dictates that the majority of classification failures occur when the model fails to focus on the main object. This even more pronounced in cases where the classification label is provided by an item occupying a small portion of the image. We believe this is an interesting phenomenon that shared with similar models relying on linear operations and not using convolutions.
Hugging Face timm
24

---

Published as a conference paper at ICLR 2024
Failure of MONet explained from the saliency map. The red figure in the middle denotes the saliency map, which is the area where the model focuses on. The correct class for the image laptop. while the saliency map shows the model focuses on the person. This is not an unrcasonable error though, since the main figure lies in the center of the image.
Figure 12:
Organel
Sasency
Cuerley
Figure 13: Failure case of MONet explained from the saliency map. The red figure in the middle denotes the saliency map, which is the area where the model focuses on. The correct class for the image is screen, while the saliency map shows the model focusing on the wall.
Figure 14: Failure of MONet explained from the saliency map. The red figure in the middle denotes the saliency map, which is the area where the model focuses on. The correct class for the image is tennis, while the saliency map shows the model focusing on the background wall, instead of on the small object.
Orignnal Imae
Celecry M4p
Cuetbey
Figure 15: Success case MONet as explained from the saliency map. The correct class for the image is sunglasses, The saliency map shows the model focusing on the face of the woman. In some cases, successful classification of a category can be achieved based on global information, even if the object occupies only small portion of the image.
25

---

Published as a conference paper at ICLR 2024
K.3 SOURCES FOR BASELINES
To further increase the transparency of our comparisons, we clarify the details on the main results on ImageNetIK. We aim for reporting the accuracy and the other metrics using the respective papers or public implementations. Concretely:
The paper of Chryyss et al. 20022) is used as the source for PDC.
Chrysos et al. (2023) is used as the source for & S.NNet18,Pi-eeeee Pi-Nets, Hybrid Pi-Nets, R- PolyNets and D-PolyNets. We clarify here once more that TT-Nets refers to the model without activation functions, while 'Hybrid 11-Nets' refers to the model with activation functions.
Yu al. (20226) is used as the source of ResNet50 performance. We updated the number to 77.2% (on ImageNetIK) to align with He et al. (2016).
The performance data for the other models in the Table 2 are sourced from their respective papers.
L ADDITIONAL BENCHMARKS AND MEDICAL IMAGE CLASSIFICATION
We introduce dataset details including dataset information for various datasets used for the empirical validation of our method.
In this section, we give brief introduction to the dataset we use and the overview statistics of datasets are shown in Appendix L
CIFARIO: CIFAR-10 (Krizhevsky et al., 2009) is a well-known dataset in the field of computer vision. It consists of 60,000 labeled images that are divided into ten different classes. Each image in the CIFAR-10 10 dataset has resolution of 32x32 pixels and is categorized into one of the following classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck. The images in the CIFAR10 dataset are of size 32 x 32 pixels. We keep its original resolution for training.
SVHN:SVHN (Netzer al. 2011 stands for Street View House Numbers.
widely used dataset for object recognition tasks in computer vision. The SVHN dataset consists of images of house numbers captured by Google's Street View vehicles. These images contain digit sequences that represent the house numbers displayed on buildings. It contains over 600,000 labeled images, which are split into three subsets: training set with approximately 73,257 images, validation set with around 26,032 images, and test set containing approximately 130,884 images. The images in the SVHN dataset are of size 32 x 32 pixels, we keep its original resolution for training.
Oxford Flower:
The Oxford Flower (Nilsback & Zisserman 2008) dataset, also known the Oxford 102 Flower dataset, is a widely used collection of images for fine-grained image classification tasks in computer vision. It consists of 102 different categories of flowers, with each category containing varying number of images. The Oxford Flower dataset provides a challenging testbed for researchers and practitioners due to the high intra-class variability among different flower species. This variability arises from variations in petal colors, shapes, and overall appearances across different species. The images in the Oxford Flower dataset are of size 256 x 256 pixels, in our training, we use bicubic interpolation to resize all images to 224 x 224 pixels.
Tiny ImageNet(Le & Yang 2015) is dataset derived from the larger ImageNet dataset, which is popular benchmark for object recognition tasks in computer vision. The Tiny ImageNet dataset is a downsized version of ImageNet, specifically designed for research purposes and computational constraints. While the original ImageNet dataset consists of millions of images spanning thousands of categories, the Tiny ImageNet dataset contains 200 different classes, each having 500 training images and 50 validation and test images. This results in total of 100,000 labeled images in the dataset. The images in the Tiny ImageNet dataset are of size 64 x 64 pixels, we keep its original resolution for training.
-mageema
MedMNIST (Yang et al. 2021) a specialized dataset designed for medical image analysis and machine learning tasks. It is inspired by the popular MNIST dataset, but instead of handwritten digits, MedMNIST focuses on medical imaging data. The MedMNIST dataset includes several sub-datasets, each corresponding different medical imaging modality task. Some
MedMNIST:
26

---

Published as a conference paper at ICLR 2024
examples of these sub-datasets are ChestX-ray, Dermatology, OCT (Optical Coherence Tomography), and Retinal Fundus. Each sub-dataset contains labeled images that are typically 28x28 pixels in size, resembling the format of the original MNIST dataset. We keep the original size for training. Due to the presence of both the RGB sub-dataset and grayseale sub-dataset in MedMNIST, we employed different model configurations accommodate this variation. The PathMNIST, DermaMNiST, BloodMNIST, and RetinaMNIST are RGB images, the rest 7 dataset are greyscale images.
ImageNet-C: ImageNet-C (Hendrycks & Dietterich 2019) is dataset of 75 common visual corrup- tions. This dataset serves as a benchmark for evaluating the resilience of machine learning models different forms of visual noise and distortions. This benchmark provides a more comprehensive understanding of model's performance and can lead to the development of more robust algorithms that perform well under wider range scenarios. The metric used in ImageNet-C: is mean corrup- tion error, which is the average error rate of 755 common visual corruptions of ImangeNet validation set. The lower the mCE, the better the result.
To ensure a fair comparison with the original paper that presented ResNet18, 88, we followed the training protocol outlined in the MedMNIST paper. We trained our model for 100 epochs without any data augmentation, using early stopping as the stopping criterion.
Table 17: The overview of the dataset use in Appendix E and Section 4.2 The in the below table since MedMNIST dataset consists of multiple sub-datasets, each containing medical images from specific categories. These sub-datasets have varying numbers of training samples, testing samples, and classes.
Dataset
CIFARI0
SVHN
Oxford Flower
Imagent
MedMNIST
Classes
10
10
102
200
Train samples
50000
73257
1020
100000
Test samples
10000
26032
6149
10000
Resolution
32
32
256
64
28
Attribute
Natural Images
Numbers RGB
Flowers RGB
Natural Images RGB
Medical Images RGB++reyscale
Image Type
RGB
M INITIALIZATION
We evaluate various initialization methods for the parameters. The results in Table indicate that the Xavier Normal initialization method yields the most favorable results. By adopting the Xavier Normal initialization,
observe improvements in the performance of the model trained CIFAR- 10. Our result in the main paper use Xavier Normal as our default initialzation. Previous works on PNs Chryssos et al. 2022 2023) have demonstrated the crucial role of appropriate parameter initialization in the final performance. In this section, we explore different parameter initialization methods for the linear layers in the model and train them on the CIFAR- 10 dataset from scratch. To minimize the impact of confounding factors, all experiments are conducted without any data augmentation techniques or regularization methods. The results are shown in Table 18
N ABLATION STUDY ON THE DESIGN OF MU-LAYER
Our original design in Eq. (1) factorizes A, but not A. One interesting question would be to explore factorizing also A as a low-rank decomposition. We follow the same idea as in A to factorize A as The new design is visualized in Fig. 16
A R- Q.
We conduct experiment CIFARIO assess the performance of the model when both branches assume low-rank decomposition. To provide fair comparison with the main model, we leave the secondary branch mentioned in the main paper. The results in Table 19 indicate that the original design performs favorably when compared to having low-rank decompositions in both branches.
27

---

Published as a conference paper at ICLR 2024
Table 18: Comparison Initialization Methods. The best performance marked in bold. Note that * in the table uses PyTorch default initialization, it depends the layer type. For linear layer of shape (out, in), the values are initialized from U (-Vk, vk), where it
Initialization Method
Top-1 Acc (%)
Top-5 Acc (%)
Xavier Uniform (Glorot & Bengio 2010)
88.85
99.38
Xavier Normal Glorot & Bengio 2010)
89.13
99.85
Kaiming Uniform (He et al. (2015)
88.56
98.70
Kaiming Normal (He et al.
2015
88.72
99.46
Lecun Normal (LeCun et al. 2002
88.95
99.51
Normal
88.06
99.34
Sparse Martens et al. 2010
88.18
99.48
Pytorch detaul("
88.37
99.49
Table 19: Ablation study the Mu-Layer design in Appendix N Although considering the second branch as low-rank decomposition performs generally worse, in some cases the difference is not large.
Baseline (ours)
Design
Design
Design 3
Design 4
Design 5
Design 6
The rank of Matrix RQ The rank of Matrix BD Acc
64
64
64
64
32
32
64
32
64
16
16
8
88.1
87.1
85.2
86.4
86.2
86.1
85.6
Two Branch Both Low Rank
Baseline(Ours)
X
R
B
B
Rank
Rank
A
Q
C
Rank
C
Unchange
*
Figure 16: Different designs of Mu-Layer as studied in Appendix N
28


Reviews:
[{"review_id": "JBoMcscA0H", "reviewer": "Reviewer_kEbz", "summary": "The authors propose a multilinear operator network which avoids nonlinear interactions. The architecture makes use of \"Poly-blocks\" composed of \"Mu layers\" which can capture up to fourth degree multiplicative interactions in input tokens. The authors demonstrate that their multilinear networks are competitive against other polynets, traditional MLPs, and ResNets on tasks such as image classification and ODE problems.", "questions": "Please see weaknesses. Is there some intentional restriction placed on the P-Net method? I believe you have pulled the numbers from [2], which is also in discrepancy with the first work (by the same authors, no less). Similarly, the ResNet 50 numbers are reported strangely. It can be easily fine-tuned to achieve 78.8%/94.5% Top1/5 accuracies, respectively, but this is not in concordance with Table 2. \n\n[2] Chrysos, G. G., Wang, B., Deng, J., & Cevher, V. (2023). Regularization of polynomial networks for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16123-16132).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The Mu layer design is simple enough and the theory underlying the layer is well-proven and explained.\n2. The MONets appear to be quite competitive against a variety of baselines on a wide array of tasks. \n3. The number of FLOPs and memory required to operate the MONet are drastically reduced compared to other baselines.", "weaknesses": "1. Fundamentally, it's unclear why the MONet should generalize well to other tasks if the Poly-block is only capturing up to fourth degree interactions. For instance, for general MLPs, results such as Cybenko's theorem tells us the MLP should capture many families of Lebesgue integrable functions but here the limit is restricted to functions that are captured by compositions of fourth degree approximations and skip connections. The surprising capacity of the MONet is not adequately discussed. \n\n2. The numbers reported for the P-Nets [1] do not appear to be accurate. With >11 parameter P-Nets, the achievable accuracy on CIFAR-10 is 94.5% and ImageNet ~77% Top-1 accuracy/ ~94% Top-5 accuracy. The exact accuracies reported in [1] would either closely-defeat or match the performance of the MONets with similar or drastically fewer numbers of parameters. Please address this discrepancy. \n\n\n[1] Chrysos, G. G., Moschoglou, S., Bouritsas, G., Deng, J., Panagakis, Y., & Zafeiriou, S. (2021). Deep polynomial neural networks. IEEE transactions on pattern analysis and machine intelligence, 44(8), 4021-4034.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RNpULTGxUD", "reviewer": "Reviewer_L73R", "summary": "The authors propose MONet, a Polynomial Network architecture that outperforms prior polynomial networks on image recognition tasks. The model uses stacked \"multi-linear\" operations to learn representations. The authors compare MONet models to state-of-the-art image classification models such as MLPMixer and ViT-16/B on image classification tasks, such as ImageNet1K and CIFAR10, along with medical image classification with MedMNIST and scientific computing with Neural ODE solver.", "questions": "* How is Mu-Layer novel compared to existing polynomial network architectures? Is the polynomial representation learned different, and why? \n* What is the VRAM usage of the MONet models compared to baselines in Table 1, does it need much more VRAM in practice to train than those baselines?\n* Why should be be interested in training ImageNet1K with MoNet instead of ViT or ResNet 50 which get comparable performance? In other words motivate your research work and method.\n* Explain the statement \"...which avoids the requirement for activation functions or other non-linear mappings\" in the conclusion. How is a polynomial mapping linear, or am I misunderstanding this statement?\n* Why are activation problems interesting to remove?\n* How are MONets more interpretable in the general case?\n* How are MONets more amenable to encryption than other models?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Achieves impressive results on ImageNet1K compared to existing polynomial network work compared to vision transformer and CNN baselines, notably better particularly in Top-5 accuracy, which should be the more informative metric.\n* Some good results in some of the categories for ImageNet-C under some image corruptions, such as weather and digital.\n* Results are compared with state-of-the-art image classification baseline models such as MLPMixer and ViTs amongst others.\n* Results are demonstrated on more than CIFAR-10/ImageNet with a tiny medical image classification dataset, and more interestingly, in the task of solving an ODE.\n* FLOPS analysis is given (and it is reasonable) in the appendix, and is presented along with model size in parameters when comparing with models in e.g. Table 2. FLOPS/parameters appear reasonable in Table 1 compared to other models, however it seems perhaps VRAM usage is much more (see below), which is practically the bottleneck in most GPU-based training.\n* One area where MONets may have more interpretability might be useful is in very specific cases, such as particular ODEs, that have a similar polynomial representation (as demonstrated in 4.4), but this seems like a very niche case.\n* An Ablation analysis on the proposed methodology in the architecture is performed.", "weaknesses": "* This is a poorly written and organized paper as it is, which unfortunately lets down what appear to be some very good results, and perhaps an interesting method/model architecture.\n* Virtually no motivation for the work aside from making polynomial networks have better generalization. We are given a sentence in the introduction \"Interpretability and encryption...\" with no reasons at all given as to why the proposed model architecture would be any more amenable to interpretation or encryption, and certainly no such results/analysis. Even the most academic work needs motivation. Getting closer to reaching state-of-the-art performance on some task alone is not enough of a motivation for a method if there are no tangible benefits over existing methods (and especially when it seems there are disadvantages in memory usage as covered below).\n* Similarly, why are we interested in getting rid of activation functions? No motivation is given for this explicitly, and the only reason I can think of given the interpretability word is to remove non-linearity in the mapping from input to output. A sentence in the conclusion backs this up: \"which avoids the requirement for activation functions or other non-linear mappings\". However, the \"multilinear\" operation in this work is itself still a non-linear mapping of input to output (a polynomial of degree 2), making this statement false.\n* The background is *way* too short and doesn't give near enough information on the current work's context in polynomial network literature.\n* While the method is detailed enough, with much more detail in the appendix on some of the most interesting bits, the paper doesn't explain much at all what specifically is different from current polynomial networks, seemingly assuming the reader should know this. It is up to the authors to explain how their method fits into the current research literature in the background and method and compare their method explicitly with the closest existing work in literature.\n* Requires 4 A100 GPUs to train for a batch size of 448 according to the authors (and they say they maximize batch size). This means that the maximum batch size on each GPU is 112. This is **much more VRAM usage** than the baselines being compared to that are achieving similar accuracy on ImageNet1K - for example, ViT B/16 can fit with a batch size of 256 on a single A100, and ResNet-50 a batch size of 512 on a single A100. Note it is not clear if these are 40GB or 80GB A100 models.\n* Bold figures in captions are misleading, and do not it turns out always represent the best result as you might expect! For example, in Table 2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9UXOueAYk", "reviewer": "Reviewer_6KA9", "summary": "This paper introduces a novel polynomial network called MONet, which exclusively relies on multilinear operators. The core layer of MONet is capable of capturing high-degree multiplicative channel-mixing operations for tokens, enabling non-linearity and, as a result, eliminating the need for activation layers. Extensive experiments were conducted on various image recognition and scientific computing benchmarks, demonstrating superior performance compared to previous polynomial networks and achieving results on par with state-of-the-art CNNs and Transformers.", "questions": "1. In solving ODE, could the authors provide more details regarding how the model recovers the right-hand side of the Lotka-Volterra formula?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The efficiency of this network design is truly impressive. In ImageNet, the Multi-stage MONet-S achieves performance comparable to the state-of-the-art, all without the use of any activations. It's also worth noting its robustness to image corruptions. When applied to solving ODEs, its performance appears strong based on the provided cases. The extensive experiments seem to highlight numerous promising characteristics of this new design, which warrant further exploration.\n2. MONet simplifies prior polynomial networks by utilizing mu-layers. It appears reasonable that mu-layers are designed to accommodate only two-degree polynomials, as stacking these mu-layers enables high-order interactions.", "weaknesses": "1. In mu-layer, is it possible to use low-rank matrices multiplication for A? How will it perform\n2. Is it feasible to employ low-rank matrix multiplication for A in the mu-layer? How would this alternative approach perform?\n3. In the ODE experiments, could the authors offer more detailed quantitative results and compare them with other state-of-the-art methods, such as NeuralODE?\n4. In the ODE experiment setup, is it necessary to use patch embeddings? Could you please provide additional details about the data processing and experimental settings for this specific task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)