Chopping Formers is what you need in Vision
Authors: Francesca Babiloni
Thomas Tanay
Matteo Maggioni
Jiankang Deng
Ales Leonardis
Stefanos Zafeiriou
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transformers
Tensor Decomposition
Deep learning Architectures

Abstract: This work presents a new dynamic and fully-connected layer (DFC) that generalizes existing layers and is free from hard inductive biases. Then, it describes how to factorize the DFC weights efficiently.
Using the Einstein convention as framework, we define the DFC as a fully connected layer with the weight tensor created as a function of the input. DFC is the non-linear extension of the most general case of linear layer for neural network, and therefore all major neural network layers, from convolution to self-attention, are particular cases of DFCs. A stack of DFCs interleaved by non-linearities defines a new super-class of neural networks: \emph{Formers}.
DFC has four major characteristics: it is Dynamic and Spatially Adaptive, it has a Global Receptive Field, and it mixes all the available channels' information. 
In their complete form, DFCs are powerful layers free from hard inductive biases, but their use is limited in practice by their prohibitive computational cost. To overcome this limitation and deploy DFC in real computer-vision applications, we propose to use the CP decomposition, showing that it is possible to factorize the DFC layer into smaller, manageable blocks without losing any representational power. Finally, we propose ChoP'D Former, an architecture making use of a new decomposition of the DFC layer into five sequential operations, each incorporating one characteristic of the original DFC tensor. Chop'D Former leverages dynamic gating and integral image, achieves global spatial reasoning with constant time complexity, and has a receptive field that can adapt depending on the task. Extensive experiments demonstrate that our ChoP'D Former is competitive with state-of-the-art results on three well-known computer vision benchmarks, namely Large-Scale Classification, Object Detection, and Instance Segmentation, suppressing the need for expensive architecture search and hyperparameter optimization. 

Under review as a conference paper at ICLR 2023
CHOPPING FORMERS IS WHAT YOU NEED IN VISION
Anonymous authors Paper under double-bling review
ABSTRACT
This work presents a new dynamic and fully-connected layer (DFC) that general- izes existing layers and is free from hard inductive biases. Then, it describes how factorize the DFC weights efficiently. Using the Einstein notation's 's convention as framework, we define the DFC as a fully connected layer with the weight ten- sor created as a function the input. DFC the non-linear extension of the most general case of linear layer for neural network, and therefore all major neural net- work layers, from convolution to self-attention, are particular cases of DFCs. A stack of DFCs interleaved by non-linearities defines new super-class of neural networks: Formers. DFC has four major characteristics: i) it is dynamic, ii) it is spatially adaptive, iii) it has global receptive field, and iv) it mixes all the avail- able channels' information. In their complete form, DFCs are powerful layers free from hard inductive biases, but their use is limited in practice by their pro- hibitive computational cost. To overcome this limitation and deploy DFC in real computer vision applications, we propose use the CP Decomposition, show- ing that it is possible to factorize the DFC layer into smaller, manageable blocks without losing any representational power. Finally, we propose ChoP'D Former, an architecture making use of new decomposition of the DFC layer into five sequential operations, each incorporating one characteristic of the original DFC tensor. Chop'D D Former leverages dynamic gating and integral image, achieves global spatial reasoning with constant time complexity, and has a receptive field that can adapt depending the task. Extensive experiments demonstrate that ChoP'D Former is competitive with ofate-of-the- art results on three well-known CV benchmarks, namely Large-Scale Classification, Object Detection, and In- stance Segmentation, suppressing the need for expensive architecture search and hyperparameter optimization.
INTRODUCTION
Convolutional Neural Networks (CNNs) have served as the undiscussed cornerstone of Computer Vision (CV) for the past decade thanks to convolutions, which, despite the hard inductive bi- ases of locally connected and shared weights, are able summarize spatial content very effi- ciently (Krizhevsky et af.) 2017] [Simonyan & Zisserman 2014 He et al.] 2016)] (Howard et al.) 2017 Tan Le| 2019). Nevertheless, in the 2020s, with the availability more abundant comput- ing resources, the role of convolutions has been challenged by the advent of Transformers (Vaswani al. 2017 Dosovitskiy al. 2020) and new "spatial-mixing" module, called Self-Altention, characterized by lighter inductive biases and high complexity.
The success of Vision Transformers (ViT) has long been attributed to Self-Attention However, new findings have recently questioned this narrative. For example, d'Ascoli et al. (2021): Wa et al. (2021 ; Liu et al. (202Tb) highlight the importance of convolutional biases Transformers for CV. Liu et al. (2022): Yu et al. (2022) demonstrate how macro design choices and training procedures alone can be sufficient to achieve competitive performance regardless of the specific spatial module used. Finally, Cordonnier et al. (2019;; Han al. (2021) 1) comment on the close link between convo- lution and Self-Attention formulations, hence blurring the line between these seemingly orthogonal operators. Here, take providing a unifying and intuitive formulation that clarifies spatial modules' role in modern archi- tectures and links existing work together.
step toward bridging the gap between CNNs and Transformers by

---

Under review as a conference paper at ICLR 2023
First, we use Einstein's tensor notation combined with tensor CP Decomposition to provide practi- cal yet principled analysis of existing literature. In essence, the principal ingredients in deep learning architectures are multi-dimensional operations that can naturally be written as decomposed tensor expressions. Here, the Einstein notation provides an elegant way to analyze neural network operators by highlighting differences among layers with intuitive notation that simplifies multi-dimensional matrix algebra (Kolda & Bader] [2009; Panagakis et al. [202]] Hayashi et al. (019)) with no compro- mises in formal accuracy. Under this lens, we formalize a generalization of existing layers with new dynamic, spatially adaptive, and fully connected building block for Neural Networks (the DFC) that represents the general but computationally complex operation of extracting the complete set of interactions within the input.
Second, we use DFCs to define super-class of neural networks, which we call Formers, where the dense and heavy DFC operators are used to create hierarchical representations of the input images. Then, to target real-world applications, usually bounded by tight computational budgets, we explore the use of CP Decomposition biases in their design. In this light, we show that Transformers' architectures can be seen as one of the possible instances of Formers and go step further by proposing new ChoP'D D Former variant. ChoP'D D Former leverages CP Decomposition, dynamic gating, and integral images to "chop" the general but prohibitively complex DFC into sequence of efficient transformations that have the potential to retain its full representational power. In particular, we identify five specific modules that can model the dynamicity with respect to the input, the adaptivity with respect to the spatial positions, and the long-range interactions via a dynamic receptive field with an overall complexity independent of the number of input tokens.
decrease Formers' complexity and integrate different inductive
Finally, this new perspective allows us to justify the empirical success of (Trans) and dis- entangle the contributions of each of their characteristics. To do so, we programmatically compare different layers and -P-Decomposed architectures on various small-scale and large-scale CV tasks. Our experiments indicate that CP- CCPDeecmposed DFC layers can effectively approximate the full DFC at significantly reduced cost, considerably outperforming its simplified variants. In conclusion, our contributions can be summarized as follows:
We provide unifying view on building blocks for neural networks that generalizes and compares existing methods via Einstein's 's notation and CP Decomposition, with a notation that deals with multi-dimensional tensor expressions without resorting to heavy tensor algebra.
We show how to use complete tensor operator that is spatially adaptive, fully connected, and dynamic (DFC) to create general neural networks, which we dub "Formers".
We connect our formulation to existing architectures by showing how Transformer and its variants can be seen as a stack of CP-Decomposed DFC operands for neural networks.
We propose ChoP'D Former, new variant of Former architecture, which is able to approximate the full DFC with a complexity comparable to a convolution with a small kernel, and is able to match, if not improve, SoTA performance on several benchmarks, including large scale classifica- tion, object detection, and instance segmentation.
2 EINSTEIN NOTATION FOR NEURAL NETWORKS
At their core, neural networks and deep learning architectures in particular are commonly built as a sequence of tensor operations (i.c., building blocks) interleaved by point-wise non-lineasssies. Tremendous interest has been dedicated to the form of such building blocks (e.g.. "MLP", "Con- volution", "Residual Block", "Dense Block", "Deformable Conv", "Attention", "Dynamic-Conv", etc.) as these are the critical components extract various meaningful information from the input. In this section, we present a general form of a neural network layer and showcase how the Einstein summation convention can be used as an alternative, short-hand and self-contained way to represent and relate building blocks for neural networks.
2.1 BACKGROUND
Einstein notation. In the rest of the paper, we adopt the notation of Laue et al. 2020). Tensors are denoted with uppercase letters and indices to the dimensions of the tensors denoted in lowercase subscripts. For instance Xiyk E R Jx is a
tensor of size I x J x K with three modes (or dimensions) indexed by i [1, T E [1, and & [1, K). Using the Einstein notation,
2

---

Under review as a conference paper at ICLR 2023
any multiplication among tensors can be written as: C. I( Uee)) As, B., where 81, 82, and 33 are the index sets of the left argument, the right argument, and the result tensor, respectively. The summation is only relevant for inner products and is made explicit by underlining tensor indexes. review
As representative example illustrate our notation, lensors. Given the tensors of order two X E RIXJ,, their Hadamard product can be written as Zi XiyYi and equivalent the algebraic notation Z X Y. Similarly, their matrix- product be written Zip Xa.Y5 and is equivalent to the algebraic notation Z XYT. Given the tensors of order one Y R and X RR' their outer product creates tensor Z E R as Zi X,Y,. It is equivalent to the algebraic expression Z = YTX.
of common operations among
CP Decomposition. The CP Decomposition also referred to as CANDECOMPPARAFACC or polyadic decomposition, is used express the factorization of multi-dimensional tensor as linear combination of components with rank one, and thus generalizes the concept of matrix singular value decomposition (SVD) tensors (Kolda & Bader, 2009). For example, let Xijk E Rix be tensor, then can define the CP Decomposition Xisk U.U as the approximation the original tensor from set of three factor matrices [U]r, U U.r. The rank of the tensor Xijk is defined as the smallest number of I components needed to generate equality in the CP Decomposition. Note that we call a CP Decomposition canonical whenever R is equal to the rank of Xisk-
2.2 THE DYNAMIC FULLY CONNECTED LAYER FOR NEURAL NETWORKS
A neural network layer is a function I' that takes as input tensor Xmc composed of m E [1, M] spatial positions (or tokens) with c E [1, C] features (or channels) and produces as output a tensor Ynd composed of [1. N] tokens with d [1, D] channels:
Ynd f f(mme)
In the following, we start by considering the special case where f is linear function before intro- ducing the more general dynamic fully-connected layer.
Linear Layers. The most general instantiation of linear neural network layer is the Fully- Connected layer (FC):
Ynd = X W Wmned
parametrized by a four-dimensional weight tensor W Wmned used to mix all spatial and channels in- formation in the input. The complexity is O(M-N .CC.D) which makes the FC layer computationally expensive, if not prohibitive, in CV tasks. However, complexity can be reduced by using priors such as weight sharing and local processing. Using the Einsum notation, we show in the Appendix that the convolutional layer, its depth-wise and point-wise variants, and the average pooling layer are special cases of the FC layer.
Dynamic Layers.
non-linear generalization of the FC layer can be obtained by turning the weight tensor into function g of the input: W W'nned ((mm)). To illustrate that the tensor W Wmned is not constant anymore but the result of a dynamic construction mechanism, we now consider a "batch" of input instances Xime created as stack of / inputs [X X2 me: ..., X! and the corre- sponding batch of output instances Yind both indexed by the new instance dimension E call such layer Dynamic Colly-Connected layer (DFC):
[1, I]. We
Yind Ximo Wirned with Wimnedd =
g( L.))
(3)
As in the FC layer, the DFC generates the output by mixing all spatial and channels information. On top of that, DFC is spatially adaptive, i.c. weights are not shared across spatial positions and dynamic or instance adaptive, i.e. processes each input differently. We wish to stress the rela- tionship between FC and DFC: i) every instance of FC is also an instance of DFC (i.e. DFC where function g is constant) and ii) there are instances of DFC that are not FC (i.e. DFC where function g is non-constant). Therefore FC is special case of DFC. Again, a number of well-known neural network layers can be framed as simplified cases of DFC. We show in the Appendix and using the Einstein notation that this is the case for Self-Attention (Vaswani et al.) 2017 Wang et al.) 2018
Dynamic Convolution (Wu et al. 2019 Hu et al. 2018) Deformable Convolution (Dai et al. 2017
Zhu et al. 2019)
3

---

Under review as a conference paper at ICLR 2023
(a)
(b)
Iistaness : [D.F]
DycCousettton
Ingut M]
Comvecation
Vineed
Ontput Tokere e [tNN]
Insut Claanel of [I.C]
Conte-Ce
Octput Clannds [1.2|
W
SelFAttcttion
Spetial Mixing
W
immed
Channel Mixing
Acapttee
DCG
Winsood
Dynamic
Figure/ 1: Part (a): Overview of Building Blocks Characteristics. The tensor W Vmmeed is repre- sentation of a general neural network layer and each of its dimensions is associated with one charac- teristic that can be used to describe existing building blocks. For example, Convolution layer has Token and Channel mixing, but not Dynamic nor Adaptive. DFC incorporates all possible char- acteristics in its formulation. Part (b): Overview of CP Decomposition. Incorporating additional characteristics in a layer has the side-effect of increasing the dimensions of the underlying parame- ter tensors. CP decomposition can be used tool approximate the parameter tensor structure while decreasing its complexity. Figure reports examples for Fully Connected layer (bluc), Self- Attention matrix (cyan), DFC layer (green).
As visible in Figure I. DFC represents a general way to leverage the complete range of interactions of the input and serves generalization of existing layers. However, the usage of the DFC is severely limited practice because of the dense structure of its weights and its high complexity, which is equal the complexity of the FC layer plus the complexity of the function g. Consequently, the question that naturally arises is: "Is it possible to use these layers efficiently in a neural network without using any strong prior on their weights?" ?" Next, instead of relying on sharing and grouping strategies, we propose to use CP Decomposition as mean to decrease DFC complexity.
2.3 "FORMERS" ARCHITECTURES VIA CP DECOMPOSITION
In this section, we describe how create efficient DFC networks. We use the CP Decomposition as the mathematical backbone to define and design compact and lightweight DFC Neural Networks, in which each layer approximates the behavior of DFC layer and implements spatial reasoning with complexity independent of the number of input positions, thus achieving overall complexity comparable to a standard convolutional layer with a small kernel.
Formers. A Former is a general architecture that models hierarchical non-lineat interactions by stacking series of blocks
Yind = a(X W iinged + Bud).
including a DFC layer with parameters Wimned, matrix of biases Bnd. and non-linear element- wise function o (e.g. GeLU or ReLU). In the following, we call DFC layer the linear operation inside the non-linearity, DFC block the DFC layer plus non-linearity, and Former an architecture consisting of one or multiple DFC blocks. discussed in Section 2.2] the use of parameters tensor W imned makes the DFC block general, but also its computation heavy.
Formers CP Decomposition. To overcome this limitation, we propose to factorize the weights of the DFC layer through its CP Decomposition:
Wimned U.UUZ U3 U.UUUU. + Camned
(5)
which represents the full tensor Wimned as linear combination of lower-dimensional factor matri- ces plus an approximation error Eimnned dependent on the choice of Rt Typically, lower R implies larger errors, while for R 2 rank(Winn.. (mimnea)) the error is zero, and the CP Decomposition is exact.
Hence, we can define the Formers CP Decomposition by replacing the DFC layer in equation with equation 5 and rearranging terms
(6) where Ximme is the result of unfolding the input Xime with a global receptive field of size N for all N output positions]] Equation
Yind ==((=(<((((eeee Jime U2 Dinr U3 ) U Uini U3 + Bnd)
6 acts low-complexity substitute of equation 4 and can be
In general, we define as "unfolding with receptive field K" the operation of rearranging the input as collection of N sliding patches of size K.
4

---

Under review as a conference paper at ICLR 2023
easily learned end-to-end sequence of fully differentiable building blocks for neural networks. In practice, as apparent from the superscripts of the U matrices in equation 6 we identify five individual - and specific operations inside DFC block: i) U channel-mixing layer embedding the C' input channels into the R-space described by the CP Decomposition; ii) U a spatial layer mixing spatial information with global receptive field, implemented as depth-wise convolution; iii) U gating layer ensuring spatial adaptivity by modulating spatial information; iv) U.,:: a gating layer generating dynamic response conditioned on the input Ximei v) U : channel-mixing layer which combines the R channels to create D output channels. Replacing DFC layers with the CP Decomposition of equation O reduces drastically the memory needed to store the weights from (M N C. D) to (l. Rt., L = maz M, D). It also reduces its computational complexity as the sum of its sequence of operations: (MM-C CC- R) + (M- N N.R) ON R) + O(n) + O(N D n),, plus the complexity of the function used to create matrix U..-
6 iss the most general representation of the CCP-Decoposed Former, derived directly from applying the CP Decomposition to equation 4 Intriguingly, particular cases of equation O can be derived by making assumptions on the factor matrices, thus generating alternative Formers architec- tures characterized by different inductive biases and complexities. Next, we showcase two particular approximation cases of equation[o] the Transformers, characterized by heavy computational require- ments, and the Chop' D Former, our new efficient variant.
Equation
Transformer. The Transformer (Vaswani et al.) 2017) is one of the most well-established and recognized designs for neural networks. The architecture is built from cascade of inverted residual bottleneck blocks, including multi-headed self-attentio block, two channel-mixing layers, and As introduced above, the transformer block is also particular case of GeLU non-linearity ?
equation 6 as
U. + Bnd) Yind = o(((Ximng Uer) Grmin U23 Ding dr
= <[(((((Xxmme W1 Jimny W )rimn UU123 )inr V2r inr V2g + Brn)
where a is GeLU nonlinearity, the matrices U4 and Uăr are the channel-mixiee inverted bottleneck, and the remaining U.... U2 U3 are combined together into a single Umnar to create a dynamic spatial mixing layer with global receptive field. Moreover, the U U122 is assumed to be built via a self-attention mechanism only for a subset H of the channels and then repeated across the r dimension. In other words, equation 7 is CP Decomposition for DFC block with three extra assumptions on its factor matrice Under a similar light, it is easy to recognize how variants of the transformer block can be analogously framed as CP-Decomposed Formers, under different set of assumptions for the DFC layer factor matrices. This design can process input of various sizes but has two main disadvantages when compared with equation 6 First, it requires smar has to be generated (and also stored in memory) all at once. Second, its computational complexity is: O(M (.M-C-R) RR) + O(M- .....))) ON D- R). plus the complexity of the self-attention mechanism used to create the tensor U122 Moreover, as in equation 6 equation, its complexity scales quadratically with the number of tokens used, which can be computationally really expensive even in the case of moderately sized inputs.
layers of the
higher memory requirements since the parameter tensor U U122
imar-
ChoP'D Formers. From the discussions above, it might seem that equation good candidate for an efficient implementation of DFC neural networks. However, in cases where the spatial size is large compared to the number of channels (i.e. L= U3- of equation 6 act as computation bottlenecks. In fact, implementing the factor matrix U2., as a desth-wise convolution with a global receptive field requires number of parameters directly proportional to the number of spatial positions. Similarly, the gating layer U3 has to allocate a parameter for each of the tokens considered. For these reasons, the application of equation 6 is limited to cases where the spatial size of the data is known in advance and small enough to fit in memory. To overcome such limitations, we propose the following modification for equation [6]
L. N), the factor matrices U2 and
First, we replace the operator U2 with more efficient spatial mixing module based on Summed- Area Table (SAT). SAT, also known as an integral image, data structure that can be used to perform fast image filtering (Crow) 1984 Viola & Jones 2004) and enables the computation of
Without lack of generality, we omit at this stage the Layernorm (LN) applied before every block and the residual connections, which are used after every block (Wang et al. 2017)
(i) U W. (ii) U2- Vile Vil (iii) 123 125
Unnnh
(Qingh Kingh).
5

---

Under review as a conference paper at ICLR 2023
Layer Up
Layer 03
Linear Layer 12.
W inmed
Self-A ltention Cieee
Liner Uieer
Modilation Eo
Layer t2
Loyer
Layer U2
GtLU
G+LU
U
GaL
TramFoormr
TramsFormer
Chorrre (Ours)
(NN*)
(N
(N)
((K)
Figure 2: Former Architecture is a stack of DFC layers, in practice often decomposed in a
sequence of smaller blocks. Although many alternative decompositions exist for the tensor W Wmmned they do not come all with the same complexity and inductive bias. The original Transformer has a spatial mixing module that is burdened by quadratic complexity O(N2), unpractical in many real cases. In contrast, our "ChoP"D D- F-Fomerr" token mixer comes with a complexity that is independent of the number of tokens considered.
pooling operations on a receptive field of arbitrary size with a constant computational cost. Ad- ditionally, SAT can be used to implement pooling operation on a learned receptive field (Zhang et al. 2019). Thus, we propose to decompose the contribution of the factor matrix 2 mer as follows:
U mr = Purg Erg
where Prrg is collection of G fully differentiable pooling layers with learnable receptive field, and Erg is the set of learnable weights used to combine their contribution. The advantages are two-fold: i) the model is able to actively learn the optimal receptive field, opting for global or local reasoning for the task at hand, and ii) spatial mixing can be performed at constant computational cost even when the receptive field is global.
Another key modification consists in reducing the memory and computational requirements of the spatial adaptive operators. Specifically, we propose to combine the effect of the two gating U.. and U3 into a single operator as
U13 U3 ((Ximc),
(9)
where the function d parametrized via a small CNN, generates dynamic and spatially adaptive weights conditioned the input Xime- To further limit complexity, the input of á can be down- sampled to a pre-defined fixed size, and then the output can be upsampled to match the original resolution, e.g., by interpolation [ As result, the complexity of this spatial adaptive operation is again constant with respect to the number of spatial positions N. This allows our formulation to achieve global reasoning with a complexity independent of the number of spatial positions, drastic improvement when compared with the spatial reasoning module of transformers (i.e. self-attention) and equation 66, both of which have quadratic complexity with respect to the number of tokens. Fi- nally, we can introduce our proposed CP-Decomposed Formers with learnable Pooling, or ChoP'D) Former for short, by replacing equation g and equation 9 in equation
6 as
Yin aa((((((((()))) iinng U Jimnr P
13 mag (irg) Erg) air inr
US Bud).
(10)
This formulation, also illustrated in Figure 2 is a CP Decomposition for a DFC block with two extra assumptions on its factor matrices. We recognize several desirable properties: firstly, it can be applied to inputs of arbitrary resolutions without compromises on the size of the receptive field, and secondly, when the number of spatial positions is higher than the number of channels, the overall computational complexity is reduced from O(M- NN) to (Ar).
3 RELATED WORK
Link between Attention and Convolutions. Han et al. [2021) comments on how the design of local self-attention resembles dynamic and depth-wise convolution with no weight sharing. Cordonnier et al. (2019) provides proof that multim self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Pérezz et al. (2019) shows that transformers are Tur- ing complete. We explore the relationship between these seemingly opposed processing paradigms when interpreted as dynamic layers (Section 2.2, Appendix), and extend this line of research by pro- viding a general framework to compare building blocks as well architecture designs Section
*The resizing functions are assumed he absorbed into e for the sake of notation simplicity.

---

Under review as a conference paper at ICLR 2023
Dynamic Neural Network Layers, The idea of using layer whose weights are adaptive to the
input can be traced back to early CNNs using max- max-pooling Qarrett et al. 2009).
Dynamic convo-
lutions emerged multiple times in the context of low-level vision (Jia et al.
[2016] Mildenhall et al.
2018
Xia et al., 2020) as well as high-level vision (Ha et al. 2016 Chen et al. 2020; Wu et al,
2019
The dynamic component is also neglected feature of attention mechanisms (Vaswani al.
2017
Hu et al. 2018), and we identify it here as the key to unlocking non-linear behavior.
Tensor Decomposition for Neural Networks. Tensor Decomposition an active area of research dedicated to the study of low-rank approximation for multi collii-dmeenional arrays and has applications in variety of fields, ranging from psychology to CV (Kolda Bader 2009) Panagakis et al. 2021).
Tensor decomposition techniques have been used to reparameterize neural network layers in or-
der to speed up their inference (Chrysos et al., 2021 2022 Ma et al., [ et al. (2014) and Novikov et al.] (215)) used CP Decomposition to speed up spatial static convolutional and FC layers. Kossaifi et al. (2020) extended this idea to spatio-temport static convolutional kernels. Dif- ferently from these works, we focus non-lineat dynamic layers and extend this trend of research by investigating a tensor decomposition for a "Dynamic Fully Connected" layer (Section 2.3 Tensor Notation for Neural Networks. Einstein notation's 's convention provides an intuitive no- tation for tensor manipulations. In machine learning, it can be used as an alternative to tensor algebra (Panagakis et al. 2021 Hayashi et al. 2019). Recently, the Einstein notation has gained traction as a practical way to improve code readability (Rogozhnikov) 2021 Rocktäschel 2018) and enable efficient tensor calculus (Laue et al., 2020). Here (Section 2.2 2.3 and Appendix) we use the Einstein notation as a way to compare building blocks for neural networks.
Summed Area Tables for Neural Networks. Summed Area Tables (SAT) an established algo- rithm in CV (Crow 1984] Viola & Jones (004) that is able to provide the sum of values within an arbitrary subset of a grid in constant time. Recently, SAT has been used to accelerate large-kerne convolution in dense prediction network for Human Pose Estimation (Zhang et al. 2019) and
dynamic large-kernel convolutions in language tasks (Lioutas & Guo. 2020). Similarly in Trans- formers. SAT enabled fast computation of a linearized attention variant (Zheng et al. 2022) and
parameter-free method to adapt size of the area to attend (Li et al. 2019). As described in Sec- tion 2.3 we leverage SAT achieve an efficient CP Decomposition for DFC layer and show its application in Formers for CV.
4 EXPERIMENTS
In this section, we report the experimental evaluation of Chop'D Former in a wide range of CV tasks. We start by comparing our CP- CP-Decomposed DFC layer of equation [0 against other possible variants in a pre-existing network to assess the contribution of individual components in a controlled setting. Then, extend our findings to more complex cases by stacking several of such blocks to create architectures with different inductive biases.
Puzzle Reconstruction The DFC layer acts as a non-linear extension of an FC layer. It has four main characteristics: being dynamic, being spatially adaptive, having a global receptive field, and mixing all the channel information. To isolate the contribution of each of these characteristies to the overall performance, we compare a DFC layer against its simplified variants: i) fully connected layer (not dynamic), ii) a convolutional layer (not dynamic, local receptive field), iii) a spatial layer represented by depth-wise convolution (not dynamic, local receptive field, channel mixing), iv) a pooling layer (not dynamic, local receptive field, no channel mixing, weights all ones) and v) channel-mixing layer represented by point-wise convolution (not dynamic, spatial mixing). Moreover, we compare its formulation with our efficient CP-Decomposed DFC variant in equa- tion 10 which, we recall, is capable to approximate the full DFC weight tensor W Wimned via CP Decomposition. Figure 33 (right) shows the breakdown of different layers in terms of complexity, characteristics, and size of the weight parameter tensors. To compare methods, we use the small- scale but challenging task of "puzzle reconstruction", where four-layers encoder-decoder network used to reconstruct an image from a "cut and shuffled" version of itself. Specifically, we ob- tain input and ground-truth pairs by dividing each sample of the MNIST dataset into 16 different patches, randomly rotating each piece, and shuffling their relative position before stitching them back together. Some examples of input and ground through pairs are visible in Figure We test the ability of different layers to enrich the representation of a network, by placing them between the encoder and the decoder used in an image- to-gge translation task. Figure (left) reports valida- tion curves for the compared methods using PSNR as the performance metric, which is a common metric used to assess pixel accuracy (higher is better). Results demonstrate that DFC layer is able
7

---

Under review as a conference paper at ICLR 2023
Layer
Weights
Form
o0
Adapticity
Receptive
Field
Dynamic FC
Wasmed
Olar cq)
Global
Fully Connected
W Wasod
O e-)
Global
Comolluional
W Waod
Olaxxee)
Local
Sparial mixing
Win
O(s.c.n)
Local
Conndrizeal Laver
Oueanted
Pooling
O(a.x.n)
Local
Seridl viin Laper
Channel mixing
W
O(siet)
Pogtag
Cheasd vnn Laye
Dynamic FC CP
Inmod
Olsic
Global
Overview of different Layers
Figure 3: Puzzle Reconstruction on MNIST. Overview of Layers right. Methods are described by complexity and flagged with an if dynamic, and with an n if spatially-adaptive. M.N indicates input and output tokens, C.D input and output channels, Jf convolutional groups, K convolutional ker- nel size, R the CP decomposition size. We assume: M N, C D, & N, и < C, it < C. Comparisons among Layers of Neural Networks - left. PSNR validation curves show how our CP-DFC method is capable to approximate a general DFC layer performance.
8
O
G
7 4
6
x
3 2
Input
Channel-mix
Convolutional
DFC-CP Layer
DFC Layer
GT
Wed
Wkod
.W WWinmed
W Winmed
Figure 4: Qualitative Comparison on MNIST dataset. Augmenting the size of the weights tensor
boosts output quality. Our DFC variant (DFC-CP) uses global receptive field and dynamic weights prediction and is able generate clean outputs and sharp digits. Moreover, it uses CP decomposition to reduce complexity, performing on par with the computationally expensive DFC layer.
to leverage the entire content of the encoded features while adapting its weights depending on the input. Although DFC achieves the best performance, it is also the layer with the highest complexity. Notably, our CP-Decomposed DFC layer approximates DFC behavior at a fraction of the compu- tational cost. The other layers, which are simplified cases of DFC, have lower complexity but also exhibit significantly lower performance.
Classification, Segmentation, Detection We extend the previous section by scaling up our analy- sis to different classes of architectures consisting of a stack of the compared layers, and three large- scale well-known CV applications: Image Classification on the ImageNet dataset, Object Detection, and Segmentation on the COCO dataset. In a similar spirit as before, we compare side-by- by-side four types of architectures created by progressively silencing various characteristics of the DFC layer. We use Former (i.e. a stack of DFC layers), MLP (i.e. a stack of FC layers), CNNs (i.e. a stack of Convolutions), and Linear Network (i.e. stack of fhaanell-miiing Layers). To separate the contri- bution of macro design choices and building blocks, we fix an overall network design. Specifically, following the best practice of et al. (2021b), we use a 4-stages hierarchical network with stage compute ratio of 1:1:3:1. We build each stage as stack of layers and GeLU non- non-lineasities. We explore two different sizes: 15M parameters and 28M parameters. Given the well-known correlation between network complexity and final performance, we roughly match FLOPss and parameters count across methods with two strategies: i) we control for each method the number of overall features used at every stage, and ii) we use CP Decomposition to separate the tensors of weights into smaller [ Table reports the performance of Chop' D Former of equation 10 against its simplified variants: first, C-Deccmppseed MLP which drops the characteristic of dynamic weights but still achieves global reasoning via spatial layer; then, two variants of CP-Decomposed CNNs (Lebe- dev al. 2014 Howard et al.] (017) which drop global reasoning in favor of local processing. The first variant uses depth-wise convolutions as spatial layers, while the other uses non-adaptive average pooling to mix spatial information. Lastly, as a lower bound for performance, we use cP-Decomposed linear network which is only able to process spatial information through the four pooling layers across stages. Results clearly show a performance progression that closely mimics the scall-scale scenario and is consistent across tasks and architecture sizes. Remarkably, the linear network can generalize relatively well across tasks, even if its spatial receptive field is only pixel
matrices
Wee refer supplementary materials for implementation details, and training hyperparameters.
8

---

Under review as a conference paper at ICLR 2023
Layer
Archisecturs
Complediy
Chociltation
Detection
Segmentaiion
Bype
Wiight
P(M)
RCG)
T5
v2.
Real
AP"
Al
Al
AP"I
AP
CR
Winned
Focner (Chop')) D)
28
4.5
2,4
K2.8
80.9
95.4
98.6
7066
69.6
86.6
86.7
40.1
42.4
61.4
636
43,
46.7
37.1
56.7
58.6
29,6
41A
PC-CP
unne
80.7
96.7
ConoCCP
Want
CNN (DooCCom))
76.9
57.9
36.2
Con-CCP
a Para
CNN (Phol)
23
15
24
4.5
80.6
78.5
95.0
94.0
67.0
68.6
54.3
85.8
38.01
40.7
59.5
62.5
41.3
44.
15.5
37.3
56.6
59.7
57.6
39.3
Linex-CP
Wat
244
23.9
91.4
60.9
80.8
29.7
50.4
31.0
28.9
47.7
30.4
4.5
26.3
92.7
65.7
82.6
30.9
$2.0
33.3
30.0
49.5
31.6
Table 1: Comparisons among CP decomposition for different Class of Architectures on Large scale classification on Imagenet and Detection and Segmentation on COCO using Mask-RCNY and a 1x training schedule. Chop'D Former approximates via CP decomposition DFC layers and outperforms less complex Neural Networks. Note that MLP cannot process the input of variable sizes and thus cannot be used as the backbone for Detection and Segmentation tasks.
Architecture
Classification
Name
Type
Token Mixer
Adaptivity [5, x.)
Receptive Field
Params
Flops (G)
TI
(M)
CoAINes- Dai et 2021
Hybrid
Conr/BCCoew/GlooulSSS
i.
3x33CGlobol
25
4.2
81.6
P0
2023
CNN
Pooling
3x3
31
5.0
81.4
ConvNext-T Liree
CNN
Depthwise-oom Conv
7x7
29
4.5
82.1
RSB-RexNet-SU
2021
CNN
Convolation
26
4.1
79,8
Swin-T [iu
DCNN
Local-Self Anention (SA)
29
4.5
81.3
GFNet.
Rio
2021
MLP
FFT
Global
32
4.6
81.5
gMLP-S ct
20213
Former
Gated-MLP
Global
20
4.5
79.6
Deit-S
Global-SA
Global
4.6
79.8
Chop'l Drrommr-
Former
Gated-SAT
4,20
Global
28
4.5
82.0
Table 2: Comparisons with other architectures for Large Scale Classification. Methods are trained
on Imagenet-Ik IK input image size 224 224 and have complexity between and GFLOPS.
wide (~ 74 TI, ~ 30AP* and ~ 29AP for the smallest of the two sizes). As apparent from the table, the CNNs achieve better results, but the use of spatial information is still limited by local processing and shared response among spatial positions and instances. Interestingly, forcing a static global receptive field is not helpful, as shown by the fact that the MLP network does not signifi- cantly outperform the CNNs. Moreover, the MLP network cannot process inputs of various sizes and cannot be used backbone for detection and segmentation tasks. On the contrary, our Chop'l D Former network approximates a set of DFC layers, calibrates its weights according to the input, and can integrate long-range interactions, outperforming CNNs variants by large margin on both the 15M and 28M parameter variants. Chop D Former gains an impressive +2 and +1.1 TI in Clas- sification. Similarly, Chop') D Former boosts results by +1.4 and +0.9AP* in detection and +1.3 and +0.5AP"M in segmentation. Comparisons against state-of-thee afetthe-rt networks of comparable size are presented in Table [ and expanded in the supplementary material. Without bells and whistles, Chop' Former remains competitive against various architectural designs and offers a good trade- off between complexity and accuracy. It maintains minimal gap with the best hest-performing method (-0.17 TI) and vastly outperforms other established Former architectural variants (+2.2 TI).
CONCLUSION
This work presents new general layer for neural networks, "the DFC", a non-linear generalization for FC layer, and new architecture design, "the Former", built as stack of DFC blocks. A DFC is dynamic, spatially adaptive, and fully connected but demands high computational requirements for deployment in real-case scenarios. To use Former architectures in CV applications, we propose to look through the lens of a unifying framework, based on CP Decomposition and Einstein notation, able to disentangle the individual characteristics of DFCs into separate components. Hence, we cast Transformers and their variants CP-Decomposed Formers using different assumptions on the factor matrices and, consequently, distinct inductive biases. Then, we propose the Chop'l D Former, a new hierarchical backbone for CV that approximates DFC blocks via CP Decomposition, leveraging the entire range of interactions via five sequential operations, including a spatial-min -ixxng module with cost independent of the number of input positions. Lastly, empirically demonstrate how each characteristic of DFC contributes to the overall performance, and we show that our CP- Decomposed k.. Chop' D) Former achieve state-of-the-art results various CV benchmarks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11030-11039, 2020.
Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos Zafeiriou. Deep polynomial neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4021-4033 2021.
Grigorios G Chrysos. Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks, In European Conference on Computer Vision, pp. 692-716. Springer, 2022.
Jean-Bape Cran-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. arXiv preprint
2019.
Franklin C Crow. Summed-area tables for texture mapping. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques, pp. 207-212, 1984.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEÉE international conference computer vision, pp. 764-773, 2017.
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-39 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Stéphane d' Ascoli, Hugo Touvron, Matthew I. Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit:
Improving vision transformers with soft convolutional inductive biases. In Inter- national Conference on Machine Learning, pp. 2286-2296. PMLR, 2021.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 1609.661000 2016.
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth-wise convolution. In International Conference on Learning Representations, 2021.
Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, and Shin-ichi Maeda. Exploring unexplored ten- sor network decompositions for convolutional neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 174.v4170. 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciiee conference on computer vision and pattern recognition, pp. 7132-7141, 2018
networks. In Proceedings of the IEEE
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference on computer vision, pp. 2146-2153. IEEE, 2009.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. Advances in neural information processing systems, 29, 2016.
10

---

Under review as a conference paper at ICLR 2023
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.
Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M Hospedales, and Maja Pantic. Factorized higher-order cnns with an application spatio-temporal emotion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
6060-6069, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Sören Laue, Matthias Mitterreiter, and Joachim Giesen. A simple and efficient tensor calculus. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4527-4534, 2020.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6555: 2014.
Yang Li, Lukasz Kaiser, Samy Bengio, and Si Si. Area attention. In International Conference Machine Learning, pp. 3846-3855. PMLR, 2019.
Vasileios Lioutas and Yuhong Guo. Time-aware large kernel convolutions. In International Confer- ence on Machine Learning, pp. 6172-6183, PMLR, 2020.
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 14:9204-9215, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11976 1986, 2022.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. Advances in neural information processing systems, 32, 2019.
Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. Burst denoising with kernel prediction networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2502-2510, 2018.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Yannis Panagakis, Jean Kossaifi, Grigorios G Chrysos, James Oldfield, Mihalis A Nicolaou, Anima Anandkumar, and Stefanos Zafeiriou. Tensor methods in computer vision and deep learning. Proceedings of the IEEE, 109(5):863-890, 2021.
Jorge Pérez, Javier Marinković, and Pablo Barceló. On the turing completeness of modern neural network architectures. arXiv preprint 1xXiv:1901.080 1901. 2019.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in Neural Information Processing Systems, 34:980-992 18800999,,000 2021.
Tim Rocktäschel. Einsum all you need- einstein summation in deep learning, 2018. URL
https://rocktt wwreckktttthttt github. 1o 20t///01///000 /t/////0/eennuu
Alex Rogazhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference on Learning Representations, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.

---

Under review as a conference paper at ICLR 2023
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Paul Viola and Michael J Jones. Robust real-time face detection. International journal of computer vision, 57(2)1337154, 2004.
Fei Wang, Mengqing Jiang. Chen Qian, Shuo Yang. Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2 2 10.00476, 2021.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901. 2019.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22-31, 2021.
Zhihao Xia, Federico Perazzi, Michaël Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti. Basis pre- diction networks for effective burst denoising with large kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11844-1 1853, 2020.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10819-10829, 2022.
Linguang Zhang, Maciej Halber, and Szymon Rusinkiewicz. Accelerating large-kemell convolution using summed-area tables. arXiv preprint arXiv:1906. 2019.
Lin Zheng, Huijie Pan, and Lingpeng Kong. Ripple attention for visual perception with sub- quadratic complexity. In International Conference on Machine Learning, pp. 26993-27010. PMLR, 2022.
Xizhou Zhu Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 9308-99 99169996,,660 2019.
12


Reviews:
[{"review_id": "wphhEIiVVE", "reviewer": "Reviewer_T6EK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel neural network layer architecture, which is derived by a tensor rank decomposition (CP decomposition) to a generalized version of a fully connected layer (namely, weights tensor of the FC that is created as a general function of the input). This new layer by design can be dynamic and spatially adaptive, it can create global receptive fields, and can mix channel information, therefore can accommodate the tensor operations for many of the existing known NN layers, such as self-attention or convolution layers.\n\nWhile the full version of the proposed generalized fully connected layer is costly and has many parameters, its CP decomposion version has reduced computational complexity that makes it comparable with SOTA vision backbones, such as Swin-Transformer and ConvNextT models.  The authors conduct experiments on ImageNet-1K and show comparable top-1 accuracy as well. \n", "summary_of_the_review": "Novel architecture based on interesting perspective on formulation of neural network layers.", "strength_and_weaknesses": "Strength:\n\n(*) The paper proposes an interesting perspective on formulation of neural network layers\n\n(*) The idea of forming the proposed generalized fully connected, and then apply CP decomposition is novel as far as I know, and interesting. The modifications proposed for the decomposed components to make it more practical, such as using SAT, or combining gating operators, are also contributing to the technical novelty of the paper. \n\n(*) On the image classification task, the performance of the proposed model is comparable to SOTA. \n    \nWeaknesses: \n\n(*) The paper proposes a novel layer and model that can be used for image classification, but it there is not enough discussion comparing it to recent ideas that go beyond ViT (e.g Swin-Transformer and ConvNextT models). It could be interesting to examine these models with your notation and perspective as well, since these achieve similar (or even better) efficiency and accuracy for the classification task.  \n\n(*) More experiments comparing the proposed model as a backbone for visual tasks might be useful for better evaluation against other models, such as experiments for detection and segmentation. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hZywRbFeuTn", "reviewer": "Reviewer_8HGv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors attempt to generalize linear layers with layers in which weights are produced by a function on inputs, motivated by the success of Transformers. Motivated by the CP decomposition, they extend the proposed layer to be more computationally efficient, and suggest a variant of a convolutional neural network architecture. They validate that the proposed architecture works on par with common approaches in image classification and object detection benchmarks.", "summary_of_the_review": "While the attempt to generalize linear layers and connect with Transformers is valuable, it seems like the resulting architecture is simply another variation of convolutional neural networks. A stronger and more thorough experimental validation is needed to validate the claims and novelty.", "strength_and_weaknesses": "- the proposed generalization of a linear layer provides a simple novel view on convolutional and Transformer networks\n\nExperimental validation is lacking.\n- while the authors propose a generalization of linear layers and Transformers, the evaluation is done only on computer vision benchmarks. Applying the proposed approach to non-computer vision tasks would provide stronger evidence for the claims\n- in the puzzle reconstruction experiment, it is not clear what information is provided by the Convolutional, Spatial mix, Pooling and Channel mix curves. As the task requires global receptive field, it is expected that the layers without it would be able to the reconstruction task. A convolutional network with more layers should be able to handle the task as well. A comparison with linear layer only should be sufficient to validate the claims.\n- in object detection, 1x schedule is not sufficient for experimental validation, as some networks might need more iterations to reach peak performance.\n- citations [1] and [2] are missing. These can also serve as stronger baselines for the experiments.\n- it is not clear is the results in Table 1 are statistically significant, or are within std of training with different random seed.\n\n[1] Wang et al., Non-local Neural Networks\n[2] Hu et al., Squeeze-and-Excitation Networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G1-9GrfEm6", "reviewer": "Reviewer_P57x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper first delivers a general five-dimensional tensor operator that can generalize many batched single-input layers that our community is frequently using. Then, the authors decompose the tensor with CP-decomposition; using this low-rank approximation of the tensor, they propose an architecture Chop'D former. Another low-rank approximation of self-attention is used (SAT) to lower the complexity. The architecture's performance was on par or better on the benchmarks like COCO detection/segmentation and ImageNet classification for small parameter models (~30M)", "summary_of_the_review": "It would be nice if some of my questions (see C3-C5) were answered. I want to give this paper a borderline accept recommendation because I believe the contributions that this simple and universal notation will bring to the community would outweigh the limited novelty.", "strength_and_weaknesses": "- (C1) It was a pleasure to read a paper trying to minimize mathematical obfuscation by employing concise einstein notation.\n- (C2) I agree with the claim that the five-dimensional tensor W in the paper is general enough to represent all trending models of the community as described.\n- (C3) A study for how low-rank r affects the performance of ChoP'D former is still needed. I even failed to find what r the authors used for the experiments in the main manuscript and the supplement material.\n- (C4) The tensor W is only for a single-input layer. I am curious how the multiple-input layer, like cross-attention layers, could be formulated with Einstien-tensor notation. Plus, if possible, how can they be CP-decomposed (and its effectiveness too.) Maybe converting the original transformer architecture (with a transformer decoder) for machine translation would be a great starting point.\n- (C5) Performances for major downstream tasks: COCO detection/segmentation and ImageNet classification are notably good. However, there is a report [1] that approximation methods tend to perform better on the small-parameter regime and worse on large-parameter models. Thus I want to see how Chop’D former works on at least a BERT-base scale (12 blocks).\n- (C6) Typo: wrong double quote character on page 2.\n\n[1] Tay, Yi, et al. \"Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?.\" *arXiv preprint arXiv:2207.10551* (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9trBWPs7HD", "reviewer": "Reviewer_a5bE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed ChoP’D Formers based on CP decomposition representation. Some modification is implemented on the proposed \"Former\". And experiments show some comparable results.", "summary_of_the_review": "The paper proposed a newly designed model based on CP decomposition. However, I still have some concerns as mentioned above. Therefore, I tend to give \"rejection\".", "strength_and_weaknesses": "Pros:\n\n- Have a try to unify nowadays models.\n\nCons:\n\nThis paper seems to be organized by (1)firstly unifying various networks; (2) then denoting networks with a CP format; (3) lastly designing ChoP’D to achieve a good performance on benchmarks. However, I have some concerns:\n  - The unifying strategy is not strict and not novel. The paper claims to unify convolutions, fully-connected layers, and transformers into a category \"Former\". \n    - The Transformers are not suitable for this category since they have non-linear structures (namely self-attention that has softmax function). In addition, this paper claims \"Without lack of generality, we omit at this stage the Layernorm (LN) applied before every block and the residual connections\". As the softmax and LN functions are both important components in a network, omitting them is not suitable. Therefore, it is unreasonable using such loose conditions to group Transformer and CNNs into one category.\n    - For the CNNs and fully-connected layers, [1] has already unified them. And [2] also gives tensorial CNNs a unified representation to initialize them in one scheme. Therefore, the idea of unifying CNNs is not novel.\n  - Unifying models into a CP format is already proposed by [3]. Equation (5) in this paper is the same as Equation (2) in [3]. Thus, such CP representation is also not novel.\n  - For the modification of the Former to construct ChoP’D Formers, there seems some limited novelty.\n\n- Unreadable writing:\n  - This paper uses improper uppercase word formats like \"Dynamic and Spatially Adaptive\". Simply using them in lowercase is ok. For example, \"Dynamic and Spatially Adaptive\" -> \"dynamic and spatially adaptive\";\n  - Grammar errors like \"Vision Transformers (ViT) success has long ...\" -> \"The success of Vision Transformers (ViT) has long...\";\n  - Overlong paragraphs:\n    - \"Puzzle Reconstruction\" on Page 7;\n    - \"Classification, Segmentation, Detection\" on Page 8.\n  - Inconsistent descriptions like \"fully-connected\" and \"Fully Connected\";\n  - The quotation mark in Latex is written as `` '';\n  - The citation should use citep and citet.\n\n- For the performance, the proposed ChoP’D Formers seem only derives some comparable results to baselines, which are not significant. So, what are the remarkable advantages of ChoP’D?\n\n\nOverall, regarding the above points, I vote for \"rejection\".\n\n[1] Kohei, Hayashi, et al. \"Exploring unexplored tensor network decompositions for convolutional neural networks.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[2] Yu, Pan, et al. \"A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Jean, Kossaifi, et al. \"Factorized higher-order cnns with an application to spatio-temporal emotion estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "54mT-rkyBI", "reviewer": "Reviewer_cu3r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The purpose of the paper is to propose a more general architecture than transformers. The proposed architecture is adaptive according to the different instance axis, spatial dimension and channels. In order to limit the complexity of such an approach the paper adopts a decomposition of the different operations. The paper evaluates in image classification, segmentation and detection.", "summary_of_the_review": "The idea of the paper is interesting but it lacks a lot of analysis to show that this approach is competitive with well established general architectures like transformers. Indeed, there is no experiment in the paper that shows the advantage of the proposed architecture over the transformers on different vision tasks like segmentation, detection and image classification. There is no experience that shows that the proposed architecture generalizes to other domains like NLP tasks and that the models scale well.\n", "strength_and_weaknesses": "Strengths:\n- The subject treated in the paper is interesting as well as the different operations used in the decomposition to have dunamic layer on the different axis.\n- The paper is quite well written and easy to follow.\n\nWeakness:\n\n- Hierarchical aspect: The decomposition of the dynamic block is interesting. However the hierarchical aspect of the architecture adds complexity and noise to the analysis. As we can see in Appendix Table 2 in order to design ChopForme we have to define the different pooling stage, the width and the depth of each. This complexify the architecture in comparison to vanilla transformers. It would be better to do an ablation of the architecture without the hierarchical aspect.\n\n- Tasks: \nThe paper proposes an architecture presented as general but there are only tasks of computer vision. It would be interesting to complete it with NLP tasks to see if the architecture is competitive with transformers.\n\n- Missing SOTA architecures: The architectures reported for comparison in the paper are not SOTA. In Table 1 appendix and Table 2 main paper. The following architecture should be added: EfficientNet-v2 [1], CoatNet [2], LeViT[3], MaxViT [4] The results of DeiT can be updated with those presented in the DeiT III paper [5].\n\n[1] Tan et al., EfficientNetV2: Smaller Models and Faster Training\n[2] Dai et al., CoAtNet: Marrying Convolution and Attention for All Data Sizes\n[3] Graham et al., LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference\n[4] Tu et al., MaxViT: Multi-Axis Vision Transformer\n[5] Touvron et al., DeiT III: Revenge of the ViT\n\n- Missing metrics: Several metrics are missing in the different tables to evaluate the tradeoffs of the proposed architecture as well as the ease of use. In addition to parameters and FLOPs, peak memory and speed should be reported.\n\n- Only small architectures:  The paper considered only relatively small architectures. Ablations are done with the tiny version and the largest model reported is smaller than ViT-B. General architectures like transformers generally scale very well so it would be interesting to add results with larger architectures to see if the method scales. The comparison Table 2 in the main paper are only with small architectures.\n\n- Overfitting evaluation: For the evaluation on ImageNet there is no measure of overfitting. But there is no separate validation set for ImageNet. It is necessary to add evaluation on ImageNet v2 in order to measure the level of overfitting of the proposed method. Especially since the more general an architecture is, the higher the risk of overfitting.\n\n- Segmentation & Detection: There is no large comparison in segmentation and detection of the proposed method with other approaches like Swin or Deit III. It is important to make this kind of comparison in order to determine if the architecture is competitive on different tasks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject