GmNet: Revisiting Gating Mechanisms From A Frequency View
Authors: Yifan Wang
Xu Ma
Yitian Zhang
Yizhou Wang
Zhongruo Wang
Sung-Cheol Kim
Vahid Mirjalili
Vidya Renganathan
Yun Fu
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient model
lightweight network design
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Lightweight neural networks, essential for on-device applications, often suffer from a low-frequency bias due to their constrained capacity and depth. 
This limits their ability to capture the fine-grained, high-frequency details (e.g., textures, edges) that are crucial for complex computer vision tasks. To address this fundamental limitation, we perform the first systematic analysis of gating mechanisms from a frequency perspective. 
Inspired by the convolution theorem, we show how the interplay between element-wise multiplication and non-linear activation functions within Gated Linear Units (GLUs) provides a powerful mechanism to selectively amplify high-frequency signals, thereby enriching the model's feature representations.
Based on these findings, we introduce the Gating Mechanism Network (GmNet), a simple yet highly effective architecture that incorporates our frequency-aware gating principles into a standard lightweight backbone. The efficacy of our approach is remarkable: without relying on complex training strategies or architectural search, GmNet achieves a new state-of-the-art for efficient models.

Published as a conference paper at ICLR 2026
GMNET: REVISITING GATING MECHANISMS FROM A FREQUENCY VIEW
Yifan Wang' Xu Ma' Yitian Zhang'
Zhongruo Wang² Sung-Cheol Kim²
Vahid Mirjalili Vidya Renganathan"
Yun Fu' FM-Global
Northeastern University UC Davis
[wang. (wang.yifan25, yifan25, ma.xul, shtiany @ @nootheaster.ed edu. yunfu@
ABSTRACT
Lightweight neural networks, essential for on-device applications, often suffer from a low-frequency bias due to their constrained capacity and depth. This lim- its their ability to capture the fine-grained, righ-frequeccy details (e.g. textures. edges) that are crucial for complex computer vision tasks. To address this funda- mental limitation, we perform the first systematic analysis of gating mechanisms from frequency perspective. Inspired by the convolution theorem, we show how the interplay between element-wise multiplication and non-linear activation func- tions within Gated Linear Units (GLUs) provides powerful mechanism to se- lectively amplify high-frequency signals, thereby enriching the model's 's feature representations. Based on these findings, introduce the Gating Mechanism Network (GmNet), a simple yet highly effective architecture that incorporates our frequency-aware gating principles into standard lightweight backbone. The effi- cacy of our approach is remarkable: without relying on complex training strategies or architectural search, GmNet achieves a new of for efficient models. 19eee
Our code can be found in
INTRODUCTION
Designing neural networks that are both highly accurate and computationally efficient is a central challenge in modern vision task. Lightweight models are essential for on-device applications, but their reduced capacity often limits their ability to capture the fine-grained details necessary for com- plex recognition tasks. A growing body of research suggests this limitation stems from spectral bias, where standard neural network architectures preferentially learn simple, low-frequency global patterns while struggling to capture high- frequency information corresponding to textures and edges Rahaman et al. (2019); Tancik et al. (020). This fundamental performance gap motivates the ex- ploration of architectural innovations that improve model's 's representational power without sacrificing efficiency. This bias is particularly pronounced in efficient models whose limited capac- ity hinders their ability learn complex, high-frequency information. This limitation motivates analysis of Gated Linear Units (GL.ss) a computationally inexpensive mechanism already proven effective various perforaerfe models De al. (2024); [iu al. (202I); Gu & Dao (2023). While their success is often attributed to adaptive information control, their impact on a network's spectral properties remains largely unexplored. We hypothesize that the element-wise multiplication at the core of GLUs, which corresponds to convolution in the frequency domain, provides direct mechanism to modulate this spectral bias and enrich model's high-frequency learning.
To build intuition, we present an example that visually illustrates how GLUs alter network's 's re- sponse to different frequency components of an image as shown in Fig. [ We take standard convolution-base lightweight building block (the top one) and create a variant by incorporating our proposed gating unit ( the bottom one). We first provide an input image decomposed into dif- ferent frequency components from low to high. The visualizations show that the baseline model primarily performs accurate on the oowffrquency information, struggling capturing crucial textural the raw image. In sharp contrast, the model
details which leads to an incorrect classification with GLU demonstrates a more balanced spectral response, effectively learning from both low and figh-frequency components form richer representation. This simple experiment provides initial

---

Published as a conference paper at ICLR 2026
illustration that gating mechanisms directly counteract the low- wweuureuency bias in many efficient architectures.
The mechanism enabling this enhanced spectral response is rooted in the convolution theorem: element-wii multiplication in the spatial domain equivalent to convolution in the frequency do- main. This operation allows the network to create complex interactions between different frequency bands, enriching the feature hierarchy. How- ever, as prior work has noted |Wang et (2020); Yin et al. 2019), naively amplifying high frequencies can make model overly sen- sitive to noise. The key, therefore, is selec- tive modulation. We contend that Gated Lin- ear Units, by pairing the multiplication with data-dependent gate and a non-linear activation function, provide exactly this control. They al- low the model to learn when to integrate high- frequency details and how much to trust them, effectively amplifying useful signals while re- maining robust to high-frequency noise.
X N
Ratee Imagge
Stm
Con Layers
Con Layer
Clactifter
Tiger cat!
Fregueiiy
Tiger Gat
Layor
Conv Layer
Clasifter
Tiger
Tiger
To put these principles into practice, we intro- duce the Gating Mechanism Network (GmNet), a lightweight architecture designed to leverage the spectral advantages of gating. By effec- tively capturing information across the full fre- quency spectrum, GmNet demonstrates that mottivatt design can lead to sub- stantial practical gains. Compared to the ex- isting methods Ma et al.) (2024a b) which also involve gating designs, our design leverages self-reinforcing gating mechanism in which the modulation and gating signals are derived from shared representation. This alignment ensures that salient variations, particularly those associ- ated with high-frequency components, are con- sistently emphasized rather than suppressed. In contrast, methods based on independent pro- jections often act as generic filters, leading to weaker sensitivity subtle variations that are critical for classification. Consequently, our approach is inherently more effective in preserving and enhancing high-frequency information. The results are compelling: without relying on advanced training techniques, our GmNet-S3 model achieves 81.3% top-1 I accuracy on ImageNet-IK. This surpasses EfficientFormer-LI by significant 4.0% margin while simultaneously being 4x faster on an A100 GPU, showcasing new state-of-the-art in efficient network design.
An illustration of how GLUs affect neural networks in classifying different frequency parts of an image. Starting with a raw image of a 'Tiger cat', we break it down into different frequency bands. The lowest frequency shows a recognizable outline, the higher frequency retains the general shape of the cat, but the highest frequency is almost un- recognizable. Predictions of different components are given in the left of different models. This ex- ample demonstrates two points: Although low- frequency decomposed images closely resemble the originals, accurate recognition of it does not guarantee accurate recognition of the original im- ages, and 2. GLUs improve the NNs' ability to learn higher frequency components effectively.
Figure 1:
means activation function.
We summarize the key contributions of this work as follows: (1) We provide the first systematic analysis of Gated Linear Units (GLUs) from frequency perspective, establishing clear link be- tween their core operations and their ability to modulate a network's spectral response. (2) We demonstrate that this spectral modulation can directly counteract the inherent low-frequency bias in many lightweight architectures, enabling them to learn more balanced and detailed feature represen- tations from both low and high frequencies.
(3) Based on these insights, we introduce the Gating Mechanism Network (GmNet), simple yet powerful lightweight architecture that achieves new state-of-te the-art in performance and efficiency, validating the practical benefits of our frequency- based design principles.
2 RELATED WORK
Gated Linear Units. The Gated Linear Unit (GLUyDauphi)) Dauphin et al. (2017), and its modern variants like SwiGLU Shazeer 2020), have become integral components in state-of-the-art deep learning
2

---

Published as a conference paper at ICLR 2026
models. Originally developed for sequence processing, their ability selectively control informa- tion flow with minimal computational overhead has led to widespread adoption. In Natural Lan- guage Processing, they central to powerful Transformers such as Llama3 Dubey al. (2024) and state-space models like Mamba Gu & Dao (2023), where they are lauded for improving training dynamics. This empirical success has spurred their integration into computer vision architectures; models like gMLP et al. (2021) have shown that replacing self-attention with simple gating-MLP blocks can yield competitive performance. However, the prevailing understanding of these mecha- nisms remains largely functional- they are viewed as adaptive 'information gates.' A critical gap exists in the analysis of their impact on network's fundamental learning properties. Specifically, no prior work has systematically analyzed GLUs from frequency perspective or connected their operational mechanism the well-documented problem of low-frequency bias in vision models.
Frequency Learning, Analyzing neural networks from frequency perspective has revealed fundamental learning dynamic known as spectral bias: networks of various types consistently learn simple, low-frequency patterns much faster than complex, high-frequency details Rahaman et al. (2019); Yin et al.] (2019); Tancik et al. (2020). While initially explored in regression tasks, this bias presents significant bottleneck for image classification, particularly in lightweight models. Due to their constrained capacity, these models struggle to capture the high-frequency information corresponding to textures and edges, limiting their overall performance. Furthermore, the of high-frequency components involves a delicate trade-off; while they are critical for accuracy, they can also make models more susceptible to high-frequency noise, impacting robustness Wang et al. (2020). Crucially, while prior work has adeptly characterized these phenomena, it has largely fo- cused on analysis and diagnosis. A clear gap remains in proposing and studying specific, efficient architectural mechanisms that can actively manage this accuracy- accuracy-robustness trade-off and explicitly counteract spectral bias within a model's design.
Lightweight Networks. The design of lightweight networks has predominantly followed two streams: pure convolution-based architectures like MobileOne Vasu et al. (2023b) and Rep Wang et (2024), and hybrid approaches incorporating self-attention, such as HfficientFormer22)) et al. (2022). While these lines of work have successfully pushed the frontiers of computational efficiency, they are built upon operations that are now understood to have a strong intrinsic low-frequency bias Tang et al. 2022); Bai et al. (2022). This foundational bias is of- ten exacerbated in the lightweight regime; the aggres- sive optimization for fewer parameters and lower FLOPs further restricts a model's capacity to learn essential high-frequency information. Consequently, the current paradigm for efficient network design contains a signif- icant blind spot: has optimized for computational met- rics while largely overlooking the spectral fidelity of the learned representations. This leaves a clear opening for new design principles that explicitly aim to correct this low-frequency bias from the ground up.
Conv
Conv
Res18
Res18-Ew
Res188 Gate
Figure 2: Block design of different vari- ants of ResNet18 where represents the element wise product and a means the activation function.
REVISITING GATING MECHANISMS FROM A FREQUENCY VIEW
We begin by defining the components associated with different frequency bands and outlining the details of our experimental setup. With decomposing the raw data z into the high-frequengy part Z and the low-frequency part z; where z = Zi, + Zj. We partition the spectrum into two complementary components based on predefined cutoff radius Specifically, the frequency representation z is decomposed into a low-frequency part 2 and a high-frequency part 2h, satisfying
2 21 Zh. (z,, 2h) = (2;r),
(I)
where ((:ir) represents a radial masking operator that separates spectral coefficients according to their distance from the origin in the frequency plane. In this formulation, coefficients located within radius r are assigned to 21, while those outside the radius are assigned to 2/,- This decomposition enables to analyze model behaviors under controlled spectral partitions.
We select three vision backbones including ResNet- 18 He et al. (2016), MobileNetv2 Sandler et al. (018) and EfficientFormrr-22 Li et al. (2023) representations to demonstrate the drawbacks of
3

---

Published as a conference paper at ICLR 2026
View in color:
Res18
Res18-Ewp
Reslè-Gate-GELU
Res18-Gate-ReLUe
Perfommance ReiNetl (1
10)
Peformmnnc on Re 10)
Accurney
139
0.35
Epoch
Figure 3: Comparison among Res18, Res18-Ewp. Res18-Gate-RLL.A and Res Res18-Gate-GELe The r represents the threshold of determining the boundary between low- wwwfrequency and nigg-frrqueecy. We plot the learning curves of Resnet18 and its variants for 100 epochs, together plotted with the accuracy of different frequency components z,. We set 10. All curves of are from the test set. The legends can be found in the top of the figure. We also provide more results with different r and different settings in the appendix.
the CNN networks and transformer-basee architectures capturing the high frequency informa- tion and how the gating mechanism improve the capability of learning high-frequene components. Modifications the network blocks of ResNet-18 are depicted Fig. 2 We evaluate the classifi- cation performance on different frequency components of the input images at each training epoch. Changes in accuracy over time provide insights into the learning dynamies within the frequency domain Wang et al. (2020). To avoid the occasionality, we calculated the average over three training runs.
3.1 EFFECT OF ELEMENT-WISE PRODUCT
Inspired by the convolution theorem, we first give an insight into why element-wist product can encourage NNs to learn on various frequency components from a frequency view. The convolution theorem states that for two functions u(x) and v<x) with Fourier transforms U and
(u v)) = 1- (U V),
(2)
where and * denote element-wing wise multiplication and convolution respectively, and the Fourier transform operator defined as F[) = F(w) = f f(t)e-jwt dt. This indicates that element- wise multiplication in the spatial domain corresponds to convolution in the frequency domain.
To its implication more clearly, consider the simplest situation: the self-convolution of func- tion. If the support set of F(w) is [- 22). then the support set of F+J()) will expand to [-20, 202). In other words, self-convolution broadens the frequency spectrum. With this enriched frequency content, neural networks have more opportunities to capture and learn from both high-frequency and low-frequency components.
3.2 How ACTIVATION FUNCTION WORKS?
We begin by analyzing how an activation function's smoothness influences the frequency charac- teristics of the features it produces. There is well-established principle in Fourier analysis that connects a function's smoothness the decay rate of its Fourier transform' 's magnitude. For a func- tion f(t) that is sufficiently smooth (i.e., its n-th derivative (()(() exists and is continuous), the magnitude of its Fourier transform, F(W)), is bounded and decays at a rate proportional to 1/|w]" for large w. This direct consequence of the differentiation property of the Fourier transform:
F()()(()) = F(w)"
This property implies that the smoother a function is (i.e., the more continuous derivatives has), the more rapidly its tigh-frequency components decay.
Conversely, functions with discontinuities or sharp "corners" where derivatives are undefined (such as the kink in ReIUUllkke activations) are known possess significant high-frequency energy. These
4

---

Published as a conference paper at ICLR 2026
View colo:
MobileNeev2
Using Gite-GELL
Using Gsle-Rel.U6
View shape:
Acc of -ec of -.. of
on MohileNav2 16)
Perfermence m. MeMileNas2
Perfontaace Mokileeee
Accuracy
Spect
Epoch
Epoch
Figure 4: Comparison among different variants of MobileNetv2. Different architectures respond differently to specific frequency component. To ensure an informative comparison, we select repre- sentative frequency thresholds tailored to each model where we 10. Additional results under other threshold configurations and other based models are included in the supplementary material.
sharp features require broad spectrum of fegh-fregeency sinusoids to be accurately represented. This leads Fourier transform that decays much more slowly. For example, a function with simple discontinuity will have spectrum that decays only at rate of 1/|w|- Therefore, we hypothesize that non-smooth activation functions will encourage the network to retain and utilize more high-frequenee information compared to their smooth counterparts like GELU and Swish, which is infinitely differentiable.
To validate this hypothesis, we conduct an experiment to compare the frequency learning of rep- resentative smooth activation (GELU) against a non-smooth one (ReLU6) within a ResNet18 ar- chitecture. As shown in Fig. E] the model using the non-smooth ReLU6 activation consistently outperforms the GELU variant in learning from high-frequecy components across different thresh- olds. This result supports two activation types. The superior performance of ReI.U6 on high-frequency data suggests that the slow spectral decay associated with non-smooth functions can be beneficial for tasks requir- ing fine-grained detail. Conversely, the GELU variant shows stronger relative performance on cow-frequency components, indicating its suitability for capturing broad, structural patterns. While a more exhaustive study is needed, this experiment provides clear evidence for the link between activation smoothness and a model's spectral learning preferences.
hypothesis, illustrating clear practical difference between these
4
GATING MECHANISM NETWORK (GMNET)
4.1 RETHINKING CURRENT LIGHTWEIGHT MODEL ARCHITECTURES FROM A FREQUENCY PERSPECTIVE
Before introducing our proposed network, we first inves- tigate the importance of capturing high-frequency infor- mation in efficient architectures by modifying existing ef- ficient models incorporate Gated Linear Units (GLUs). Specifically, we select one representative architecture: the pure CNN-based MobileNetV2 Sandler al. (2018). We replace the activation functions in their MLP blocks with simple GLU. Detailed architectural modifications are provided in the appendix.
stem
3x3 Conv Ise3 unillee
Downsample
ReLU6
+ Block
Conv
Downsample
stridem2
Block
DW-CCmv
Downsample
As shown in Fig. 4 we present the testing accuracy curves under the frequency threshold r = 10. Our re- sults demonstrate that integrating GLUs improves clas- sification accuracy on high-frequency components. No- tably, this improvement in high- fregurryy classification also correlates with a gain in overall performance. Fur- thermore, we observe that using GELU as the activation function within the GLUs enhances performance on low- frequency components, though it has a relatively minor effect overall accuracy. These findings suggest that effectively modeling high-frequency information is more crucial for improving the performance of lightweight neu-
Linear
Block
Downsample
ReLUU
Linear
Block
DWW-Covv Ko- stride))
Pool PC
Figure 5: GmNet architecture. Gm- Net adopts a traditional hybrid archi- tecture, utilizing convolutional layers to down-sample the resolution and double the number of channels at each stage.
5

---

Published as a conference paper at ICLR 2026
ral networks. It underscores the critical role of frequency-aware design in the lightweight networks. Moreover, we also conduct similar experiments on the transformer-beet -bsed model EfficientFormer Li et al. 2023) which can be found in the appendix.
4.2
ARCHITECTURE OF GMNET
To address the limitation of low-frequee lowffrequency bias for current lightweight network designs, our pro- posed method named as GmNet integrates a simple gated linear unit into the block as illustrated in Fig. GmNet offers both conceptual and practical advantages encouraging the model to learn from broader range of frequency regions, especially the high-frequency domain.
GmNets employ an extremely streamlined model architecture, carefully designed minimize both parameter count and computational speed, making them particularly suitable for deployment in resource-constrained environments. We incorporate two depth-wise convolution layers with ker- nel sizes of 7 x at the beginning and end of the block respectively to facilitate the integration of low- and high-free gegh-freuencc information. At the core of the block, we have two x convolution layers and simple gated linear unit. We use the ReLU6 as the activation function.
GmNet simplified GLU structure for reasons: (1) keep the model lightweight as possible, reducing computational load; and (2) ensuring that high-frequency signals can better enhanced without adding any additional convolutional or fully connected layers within the GLU. Furthermore, our gate unit is more interpretable, aligning with our analysis of GLUs in the frequency domain. Experimental results and ablation studies consistently demonstrate the superiority of our model, validating its design in accordance with our GLU frequency domain studies. We also show that the simplest structure achieves the optimal trade-off between efficiency and effectiveness.
EXPERIMENTS
In this section, we provide extensive experiments to show the superiority of our model and ample ablation studies to demonstrate the effectiveness of components of our method.
5.1 RESULTS IN IMAGE CLASSIFICATION
Experimental Setup. We evaluate our models on the ImageNet-1K benchmark using
input resolution of 224 224 for both training and inference. To construct dif- ferent GmNet variants, we vary the stage depth, embedding width, and expansion ratio while preserving the overall archi- tectural design. Detailed configurations of each model scale are provided in the appendix. All networks are optimized from random initialization for 300 epochs using the Adam W optimizer with ini- tial learning rate of x 10-3_ The global batch size is set to 2048. Additional train- ing hyperparameters. including data aug- mentation and regularization settings, summarized in the supplementary mate- rial. For latency evaluation, we export the trained Torch models to ONNX format and measure inference time on both an NVIDIA A100 GPU and a mobile device (iPhone 14). To ensure fair deployment comparison, the mobile benchmarks are conducted via CoreML conversion. Im- portantly, re- parameterization, distil- lation, or architecture search techniques applied; all reported results correspond standard end-to-end training.
Table 1: Comparison of Efficient Models on ImageNet- 1k. Latency is evaluated across various platforms.
Top-1
Latency
GPU Mobile
FasterNei-T
2053
71.9
72.0
ShumeV2.11
72.6
73.7
May ComeS
73.8
2.2
0.9
SuzNe: Ma ctal
20223
748
1,9
0.9
LSNa-11 Wing dr
749
11.4
AL3
2.9
1.8
GinNei-
75.5
3.7
2U6
1.6
1.00
EllcicmMMod-a Mactall
20144
76.01
4.7
SU.
2.3
18.2
Fasteret- Ti Chel (UVA)
76.2
7.6
0.9
2.5
1.0
Emiceeti mij 2022
73.2
12.3
1.3
12.1
1.4
SurNes-S Ma p32441
77.3
5.8
017
2.3
1.1
MobileOne / Viso 21 B.350
77.4
7.5
1.3
1.9
1.0
77.4
3.0
1.1
EthiceetForme7 GinNei52
2013
Eiticeeuaad
75.3
22.7
SunNee-S4
78.4
Switifumne
RepVIT-M
Wine c0
78.6
UniRapt KKNRes Diing
78.6
CinNeisS3
79.3
1:2
RepXIT-MI. Wine
2034
79.4
1.2
MobileOne S1ap EL AU043)
79.6
14.8
2.9
2.9
1.8
FasTVITS11 Virrer RMAAT
79.8
1.8
3.3
1.6
MobileNay Sermtee Ferereall Hirerrl
2024
79.9
1.0
9.2
1.4
LSN Wilingg et
80.3
23.2
13
6.2
3.6
Emicleu www.erz.......... 3.
21123
90.4
12.7
13
5.4
1.6
EflcienMad- ManddMMaar el
31740
$1.01
12.9
1.4
4.5
35.3
$1.2
14.0
233
6.4
1.7
LAVT-250 25T
81.5
18.9
6.7
31.4
GmnNel

---

Published as a conference paper at ICLR 2026
Compared with the state-of-the-art. The experimental results are presented in Table [ With- out any strong training strategy, GmNet delivers impressive performance compared to many state-of-the-art lightweight models. With comparable latency GPU, GmNet-S1 outper- forms MobileV2-1.0 by 3.5%. Notably, GmNet-S2 achieves 78.3% with only 1.9ms on the A100 which is a remarkable achievements for the models under 1G FLOPS. GmNet-S3 out- performs RepViT-M1.0 and StarNet-S4 by 1.9% and 0.9% in top-1 accuracy with 1.1 ms and 1.4 ms faster on the GPU latency, respectively. The im- provements on the speed are over 30%. Additionally, with similar latency, GmNet-S3 delivers 1. 1.7% improve- ment on the accuracy over MobileOne-S4. GmNet-S4 achieves 2x faster compared Rep V on the GPU and it surpasses MobileOne-S4 of 2.1% 1% under the simi- lar latencies of both GPU and Mobile. LeViT-256 [Gra-
@mhitt
42
00-
Mibilonne
UniReng UnfeengWlee
Starket 34
ham et al. ( matches the accuracy of GmNet-S4 but runs twice as slow on a GPU and 16 times slower on an iPhone 14 The strong performance of GmNet can be
MobileDree 52
Hep Mep-
Fastegreti
Ghostyer13
largely attributed to the clear insights of gating mecha- nisms and simplest architectures. Fig. [6) further illus- trates the latency-
75
Staffiee-52
LSkiet T
Mobilel2xx1.2
orfee-off across different mod- els. GmNet variants achieve substantially lower latency compared to related works, while maintaining competi- tive or superior Top-1 accuracy. More comparisons and results can be found in supplementary.
74-
wrV2-50 Efficenn Eficeentttt-as
Chougatt.
1335
2:0
2.55
2.50
2:75
3,00
3.25
Latency (ms)
Figure 6: Trade-off between Top-1 ac- curacy and latency on A100.
5.2 ABLATION STUDIES
More studies on different activation functions. To further explore the effect of different activation functions, we trained various GmNet-S3 variants ImageNet-Ik. As illustrated in Fig. 5 we replaced ReLU6 with GELU, ReLU or remove the activation function. To better reflect the differences between different models, we set the radii to larger range/ As shown in the Table. 2 we can find that, the increases on classifying the high-frequency components are sig- nificant comparing models using and not using the activation functions. For example, comparing results of "Identity' and 'ReLU' with the im- provement of 11% on the raw data, improvement high-frequencies is over 3 times on average. 'GELU"
Table 2: The accuracies of classifying the raw data and their components under different activation functions ImageNet-1k, We gradually increase the radii by step of 12. This result is the average of five testings.
Activation
Identity
ReLU
GELU
ReLUU6
Raw data
70.5
78.3
78.4
79.3
Frequency
Low
High
Low
High
Low
High
Low
High
and 'ReLU' shows advances on low- /high- frequency components respec- tively compared to each other. This aligns with our understanding of how different types of activation functions impact frequency response. Notably, the closer performance of models with Identity and ReLU/GELU at low frequencies suggests the low-frequency bias of convolution- networks.
r 12
9.79
12.6
12.0
45.9
12.7
41.5
14.8
51.7
r 24
38.1
13.5
40.0
41.6
12.1
36
52.9
0.7
56.2
4.9
58.7
3.9
55.2
4.7
r= 48
63.2
0.5
64.5
2.3
66.1
2.1
64.4
2.5
r= 60
66.6
0.9
69.4
1.0
70.7
71.1
1.4
Moreover, even considering the improvements on the raw data, model using the ReLU6 shows obvious increase on the eigh-frequency components compared to the model using GELU es- pecially when we set to 12, 24, 36. Compared to the model with ReLU, ReLU6 is more effective in preventing overfitting to high-frequency components since it has better performance on low-frequencies. Considering performances of ReLU, GELU, and ReLU6, can observe that achieving better performance on high frequencies at the expense of lower frequencies does not necessarily lead overall improvement, and vice versa. To get better performance on the raw data, it is essential to enhance the model's 's ability to learn various frequency signals.
Comparison with existing methods from the frequency perspective. As addressed in Table 2 model should achieves strong performance across different frequency components deliver
7

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of different GLU designs for GmNet-S3 on ImageNet-IK. Here, LN, DW, and Pool represent layer normalization, depth-wise convolution with a kernel size of 3, and average pooling with a 3x3 window, respectively. We underline all notable scores in classifying the different frequency decompositions. Considering gaps of overall performances, an improvement which is remarkable should exceed 1.0. This result is the average of five testings. We also provide more variants of GLUs in the supplementary materials.
GLUs
Top-1 (%6)
Purams (M)
GPU
12 High
24
36
48
60
(ms)
Low
Low
High
Low
High
Low
High
Low
High
(ar)
LN(c)
78.9
7.8
2.9
12.1
47.6
41.6
10.9
56.4
5.2
64.7
2.4
69.8
1.2
(() DW(z)
79.0
8.0
2.4
12.3
49.0
42.7
9.6
58.1
4.6
65.7
2.3
71.2
(()) Poo)(2:)
78.6
7.8
14.2
50.1
42.3
10.8
55.8
4.9
63.8
2.7
69.9
1.3
(ar) FC(x)
79.2
20.2
3.6
10.8
51.4
39.6
8.7
52.6
4.4
62.9
3.4
69.9
2.4
(x)
79.3
7.8
2.1
14.8
51.2
41.6
12.1
55.2
4.7
64.4
2.5
71.1
better overall performance. However, both pure convolutional architectures and transformers ex- hibit a www-frequeccy bias, as discussed in Bai et al.] (2022); Tang et al.] (2022). Therefore, en- hancing the performance of lightweight model depends its ability to more effectively cap- ture high-frequency information. To address the advantages of GmNet on overcoming the low- frequency bias, test some existing models on different frequency components of different radii. We select three kinds of typi- cal lightweight methods for compari- son including pure conv-based model MobileOne-S2 Vasu et al. 2023b), attention- based model EfficientMod-
Table 3: Comparison with recent methods. We test models on the high~/low-freqeene components on the ImageNet- Ik. The highest values of each columns are highlighted.
xs Ma et al. (2024b) and model also employing Gl.Us-like structure
Methods
Top-1 (%)
High
=
12
Low
High
21
Low
High
c=
30
Low
MobileOne-S S2 Vasu 20233
77.4
35.0
11.5
6.5
36.9
2.4
53.5
StarNet-S4 Ma et al. (2024a). As
78.3
12.9
9.4
3.5
3 accuracies of low-frequency components are close among different models considering the overall performance. However, it shows that GmNet-S3 clearly surpass the other models in high frequency components. For example, GmNet-S3 has a 6.3% improvement compared EfficientMod-xs when 12 and 2.7% increase when 24. For StarNet, which also uses a GLU-like structure with dual-channel FC, it struggles to effectively emphasize gighfrequency signals. The simplest GLUs design can achieve better balance between the efficiency and the effectiveness.
shown in Table
StarNet- $4 Ma ara 20244
78.4
43.3
13.8
9.4
41.3
3.4
54.8
GmNet-S3
79.3
51.7
14.8
12.1
41.6
4.7
55.2
Study on designs of the GLU. In GmNet, the gated linear unit adopts the simplest design, which can be defined as (x) x. For comparison, we modify the GLU design and conduct experiments to test performance on raw data well as decompositions at different frequency levels. As shown in the Table 4 the simplest design achieve the best performance both on effectiveness and efficiency for the overall performance. For the decomposed frequency components, we observe clear differences among various GLU designs. The GLU of (x)) x demonstrates significantly higher accuracy in classifying high-frequency components. For example, for 12 and 24. the GLU with (Ir) x shows an improvement of 4.1 over the LN design and 2.5 over the DW design. This indicates that the simplest GLU design is already effective at introducing reliable high-frequency components to enhance the model's ability to learn them. Designs aimed at smoothing information show a notable improvement in some low-frequency components. For instance, with similar overall performance, the GLUs using o (z)) DW(x) and (xx) LN(xx) achieve better results on low-frequency components when the radii are set to 24, 36, and 48. The model using a linear layer in GLUs offers performance comparable to GmNet-S3 and is adept at learning low-frequency features. However, its placement at high-dimensional stage problematic. This design choice leads to an excessive number of parameters and significant increase in latency. Moreover, depth-wise convolution more effective than layer normalization encouraging neural networks to learn from low-frequency components which is also more efficient. For the design with the average pooling, it does not perform better in classifying because pool(z) acts an overly aggressive high-pass filter, which does not retain the original high- gighffreueency signals in x well and instead introduces more frehfrry noise.
signals. This may be
8

---

Published as a conference paper at ICLR 2026
40
100
Ideatity GELU
Identiry
30
300
GELU
Number
ReL.U
Number
ReLU
ReLU6
20
200
10
0.97]
0.1562
0.2973
0,4384
0.8617
Bandwidths
Bandwidths
Figure 7: The histogram illustrates the distribution of bandwidths of convolution kernels. Band- widths represents the capability of convolution kernel for capturing various frequency informa- tion. We use weights of the convolution layer which under the GLU in the first block (left) and the last block (right) of the GmNet-S3. All modals are trained on the data of the ImageNet-Ik Ik. In general, the further the distribution shifts to the right, the stronger the convolutional kernel's 's ability capture signals of different frequencies.
Bandwidths analysis of convolution kernels. As discussed in the Tang et al. (2022), the convo- lution layer may play roles of 'smoothing' the feature which means has low- low-frequency bias. Experiments on studying weights of the convolution layer is insightful to give more evidences of how GLUs effect the learning of different frequency components Wang et al. (2020); Tang iet al. (2022); Bai et al. (2022). In this paper, we propose using the bandwidths of convolution kernels to represent their ability of responding to different frequency components. Specifically, a wider bandwidth indicates that the kernel can process broader range of frequencies, allowing capture diverse frequency components simultaneously and thereby preserve rich information from the feature. As illustrated in Figure 7 the distributions of the ReL.U model suggest that its convolution kernels tend focus on narrow range of frequency components leading to relatively lower bandwidths. It indirectly reflects an overemphasis the model using GELU exhibits better distribution in the top convolutional layers, still has distribution shift in the bottom convolutional layers. Compared to
high-frrquency components. Although
low-frequency bias, leading other activations, the enhanced bandwidth distribution of the model using ReLU6 demonstrates bet- ter generalization for this task. The properties of the convolution kernels align with results in Table
Contribution breakdown. We show the ablation study of different block designs and adjust the dimensions/number of blocks under strictly matched FLOPs, parameter count, and training settings, and re- port the mean std over three ran- dom seeds. We replace the 7x7 dw- conv with linear layer and replace the GLU with ReLU as the base- line block. The results are shown
Table 5: Contribution breakdown under matched FLOPs and parameters (mean + std).
Variant
Params (M)
FLOPs (G)
Top-1 Acc (%)
Baseline
7.82
1.24
71.5 : 0,2
+ 7x7 DWConv
7.82
1.28
78.1 + 0.1
+ Gate (Identity)
7.82
1.24
69.2 - 0.3
+ Gate (ReLU)
7.82
1.24
78.0 + 0.2
in Table 5 Overall, Table demon-
Gate (GELU)
7.82
1.24
77.9 + 0.1
strates that the performance gains in- troduced by gating mechanism distinct improvements beyond what
Gate (ReLU6)
7.82
1.24
78.5 - 0.1
ReLU6
7.82
1.24
77.9 + 0.1
Full GmNet
7.82
1.24
79.2 + 0.1
other components alone can offer.
CONCLUSION
This paper tackled the prevalent low-frequency bias in lightweight networks through a novel frequency-based analysis of gating mechanisms. We found that in a Gated Linear Unit (GLU), element-wise multiplication introduces valuable high-frequency information, while the paired acti- vation function provides crucial control to filter for useful signals over noise. Our resulting model, the Gating Mechanism Network (GmNet), validates this approach by setting art in efficient network design. This work demonstrates that a frequency- aware methodology is a promising path toward creating future models that both efficient and representationally robust.
of-th-

---

Published as a conference paper at ICLR 2026
7 STATEMENTS
7.1 ETHICS STATEMENT
In our paper, we strictly follow the ICLR ethical research standards and laws. To the best of our knowledge, our work abides by the General Ethical Principles.
7.2
REPRODUCIBILITY STATEMENT
We adhere to ICLR reproducibility standards and ensure the reproducibility of our work. All datasets we employed are publicly available. We will provide the code to reviewers and area chairs in the supplementary material.
REFERENCES
Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transformers by revisiting high-frequency components. In European Conference Computer Vision, pp. 1-18. Springer, 2022.
Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo, Song Wen, Chul-Ho Lee, and S-H Gary Chan. Run, don't t walk: chasing higher flops for faster neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12021-12031, 2023.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. International conference on machine learning, pp. 933-941. PMLR, 2017.
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Al- bert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402. 19427, 2024.
Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao, Lin Song, Xiangyu Yue, and Ying Shan. Unireplknet: A universal perception large-kernel convnet for audio video point cloud time-series and image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5513-5524, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Benjamin Graham, Alaacldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12259-12269, 2021.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0073 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang. and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35: 2112294412944 2022.
Yanyu Li. Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for mobilenet size and speed. Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 16889-16900, 2023.
10

---

Published as a conference paper at ICLR 2026
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mips. Advances in neural information processing systems, 34:9204 9215, 2021.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Xu Ma, Xiyang Dai, Yue Bai, Yizhou Wang, and Yun Fu. Rewrite the stars. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5694-5703, 2024a.
Xu Ma, Xiyang Dai, Jianwei Yang, Bin Xiao, Yinpeng Chen, Yun Fu, and Lu Yuan. Efficient modulation for vision networks. arXiv preprint arXiv:2403. 19963, 2024b.
Danfeng Qin, Chas Leichner, Manolis Delakis, Marco Fornoni, Shixin Luo, Fan Yang, Weijun Wang, Colby Banbury, Chengxi Ye, Berkin Akin, et al. Mobilenetv4: universal models for the mobile ecosystem. In European Conference on Computer Vision, pp. 78-96. Springer, 2024.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301-5310. PMLR, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiell Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 4510-4520, 2018
AbdeIrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Swiftformer: Efficient additive attention for transformer-base real-time mobile vision applications. In Proceedings of the IEEE/CVF international conference on com- puler vision, pp. 17425-17436, 2023.
Noam Shazeer. Glu variants improve transformer. arXiv preprint urXiv:2002.05202 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neurul information processing systems, 33:7537-7547, 2020.
Ling Tang, Wen Shen, Zhanpeng Zhou, Yuefeng Chen, and Quanshi Zhang. Defects of convolutional decoder networks in frequency representation. arXiv preprint arXiv:2210.09020. 2022.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5785-5795, 2023a.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mo- bileone: An improved one millisecond mobile backbone. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 7907-7917, 2023b.
Ao Wang, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Repvit: Revisiting mobile
cnn from vit perspective. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15909-15920, 2024.
Ao Wang, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Lsnet: See large, focus small. arXiv preprint ar 25032255.33333, 2025.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8684-8694 2020.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. fourier perspective model robustness in computer vision. Advances in Nearal Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 IMPLEMENTATION DETAILS
A.1.1 PSEUDO-CODES OF MODEL ARCHITECTURES
In modified ResNet18, featured in Fig. 2, adjust the activation function the different variants. As an example, we provide the pseudo-codes of Bes18-GtteRee in the Algorithm
Algorithm Pseudo-codes of Res 8-Gate-RReLU
def (lock(x, in planes, planes)
out Conv2d(x, in in_pnes,s, planes, 3, 1) out BatchNorm2d(x) out ReL.U(out) * out
out = Conv2d(out, planes, planes, 3, 1, out BatchNNom22(out)
out += self )
out Rel.t()ut) return out
For the proposed GmNet, featured in Fig. 4, we provide the pseudo-code of GmNet in the Algorithm 2 Also, for ease of reproduction, we include separate file in the supplementary materials dedicated to our GmNet.
Algorithm 2 Pseudo-codes of GmNet
def Block(x, dim, ratioo
input= x
DWConv2d(x, dim, dim, 7, group=dim)
BatchNorm2d(x)
Conv2d(x, dim, mlp_ratio& I)
X ReLUU6x))
Conv2d(x, mlp_ratio*dim, dim, X BatchNorm2d(x)
DWConv2d(x, dim, dim, 7, 3, group=dim)
x = input + drop.path(layer_ssalee())
return
A.1.2 TRAINING RECIPES
We first provide the detailed training settings variants of ResNet18 in Table
A. 1.3 GMNET VARIANTS
We provide the setting of variants of GmNet in Table,
Variant C1
depth
ratio
Params
FLOPs
S
40
2,
10,
3,3,3,
3.7 M
0.6
GmNet-S2
48
2,, 2,2,8.3.
3.3.3,2
6,2 M
0.9
Gm
48
3,3, 3.3
4,4.
7.8 M
1.2 G
GmNet-S4
68
3,3,
4,4,4,4
17.0
2.7 G
Table 6: Configurations of GmNet. We vary the embedding width, depth, and gating ratio to con- struct different model sizes of GmNet.
We also provide detailed training configures of GmNet in this section shown in the Table g
12

---

Published as a conference paper at ICLR 2026
config
value
image size
32
optimizer
SGD
base learning rate
0.1
weight decay
5e-4
optimizer momentum batch size
0.9
128
learning rate schedule training epochs
cosine decay 100
Table 7: Res18 variants
training setting
config image size
value
224
optimizer
AdamW W
base learning rate
3e 3
weight decay
0.03
optimizer momentum
B1, B2 = 0.9, 0.999
batch size
2048
learning rate schedule
cosine decay
warmup epochs
training epochs
300
AutoAugment
rand-m -mstd0.5innc
label smoothing
0.1
cutmix
0.4
color jitter
0.
drop path
O.(S1/S2), 0.02(S3/ S4)
Table 8: GmNet training setting
A.2 MORE GLUS DESIGNS.
In this section, present additional block designs to demonstrate the efficiency and effectiveness of our proposed architecture. As shown Table 9 we conducted experiments incorporating fully connected (FC) layers into the Gated Linear Units (GLUs). Adding extra FC layer to branch of the GLUs may slightly improve performance. However, this modification significantly increases the number of parameters and reduces the model's latency.
Moreover, this enhanced architecture does not perform better in classifying different frequency com- ponents. The reason is that features processed by FC layer cannot effectively emphasize various frequency components. While adding more parameters and employing different training methods might enhance the capability learn different frequency components, in lightweight models, the simplest GLU design often delivers better performance. This observation is consistent with findings from many recent studies on lightweight models.
A.3 MORE COMPARISONS OF THE MAIN RESULTS.
We provide more comparisons of the main results Fig. g We plot a larger latency-ace trade-off figure to include more methods including EdgeViT, MobileNetV2 and GhostNet.
A.4 RESULTS ON CUB-100.
We evaluated GmNet-SI on the CUB-100 1000-000do dataset. The results are competitive, as shown Table I0 Compared to ShuffleNet-V2 GmNet-S1 achieves better performance with smaller model size.
13

---

Published as a conference paper at ICLR 2026
GmNet
RepVIT-M1.1
EfficientFormerV2-S2
EfficientMod-s
MobileOne-S4
FastVIT-S12
Top-1 Accuracy (%)
Fasternet-T2
RepViT-M1.0
StarNet-S4
MobileOne-Si
EdgeViT-xs
StarNet-S3
RepViT-MO.9
Fasterpet-T1
GhostNet1.3.
StarNet-S2
EdgeViT-xxs
MobileV2-x1. Mobiee
Mobilev2-x2.0
fffcieentod-xs
EfficientFormer22--00
GhostNet1.0
EfficientMod-xxs
1.5
2.0
2.5
3.0
3.5
4.0
5.5
Latency (ms)
Figure 8: Top-1 accuracy vs Latency on A100. Our models have significantly smaller latency com- pared related works.
Table 9: Comparison of different GLU designs for GmNet-S3 on ImageNet-IK
Top-1
Parums
GPU
12
24
36
48
60
GLUs
(%)
(M)
(ms)
Low
High
Low
High
Low
High
Low
High
Low
High
a(x) FC(x)
79.2
20.2
3.6
10.8
51.4
39.6
8.7
52.6
4.4
62.9
3.4
69.9
2.4
a(FC(z)) a
79.6
20.2
9.4
48.9
35.0
9.1
51.1
3.6
62.1
68.7
2.4
a(z)
79.3
51.7
41.6
12.1
55.2
4.7
64.4
2.5
71.1
1.4
A.5 PERFORMANCE WITH SWIGLU AND SILU
We have conducted experiments with the SiLU variant CIFAR-10 to illustrate the learning dy- namics in Fig. 14. Following the reviewer's advice, we adapted both SwiGLU and SiLU to GmNet- S3 and trained the models under the same settings. The results in Table show that both activations improve performance compared to the baseline configuration, which removes activations inside the GLU. However, their gains remain consistently smaller than those achieved by our proposed design, confirming that the proposed GmNet block better aligned with the frequency characteristics and architectural constraints of lightweight models.
A.6 PERFORMANCE CHANGING RELU6 TO RELU.
Table 12 shows the performance change when replacing ReLU6 with ReLU across multiple GmNet variants. The degradation is consistent across all model sizes.
A.7 STUDIES ON ALIASING AND ROBUSTNESS
We train Res18-Gate-ReLL and Res18-Gate-GELU on CIFAR-10 following the setting of to
show how different activation functions affect robustness. We conducted adversarial robustness ablation under PGD attacks. As shown below. the ReLU-based wLi-which emphasizes higher- frequency components-exhibiee slightly lower PGD accuracy:
14

---

Published as a conference paper at ICLR 2026
Variant
Params (M)
Top-1 Ace (%)
ShuffleNet-V V2
3.5
76.0
GmNet-S1
3.1
81.5
Table 10: Results on CUB-100.
Model Activation
Top-1 I Acc (%)
Baseline (Identity)
70.5
SiLU
77.9
+ SwiGLU
77.2
Proposed (ReLU6-GLU)
79.3
Table 11: Results of using SiLU and SwiGLU in GmNet-S3.
Besl---Gate-eLLU: 45.7% PGD accuracy
Res18-Cate-GELAN 46.7% PGD accuracy
Clean accuracy remains similar for both variants (82.2% vs. 82.8%), but the ~1% drop in PGD robustness for the ReLUU gate is consistent with prior findings that stronger high-frequency reliance increases vulnerability to aliasing and adversarial perturbations.
A.8 EXPERIMENTS ON CONVNEXT
To show the generalization of gating structure, we adapted our GLU to ConyNeXt-Tiim Tiny and trained the model using the same setting GmNet ImageNet- 1k. As shown in Table B3 the performance without introducing any additional computational cost.
GLU improves
The improvement is smaller compared to GmNet, which may result from the fact that large CNNs less affected by indivattoo-inducce frequency bias. Block design should be architecture-dependen..
A.9 MORE RESULTS ON EIFORMERY2 AND MOBILENETV2.
MobileNet-V? V2
Eformer-v2 MLP
Ixl Conv BN
3x33 Conv
BN
1x1 Conv
ReLU
3x3 Conv
EN
1x1 Conv
Figure 9: The illustration of modifications of MobileNetV2 and EfficientFormer-V V2.
We first show the illustration of the modifications for both models in Moreover, plot the testing curves with different settings thresholds. As shown in Fig. [3] the overall performance trend is consistent with the charts and figures presented in the main text. The model using ReLJU6 shows
15

---

Published as a conference paper at ICLR 2026
Variant
ReLU6 ReLU
GmNet-S1
75.5 74.7
GmNet-S2
78.3 + 77.4
GmNet-S3
79.3 78.3
GmNet-S4
81.5 + 80.5
Table 12: Performance degradation when replacing ReLU6 with ReLU,
Model
Top-1 Acc (%)
Params (M)
FLOPs (G)
ConvNeXt
82.5
28.6
4,46
ConvNeXt GLU
83.0
28.6
4.46
Table 13: Comparison between ConyNeXt-Tiny and its GLU-enhanced variant.
better performance on the high-freque frgh-frequency components where the overall performance also surpasses other models. Meanwhile, the model using GELU performs better on wwwffreuunccie. The extra results demonstrate the effect of GLU on helping model learning various frequency information and improvements on high-freet freuerryyyncy information have more impact on the final performance.
A.10 QUANTITATIVE SPECTRAL EVIDENCES.
To further explore the effect of different activation functions, we provide the quantitative spectral evidence. Based the GmNet-S3 variants, computed multiple layers and model variants as shown in Table [4] Firstly, we extract the layers before/after the gate and compute their high/low-frequency energy ratios show the spectral changes. Also, to demonstrate how different activation functions affect the model's frequency response, we compute the gigh/low-frquencyy energy ratios of the first DW-Cony layers in each stages.
energy ratios across
We define the low frequencies as the central 1/4 region of the 2D spectrum. Table 14 shows that a consistent spectral pattern distinguishing smooth and non-smooth activations. For GELU, the transition from to g typically increases the low-frequency response, and across all stages GELU yields the lowest high/low ratios. This aligns with its smooth functional form, which naturally biases the network toward low-frequency representations. In contrast, ReLU and ReLU6 systematically amplify high-frequency components. the early stages (Stage and 1), ReLU exhibits the strongest high high-frequency response, reflecting its non-smooth activation behavior and its tendency to preserve or enhance sharp transitions. In deeper layers (Stage 2 and 3), ReLU6 produces the highest high/low ratios, suggesting that its clipped nonlinearity becomes more potteniaalt explaining why the ReLU6-based model obtains the stages, blocks, and models, and they
influential as depth best overall performance. These effects stable directly support our hypothesis: smooth activations such as GELU favor low-frequency features, whereas non-smooth activations (ReLU/ReLU6) amplify high-frequency content. This yields clear falsifiable prediction if smooth activation were ever to systematically exceed ReLU/ReLU6 in gigh-freueecc ratios under comparable settings, our hypothesis would be invalidated- strengthens the explanatory robustness of the spectral analysis presented in Sec. 3.2
View color:
Res18
DecID-Eww
BesLB-Gaate- GELU
ReslB-Gartt Rel
View
Aze
Ace
Accuracy
spach
Figure 10: Additional results under different threshold configurations of different variants of ResNet18.
16

---

Published as a conference paper at ICLR 2026
View color:
E6omerX22
Using Gate-GELU
Using Cate-ReLLU6
View in shape:
Ace
- of -.- Acc of
Aalamanccmm Miekkhen 3)
Pafesman 0LS. No
Accuracy
Spech
Opech
Figure 11: Additional results under different threshold configurations of different variants of Mo- bileNetv2.
View BRrmerV2
Gate-GiLL
Gate-ReLUe
View shape:
of
Acc of% -.- Ace of
Reformmn en of Elmmer22 fr
Porfrraas 2fEmman to-
Perfornvnce cot Lormnn??
Accuracy
Epoch
Epoch
ppoch
Perfermnne en Blomen (r
Perfomazee 5, of (r-
Epech
Epach
Figure 12: Additional results under different threshold configurations of different variants of Efficientformer-V2 V2.
Vee cader
2el nerv
Using Gate- BEUU
Using Bate ReLUS
Vie ia chope
- Aoo of
- of
Hobiletetv2
HobileHetu2
Sformerv2
#formenu2
spach
Figure 13: Additional results under different threshold configurations of different variants of Mo- bilcNetv2 and BfficientFormer-22 (Eformerv2).
17

---

Published as a conference paper at ICLR 2026
Table 14: High/Low Frequency Ratio Comparison. We computed the spectral changes from the layer before to the layer after the gate which are defined as f and g respectively. We also compute the high/low-freeee frequency energy ratios of the first 7x7 DW-Conv layers of each stage.
HightLan Frequency Ratio Changes befordofter the gate variants
Stage
Layer Pair
Rel.16 (H/).)
GELU (HI).)
Rel.U (Hr..)
[-g
0.1195
0.1200
0.5674
1.1
[-g g
0.0989
- 0.142
0.0423 - 0.0022
0.1422 - 0.1751
2.1
0.0386
0.0706
0.0252 0.0013
0.0018 0.0030
g
0.0019
0.0281
0.0106
HightLow Frequency Ratio Comparison at DW-Com layers
Stage
Layer
(HIL)
(HML)
ReLU6 (HVL)
0.1
1st DW-Couw
0.1203
1.1553
0,7057
Ist DW-Com
0.0995
0.3040
0.2452
1st DW-Coow
0,0381
0.0024
0.0697
3,1
1st DW-Coov
0,0057
0,9088
0.0142
A.11 EXTRA EXPERIMENTS ON GATING MECHANISMS.
In this section, we present additional experimental results demonstrating how different activation functions affect frequency learning. Firstly, we have conducted experiments with different settings of r of Fig. 3 in the main paper. We have set to (4. 7] respectively. The results are shown in Fig., the performances are aligned with our analysis in Sec. which indicates that the results shown in Fig. 33 are not accidental.
Specifically, we conduct experimearents using another smooth activation function, Swish (SiLU), and non-smooth function, ReLU6.
As shown in Figures 44 and 14b] ReLU6 performs similarly to GELU. Although ReLU6 may intro- duce more high-frequency components, it does not perform well as ReLU for two main reasons: (1) ReLUU caps the activation values. can reduce the model's sensitivity to high-frequency com- ponents where those components are often associated with higher activation values. (2) The use of low-resolution images can adversely affect the performance in classifying high-frequency compo- nents, as finer details are lost, making it harder for the model to learn these features. Figures 14c and present comparisons between SiLU (Swish) and ReLU, as well as between SiLU and GELU, respectively. The Res18-Gate-Sin the range r E (0,r)). This indicates that SiLU has a greater smoothing effect on the information, encouraging the model to learn more effectively from lower frequencies.
performs better on lower-frequenc components, specifically in
Moreover, we investigated the impact of different training strategies to understand their effects on model performance. Specifically, as plotted the Fig. [5| we replaced the optimizer with Adam, setting its learning rate to 0.001. While the training curves showed noticeable variations compared the baseline setup, the overall performance differences among the model variants remained con- sistent. This consistency indicates that the observed behaviors are robust to changes in optimization strategies.
We also conduct experiments on setting the redii to different sets. As shown in Figs. 16 and we set radii to [0, 4, 8, 12, +oo and [0, 8, 16, 24, +00 respectively. When the frequency intervals are too small, the differences between the methods become less pronounced, especially in the lower- frequency components. When the frequency intervals are too large, all models struggle to classify higher-frequent components. Although differences between models become more pronounced in the lower-frequent components, such as between GELU and ReLU activations, to better understand the training dynamics, it is necessary examine the differences in the freeurruy components as well. Therefore, we decide to display the results of setting radii to [0, 6. 12, 24, +oo] in the main body of the paper to have better understand of the training dynamic of different variants. These sults provide valuable insights into the functionality of the gating mechanisms. They suggest that the interaction between the element-wise product and the activation functions is general phenomenon.
A.12 DOWNSTREAM TASKS.
We further provide the results of downstream tasks of Object Detec- tion. Instacce Segmentation and Semantic Segmentation. Firstly. we con- ducted experiments on GmNet-S3 on MSCOCO 2017 with the Mask RCNN
18

---

Published as a conference paper at ICLR 2026
Mopp dhoug wne
(a) Comparison between Res Res18-Ca 1e--11---GGGLLLE GELU and Res Ge.-8-GGtt-ReLL.6.. /80.5
(b) Comparison between Res18 A---------ieee Gate--atteea ReLU and Res18 R-LLR.
Comparison between Res18- 1aat-------eeeee Gate-GELU and Res18-Gr S-Ga---GGeSSet Gate-SiLU.
(d) Comparison between Res Bas18-Gate-ReLe and Das18-Gate-SiLU.
Figure 14: Learning curves of Resnet18 and its variants for 100 epoch. The radii to [0, 6, 12, 18, +00}
Table 15: Object detection & Instance segmentation& Semantic seg-
framework for object detection and instance segmentation. Our method shows better performance compared to the existing methods RepViT-M1.5
mentation. The latency is tested on iPhone 14 by Core ML Tools.
Backhone
Latency (mss]
Ap
Object Detection
APt
APh
Apmest
Instance Segmentation
AP
AP
Samantic mloU
BriciendFommrr L3
124
41.4
63.9
44.7
38.1
61.0
40.4
43.5
RepVIT-MIL
69
416
63.2
45.3
38.9
60.5
41.5
43.6
GmNNet-3
52
422
63.4
46.7
40.1
61,2
42.9
44,6
Wang et al.] (2024) and ElficientFormer-L3 Li al.
of latency, Aphor and Apmask Specifically, GmNet-S3 outperforms RepViT-M1.5 significantly by Meanwhile, GmNet-S3 has 1.7 ms faster on the Mobile latency and more than 2 times faster than EfficientFormer-L3.. For the semantic segmentation, we conduct experiments on ADE20K to verify the performance of GmNet-S3. Following the existing methods, we integrate GmNet into the Semantic FPN framework. With significant improvements the speed, GmNer-S3 still match the performance on semantic segmentation task with RepV-T-M1.5 and ElficientFormer-L3..
(002)) with better efficiency in
under similar model sizes.
2.4 Apmask and 1.4 APhoz
A.13 THE USAGE OF LARGE LANGUAGE MODELS (LLMs)
We used GPT for polishing grammar and improving readability. All research ideas and analyses were conducted by the authors, who take full responsibility for the content.
19

---

Published as a conference paper at ICLR 2026
View color
Ros19
ResdB Exi
Resis Cate Rehu
Res Gute GEU
View
-Aoe of
- Train Acc
Test Aue
13.1
(a) Comparison between Res18 Res18-Ewp.
1007
(b) Comparison between Res Se--Gate-ReLL and Res Res18-ENe 18--p.
(c) Comparison between 1------t---eLD and
Figure 15: Learning curves of Resnet18 and its variants for 100 epochs with optimizer Adam. The learning rate is set to 0.001. Radii are to {0, 6, 12, 18, +o0)
View color
Res18
Rest8-Bup
Rest8- Gate Rebu
Rec Gate GELU
View Shape
of
Train Ace
Test Ace
(a) Comparison between Res18 and Res18-Eww
retly -10
(b) Comparison between Res18-Gate-Rer Gate-ReLU and Res18-Ewp
(c) Comparison between 1as18-Gtt--RLU and Resl8-Gate-Gii
Figure 16: Learning curves of Resnet18 8 and its variants for 100 epochs with optimizer of SGD. Radii are set to {0, 4, 8, 12, } When the frequency intervals are too small, the differences between the methods become less pronounced, especially in the lower-frequency components. However, remains evident that the different variants have distinct effects on learning the various frequency components.
20

---

Published as a conference paper at ICLR 2026
View
Best--opp
shape
Moth
200
(a) Comparison between Res18 and Res18-Ewp.
Comparison between Res18-Gate-Be ReUc and Res1 1esp8-Ewp.
(c) Comparison between
and Res18-Gate-GELU
Figure 17: Learning curves of Resnet18 and its variants for 100 epochs with optimizer of SGD. Radii are set to {0,8, 16, 24, +oo When the frequency intervals are too large, all models struggle classify higher-frequency components. Although differences between models become more pro- nounced in the lower- fower-frequency components, such as between GELU and ReLU activations, to better understand the training dynamics, necessary examine the differences the high- -igufireye components as well.
21


Reviews:
[{"review_id": "iu6tDqbl3T", "reviewer": "Reviewer_krko", "summary": "This paper introduces GmNet, a new architecture for lightweight Convolutional Neural Networks (CNNs) that achieves state-of-the-art performance. The core idea is the adaptation of the Gated Linear Unit (GLU) mechanism, which has been highly successful in language models, to the domain of efficient computer vision. The authors present a systematic analysis of GLU from a frequency perspective, demonstrating that the element-wise multiplication within GLU acts as a convolution in the frequency domain, which can selectively amplify high-frequency signals. Based on these insights, they introduce the Gating Mechanism Network (GmNet), a simple and efficient architecture that incorporates these frequency-aware gating principles. Without complex training strategies, GmNet sets a new state-of-the-art for efficient models, with the GmNet-S4 variant achieving 81.5% top-1 accuracy on ImageNet-1K, validating the effectiveness of designing lightweight models that can learn from a full spectrum of frequencies.", "questions": "- The swiGLU and SiLU are commonly used in the gated MLP of current LLMs design, and many of those LLMs have demonstrated the effectiveness of these activation functions. It would be better to evaluate these two activations in CNN models to see whether they can also improve model accuracy.\n- Typos:\n\t- L45: showed -> shown\n\t- L318: traini ng -> training\n\t- L427: Moverover -> Moreover\n\t- L86: duplicated sentences: \n\t\t- Consequently, our approach is inherently more effective in preserving and enhancing high-frequency information. \n\t\t- As a result, the former structure is inherently more effective in preserving and enhancing high-frequency information.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper shows not only the number of FLOPs and parameters, but also the model latency on both GPU and a mobile device, which demonstrates the effectiveness of the proposed lightweight model.\n- The proposed GmNet block is simple and clean, integrating the gating unit effectively without excessive architectural complexity. But the state-of-the-art (SOTA) empirical results on the ImageNet benchmark show its effectiveness.\n- The authors perform thorough ablation studies that validate their design choices. They compare different activation functions and different GLU designs.", "weaknesses": "- The novelty of the proposed method is limited. In the current LLM design, GLU variants are widely used in the gated MLP modules. Those LLM models already demonstrated the effectiveness of those designs.\n- In CNNs, many \"quasi-attention\" methods, e.g., SENet[1], GCNet[2], have already explored the element-wise multiplication of the activation tensors, and their results show that the gated design in CNNs can help achieve better model accuracy, similar to the findings of this GmNet. It would be better to compare GmNet with those methods to demonstrate its advantages.\n- The results in Table 6 cannot support the hypothesis in Section 3.2 (or Figure 4) that ReLU6 can be better for high-frequency data and not good for low-frequency data. The accuracy gain should mainly come from the activation function design, which leads to better model performance in general. This improvement doesn't seem to be related to how the activation function processes low or high frequencies.\n\n[1] Jie Hu, et al. Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks, NeurIPS, 2018\n\n[2]Yue Cao, et al. GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond, ICCV workshop, 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ahzO117cks", "reviewer": "Reviewer_i7B2", "summary": "This paper reinterprets GLU gating from a frequency-domain standpoint and derives design rules that are instantiated in a compact backbone called GmNet. Concretely, the model uses a shared-representation gate, 7×7 depthwise convolutions to mix frequency bands, and favors ReLU6 for a more stable high/low-frequency trade-off. The authors also propose a DFT-based frequency-split evaluation protocol and run ablations on activation choice, gating variants, and latency–accuracy trade-offs. Experiments show GmNet achieves good performance on ImageNet-1K, with runtime comparisons against efficient baselines. Evidence is provided mainly on ImageNet classification. Overall, the primary contribution is an explanatory frequency view that motivates a modest architectural tweak for efficient vision models, supported by single-task experiments and ablations.", "questions": "1.\tWould you be able to provide quantitative spectral evidence—e.g., per-layer power spectra before/after gating, high/low-frequency energy ratios, and responses to controlled sinusoid/wavelet probes—and also formulate a clear, testable proposition about the gate’s frequency response that could, in principle, be falsified by these experiments?\n2.\tUnder strictly matched FLOPs, parameter counts, and training recipes, could you report a contribution breakdown for (a) baseline, (b) +7×7 depthwise conv only, (c) +gate only, (d) +ReLU6 only, and (e) full GmNet, including mean ± std over at least three random seeds?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Recasts GLU with a frequency-domain hypothesis and condenses it into a single, drop-in GmNet block (shared-representation gate + flanking 7×7 depthwise convs, ReLU6), keeping parameters/latency overhead small and integration into lightweight CNNs straightforward.\n2. On ImageNet-1K, reports SOTA/near-SOTA accuracy–latency trade-offs against strong efficient baselines under matched settings, with consistent gains across model scales.", "weaknesses": "1.\tThe frequency claim remains qualitative and no per-layer power spectra, gain/attenuation curves, or controlled sinusoid/wavelet probes; “high-frequency amplification” is hard to verify quantitatively.\n2.\tArchitectural novelty is modest (GLU + 7×7 DWConv + ReLU6); gains may conflate with kernel size/width/depth/recipe. Lacks a contribution breakdown under strictly matched FLOPs/params and training budgets.\n3.\tEvidence is centered on ImageNet in the main text; downstream/cross-dataset/robustness results are relegated to the supplement and explored shallowly, limiting generality.\n4.\tMeasurement/reporting gaps: mostly single-run top-1 without confidence intervals; latency lacks a standardized, reproducible protocol (hardware, batch size, warm-up, operator fusion). \n5.\tPositioning vs contemporaneous gating/frequency methods is under-specified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LK6l3lctqT", "reviewer": "Reviewer_HTpE", "summary": "The paper introduces a theoretical study and experimental evaluation of Gated Linear Units (GLU) in CNNS from a Fourier perspective. In the theoretical part, the authors analyze the effects of non-linear activations of different variants of  ReLUs and GELUs and their properties in frequency space and their effect on the ability of networks to learn functions with high frequency components. Subsequently, the paper derives a theoretical analysis of GLUs in the same context by deriving the frequency mixing effect of element-wise multiplications in the frequency domain from the Convolution Theorem. This leads to the theoretical conclusion, that GLUs are suitable to allow networks to learn to preserve higher frequency information.\n\nThe paper then implements GMNet, a GLU enhanced version of MobileNetV2 and shows superior performance on ImageNet in comparison to SOTA efficient image classification networks. The authors conclude, that the effect of GLUs is most beneficial efficient network architectures (for mobile devices), which are known to have a stronger in low-frequency bias.", "questions": "Q1:  I'm curious what the GLUs are actually learning. Vitalization of the GLU weight by layer and the resulting changes in the frequency spectrum would be very informative. It would be very interesting to know in which parts of the network higher frequency information is relevant \n\nQ2: the evaluation focuses on efficient nets. it would be very interesting to see GLUs in large modern CNNs like ConvNext (v2). I would speculate, that the larger kernels in large CNNs (which are less band limited) mitigate the effects of GLUs. This would not harm the results of this paper, but would further increase the general insights. Have the authors tried this?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important theoretical question: the properties of simple network components liker activation functions and element-wise multiplications (and a such GLUs) have not been studied even though these properties of other components (like convolution filters, padding and down-sampling units) have shown significant impact on the performance and robustness/generalization abilities of CNNs.\n\nOn the practical side, the introduced GMNet confirms the theoretical analysis and shows significant performance increases over efficient SOTA architectures.", "weaknesses": "N1: The introductory part is hard to read for someone who is not working on related problems. Especially the difference between the frequency representation of inputs (read data and feature maps in later layers) and the the frequency representation of the learned decision function is not well explained and leaves unclarities to which the the low-frequency bias is actually relating.\n\nN2: Also the introductory and related work section somewhat neglect the bigger picture of similar studies conducted on other components (especially convolutions and down-sampling) - for example [1] and [2]. While some works are cited, the relation of these works are not put into perspective.\n  \nN3: as the authors mentioned, basing decisions increasing on high frequencies, can result in aliasing [3] and significant loss of robustness [2]. Here an ablation study would be very beneficial.\n\nN4:  the experimental evaluation is limited to low resolution ImageNet data. This also directly limits the frequency representation. It would be good to see how the proposed approach behaves on high resolution data. \n\n**Minor:**\n* Figure 1 is not matching it's caption: there are only two different frequency inputs (not three) and where is that frog? Also the schematic illustration of fig 1 does not actually show the effect of GLUs (on real data)-> it actually would be nice to have a plot like this: real data -> ReLU vs GELU vs GLU output in spatial and frequency domain.\n\n\n[1] Durall et. al. \"Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Grabinski, et al. \"Frequencylowcut pooling-plug and play against catastrophic overfitting.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Grabinski, Julia, Janis Keuper, and Margret Keuper. \"Aliasing and adversarial robust generalization of cnns.\" Machine Learning 111.11 (2022): 3925-3951.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b9b3ZGDd9f", "reviewer": "Reviewer_KXHh", "summary": "The paper deals with lightweight networks for edge devices. The propose to use gating mechanisms like GLUs to directly counteract the low-frequency bias present in many efficient architectures. They analyse GLUs from a frequency perspective and present a lightweight architecture they call Gating Mechanism Network (GmNet) on top of MobileNetV2. The study different gating mechanisms and show that the simplest linear one performs best. They get state-of-the-art resuts on ImageNet", "questions": "Q1) Which dataset do Fig3/4 refer to? Is it all on ImageNet1k?\n\nQ2) Why is the performance across epochs important in Fig3/4? what does the x-axis tell us over different epochs (eg vs re porting final performance)? It is hard for me to understand why the authors claim that \"the convergence is faster\" in the discussion, when the rightmost figure shows more or less equal convergence trends.\n\nQ3) Could GmNet be adapted to recent transformer based architectures like RepVit or Lsnet?\n\nQ4) \"As shown in Fig. 4, we present the testing accuracy curves under varying frequency thresholds\" - isn't the treshold (r) set to 10 everywhere? Apologies if i misunderstood.\n\nQ5) What is the top-1 ImageNet1k performance on GMNet-S1/2/3/4 when simply changing the ReLu6 to simple ReLU (keeping the exact same architecture otherwise)? \n\nQ6) How do you justify that the simplest, linear gating mechanism works best (eg in Tab 4/8)?\n\nQ7) Is there a  typo in the caption of Fig 1 where it mentions \" retains the general shape of the frog\"?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper analyses GLUs from a frequency perspective and links their core operations to the ability to modulate a network’s spectral response. They use that to counteract a low-frequency bias in lightweioght architectures\n* It is nice to see that linear gating seems to perform best \n* The paper reports strong results on ImageNet1k when basing GmNet on MobileNetV2 (2018). It also gets strong results on COCO for Object detection & segmentation.", "weaknesses": "1) The paper offers limited theoretical depth and bases the analysis on empirical Fourier observations. The analysis section (Sec 3) is weak in scope and effectively only studies how a smooth (Gelu) and \"non-smooth\" (ReLU6) activation function affect classification performance of a MobileNet durign training for low/high frequency components of the input image.\n\n2) There is a lot of emphasis on results with low radii $r$. Although it is clear that the proposed method improves over other methods for high frequencies (Table 3), in terms of overall performance, the gains are small over the state-of-the-art (<1%). It is unclear to me how top-1 overall gain relates to the analysis in Sec 3 or results on low/high frequencies.\n\n3) The paper would be strong if results on more datasets/tasks were presented. (E.g. Places or fine-grained classification datasets like CUB/Cars etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)