Toward Principled Flexible Scaling for Self-Gated Neural Activation
Authors: Sudong Cai
Shuyuan Zheng
Bingzhi Chen
Shuai Yuan
Chuan Xiao
Jianbin Qin
Bing WANG
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Activation Functions
Principled Neural Activation Modeling
Neural Activation Interpretation
Non-local Information Modeling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We identify, elucidate, and address the underexplored non-local tension problem and introduce FleS, a self-gated activation function that enhances discriminative visual recognition through adaptive scaling.

Abstract: Neural networks necessitate nonlinearities to achieve universal approximability.
Traditional activation functions introduce nonlinearities through rigid feature rectifications.
Recent self-gated variants improve traditional methods in fitting flexibility by incorporating learnable content-aware factors and non-local dependencies, enabling dynamic adjustments to activation curves via adaptive translation and scaling.
While SOTA approaches achieve notable gains in conventional CNN layers, they struggle to enhance Transformer layers, where fine-grained context is inherently modeled, severely reducing the effectiveness of non-local dependencies leveraged in activation processes.
We refer to this critical yet unexplored challenge as the **non-local tension** of activation.
Drawing on a decision-making perspective, we systematically analyze the origins of the non-local tension problem and explore the initial solution to foster a more discriminative and generalizable neural activation methodology.
This is achieved by rethinking how non-local cues are encoded and transformed into adaptive scaling coefficients, which in turn recalibrate the contributions of features to filter updates through neural activation.
Grounded in these insights, we present **FleS**, a novel self-gated activation model for discriminative pattern recognition.
Extensive experiments on various popular benchmarks validate our interpretable methodology for improving neural activation modeling.

Published as a conference paper at ICLR 2026
TOWARD PRINCIPLED FLEXIBLE SCALING FOR SELF- GATED NEURAL ACTIVATION
Sudong Cai 13, Shuyuan Zheng Bingzhi Chen?, Shuai Yuan?, Chuan Xiao?, Jianbin Qin', Bing Wang
'The Hong Kong Polytechnic University: 2The University of Osaka; 3 Beijing Institute of Technology, Zhuhai; "Shenzhen University (sudong. cai, bing-w. wang j pu.lyu. edu.hk; blenglist.osaka-ul
ABSTRACT
Neural networks necessitate nonlinearities to achieve universal approximability. Traditional activation functions introduce nonlinearities through rigid feature rectifi- cations. Recent self-gated variants improve traditional methods in fitting flexibility by incorporating learnable content-aware factors and non-local dependencies, abling dynamic adjustments to activation curves via adaptive translation and scaling. While SOTA approaches achieve notable gains in conventional CNN layers, they struggle to enhance Transformer layers, where fine-grained context is inherently modeled, severely reducing the effectiveness non-local dependencies leveraged in activation processes. We refer to this critical yet unexplored challenge as the non-local tension of activation. Drawing on a decision- maaiin perspective, we systematically analyze the origins of the non-local tension problem and explore an initial solution to foster a more discriminative and generalizable neural activation methodology. This is achieved by rethinking how non-local cues are encoded and transformed into adaptive scaling coefficients, which turn recalibrate the contri- butions of features to filter updates through neural activation. Grounded in these insights, we present FleS, novel self self-gatce gated activation model for discriminative pattern recognition. Extensive experiments various popular benchmarks validate our interpretable methodology for improving neural activation modeling.
INTRODUCTION
The essence of neural operations in pattern recognition lies in approximating the underlying input- output relationships, which are inherently nonlinear at the level of individual neurons. This neces- sitates the use of nonlinear activation functions for learning effective neural representations (Cy-
benko 1989 Hornik et al.) 1989 Hornik 1991 Leshno et al. 1993). Conventional activation
functions (Duaas et al.) 2000 Nair& & Hinton 2010), inspired by the neuronal stimulus-response
mechanism (Serre et al. 2005 Serre 2006, Serre et al. 2007 Kouh 2007), model neural activation
in rigid paradigm. Recent efforts (Hendrycks & Gimpel 2016 Elfwing al. 2018 Ma et al. 2021)
have been made to enhance the fitting flexibility of activation by leveraging smooth self-gating or incorporating content-aware inductive biases. typical self-gated activation process & IR. IR can be expressed as:
(ā) p
(I)
where each ă = (w, x> + b E R represents projected (affine/linear transformed) feature element, computed from the inner product of filter vector w and feature vector æ, together with bias term b. A weighting function, IR - IR then assigns gating weight (Ø) (which typically lies in the interval (0, 1)), to recalibrate the pre-activation T.
However, current inspirations (Biswas et al. 2202b a Misra 2020 Ramachandran et al.) 2018) for activation modeling largely stem from empirical heuristics (e.g., biological cues), while the mecha- nism of effective actívation remains abstract and lacks robust theoretical guidance. This explanatory gap hampers activation modeling and evaluation (Cai 2024b), significantly limiting further progress.
"orresponding authors: Bing Wang and Shuyuan Zheng Code: https: / thub. com http:///oooooooouccc//ccaaaa///e

---

Published as a conference paper at ICLR 2026
For example, recent approaches have incorporated dynamic non-local cues to enhance self-gated activation and provide additional fitting flexibility (Ma et al. 2021 Chen et al.) 2020). Although these methods yield gains in standard CNNs, they fall drastically short in enhancing Transformer layers, which inherently encode fine-graine graiird non-local dependencies outside the activation module. More intuitively, aggregating gains from non-local cues learned both within and beyond activation processes appears contradictory. We refer to this critical yet unstudied challenge as non-local tension. Grounded in a decision-making perspective, our work is the first to investigate the non-local tension problem in self-gated activation and to propose a principled approach for its resolution.
Intuition 1.1 (Decision- Decision-mmainn--ioo
interpretation of activation). Our interpretation is inspired
by multi-criteris decision making, in particular grey relational analysis (Deng) 1 1982 Lite 2025 and
related models (Rezaei 2016 Qin et al. 2017 Xu al. 2020 Joshi & Kumar 2016 where the goal is to score and rank alternative solutions based on given set of criteria, often by comparing each alternative to one or more ideal solutions when raw criteria values are not directly comparable. We view the neural affine-actio activaaion pipeline of single neuron an instance of this setling and make the following identification. Specifically: (1) each filter (weight vector) w acts as an updatable ideal alternative (solution) that aims approximate
underlying ideal pattern w x ; (2) each feature vector x. is treated as a realistic alternative that proposes a candidate update divection for wo; (3) the channels serve as decision criteria, since both w and
represented as channel-wise vectors; and (4) the pre-activation i = [w]]] ||:|| cos u + b is viewed as an indication signal of importance score for x with respect to w where the feature-to to--ee-----ftterr similarity cos tw æ is the primary contributor, and the filter norm (w), feature norm ||a]], and bias term b act as rectifying components. From this perspective, the weighting function p(.) acts as a sign-aware recalibration mechanism for indication signals of importance across alternatives and criteria, analogous to determining decision weights in multi-criteria decision making. Under this view, the activation process can be interpreted as a form of directed featttr selection, and we leverage this conceptual lens to explore non-local tension.
In this view, we identify a key underlying factor-the the trivially discriminative gating weights phenomenon as a major cause of non-local tension: given two feature (vectors) æ; and æ, and a filter (vector) w, all associated with the ideal patterns w eevn if x; is significantly more important than 2. w.r.t. u** the smooth weighting function p may assign them close gating weights p(.:) and ((zz), leading to only trivial difference in the recalibration of their importance scores z; and zy- As gating weights can modulate the contributions of features to filter updates (refer to Sec. Sec. A for a detailed discussion). insufficient discriminative power in the assigned weights may limit the elfective use of features for filter learning. Consequently, this phenomenon leads to a situation where, when Transformer layers integrate beneficial information provided by non-local cues into a feature, the activation does not correspondingly increase the gating weight to reflect the pre-activation's 's enhanced importance, thereby causing the non-local tension problem (Intuition 3.2)
We identify the saturation behavior of p in typical self-gated activation models as a critical cause of the trivially discriminative gating weights phenomenon. We refer to this underlying issue as the convergence limitation. Specifically, assume that ă monotonically reflects the relative importance of a feature x w.r.t. the w * When both ī; and ē. are relatively large positive values but ã; is significantly larger than ē.., the gating weights p(ā:) and p(ā:) remain distinguishable to effectively recalibrate the pre-activations o; and ž.. enabling the activation mechanism to effectively emphasize or suppress the contributions of features (to filter update). However, saturating gating functions p. such as the Sigmoid and ERF-based functions, tend to lose discriminability under the above condition, causing the contrast between feature contributions to vanish. Accordingly, we interpret the non-local tension problem as downstream effect of this limitation within self-gated activation.
This identification motivates our novel remedy, FleS, which addresses trivially discriminative gating weights by modeling flexible scaling coefficients. Guided by decision- decision-making principles, these coefficients adaptively control the bound and steepness of p. enabling it to attend to informative response intervals in accordance with appropriate non-local cues (see Sec. 4. Consequently, FleS sustains fine-grained recalibration for activation, preserving meaningful differences among relatively important features under convergence limitation. thereby mitigating the non-local tension challenge.
Our main contributions are threefold: spired activation analysis tools. We identify the convergence limitation tension, and highlight flexible scaling as a critical property for overcoming this limitation, enabling more discriminative neural activation. (2) Based on (1) we address the under-explored non-local
(1) We present new insights that extend marisiin-making...
key cause of non-local
2

---

Published as a conference paper at ICLR 2026
tension problem by presenting the novel activation model FleS, which extends the methodology for interpretable neural activation modeling for pattern recognition. (3) Extensive experiments across popular vision and NLP benchmarks validate new insights, highlighting the effecctiveness, versa- tility, robustness, and extensibility of the FleS methodology, and demonstrating its notable advantages over SOTA activation methods, especially in neural networks with non-local token mixers.
2 RELATED WORK
Inspired by the primate stimulus-responss mechanism (Serre et al. 2007 2005; Kouh 2007).
activation functions such Softplus (Dugas al. 2000) and hard approximation ReLU Nair Hinton 2010) were proposed. ReLU, in particular, leverages rigid 0/1 mask to activate features, effectively mitigating gradient vanishing in range-limited nonlinearities (e.g.. Sigmoid and Tanh) and motivating series of variants: LeakyReLU (Maas et al.) 2013) alleviates "dead" units via a leakage factor, while PReLU (He et al. 2015) learns the negative slope adaptively. More recently, self-gated alternatives relax such rigidity: SiLU (Elfwing et al., 2018) enables soft selection via sigmoid gate, GELU (Hendrycks & Gimpel| 2016) performs smooth feature recalibration based on Gaussian Error Function (ERF), and Mish (Misra (020)) combines Tanh and Softplus to form smooth weighting curve. Although these self-gated functions improve the fitting capability of conventional activation methods, their adaptability remains limited.
SOTA activation designs improve adaptivity via dynamic bounds and context-aware gating. Swish (Ra- machandran al., (2018) parametric SiLU) scales inputs within sigmoid gate; ACON-C MMa et al. 2001) further adds a learnable bound; [Biswas et al. (2022a) extend GELU with ERF-based parametrizations (ErfAct, Pserf via Softplus); SMUs Biswas
2022b) use a smoothed maximum to enhance ERF-style rectification; and Meta-ACON (Ma et al. (021) generalizes lightweight chan- nel attention (Hu al.) 2020) for context-conditioned modulation. However, these gates/attentions inject relatively coarse non-local cues; on architectures that already model non-local context (e.g.. Transformer layers), this often induces non-local tension, limiting gains and applicability.
More related our work. Cai (2023) interpreted neural activation from a decision-making perspective and identified the overlooked mismatched feature scoring (MFS) problem. They demonstrated that standard CNNs, by addressing the MFS problem, can be strengthened to rival advanced Transformers in image recognition solely by leveraging effective activation functions without major architectural changes (Cail 2024a) Nevertheless, prior interpretations underexplored the contradictory use of different forms of non-local cues, thus struggling to enhance Transformers due to non-local tension.
In this work, introduce insights to extend activation analysis by elucidating the convergence limitation in typical self-gater activation. This enables us to derive the first solution to non-local tension by introducing flexible, FleS-style scaling mechanism with explainability, designed to adaptively recalibrate the bounds and steepness of activation functions to sufficiently leverage the contributions of features.
3 NON-LOCAL TENSION CHALLENGE
We elucidate the cause of the non-local tension challenge (Fig. 11 in self-gated activation, following step-by-step analysis. Our investigation is grounded in a simple yet effective decision-makinn lens, which forms our methodological insights and motivates our FleS activation model as the first solution. We first introduce the preliminaries and then clarify how the convergence limitation induces the trivially discriminative gating weights phenomenon, which eventually triggers the non-local tension challenge. Formal proofs and derivations supporting this section are provided in Sec. A
3.1 PRELIMINARY
Our analysis is based on the preliminary settings (Cail 2023 2024a), which involve simple settings with image inputs: (1) A network includes T sequential learning layers, where each layer indexed by 2, T. (2) Let E C²C H" xL² denote the input feature map of layer-r, where C* and H" x L" represent the number of channels and the spatial resolution, respectively. (3) The operation layer-7 spatial location (h, E QH xI> is defined as x x == @ (#z (h, I)), where un (c) C and xx (h, 1) ROT denote the e-th filter vector and the feature vector location (h,
3

---

Published as a conference paper at ICLR 2026
Paelat
Paraeey Sare
Reveggilng Parcke
VITTON
64)
Cause of Non-local Tension
Figure 1: Illustration of the non-local tension challenge, and the key intuitions behind our remedy, FleS (Intuition 3.2). (a) In a typical self-gated activation ((x), saturating, monotonically non- decreasing function p(ī) is used to weight a rreevaiivvtiion ž. thus recalibrating the contribution of the corresponding input feature vector
the filter w. When the pre- activatiias become sufficiently large-even when they are identified as important by the non-local mechanism and pushed to even larger values-p assigns almost indistinguishable gating weights to such that have notably different importance levels. This convergence limitation causes the additional contributions brought by non- local cues to be nearly neutralized (trivially discriminative gating weights), which in turn triggers non-local tension. (b) and (c) depict intuitively accessible strategy underlying FleS to alleviate non-local tension: an adaptive dual activation scaling, where vertical scaling rectifies the activation range and horizontal scaling modulates the activation steepness.
respectively. Here, S2 x- represents the spatial lattice of X", and 2. (h, (wn xx (h, 1)>+be denotes the pre-activation obtained by applying an affine transformation, parameterized by the filter w (c) and bias be, to the feature vector x (h,(). (). Notably, (i) layer-r includes C filters and (ii) é IR - E represents an activation function. For simplicity, we omit the layer index (T) and pixel coordinate (h. 1) in the following discussion. For example, w" (c), x (h. 1), #7 (h, 1), and be become w. æ, 2, and b respectively. Our analysis focuses on a typical self-gated activation process ø : R - R. perspective, we
which can be expressed by Eq. (1 (i.e., & (T) = م (T) interpret the pre-activation ã as an importance measure associated with the input feature æ relative to the ideal pattern ** Furthermore, we treat p(ū) as a gating weight that modulates the response 3 to emphasize/suppress the contribution of x: to the update of un. More intuitively, the more important is for filter update, the larger the assigned weight p(ž) should be (the supporting reasons are elaborated in Sec. A.I). To ensure convergence, P is commonly assumed to satisfy (Wul 2022): (1) lim 00 p (T) ह O: (2) lim / (z) .M > O. Moreover, our work considers a constant-sign monotonic function P to ensure effective self -ated activation by adopting a relevant conclusion ([Cai 2023 Proposition 2)) and assumes p is non-negnn non-negative without loss of generality. Note that omit
T). From a decision-makinn
normalization layers (e.g.. BN (Toffe & Szegedy 2015) and LN (Ba et al. 2016)) in the analysis for simplicity, as their inclusion does not affect the conclusions.
3.2 PROBLEM ANALYSIS
Cause of trivially discrininative weights. We identify the trivially discriminative gating weights phenomenon as a key underlying factor that triggers the non-local tension, which widely exists in the P of popular/SOTA self-gated activation functions (e.g., Sigmoid SiLU (Elfwing et al., 2018) and ERF in GELU (Hendrycks & Gimpel 2016)).
To appropriately modulate the contribution of feature æ to updating a filter w by recalibrating its raw response (i.e., the importance score) ã we further assume the weighting function P satisfies two basic properties. Specifically, for arbitrary æ; and æj. and a given filter w corresponding to the ideal pattern w" * their responses ã; and ãj are expected to satisfy:
(a) Proper Importance
Scoring: 3; > ē if æ; is considered more important than æ; relative to w * (b) Importance Weight Alignment: p (5x)) > p (ā²z > Of ē.; > ãÿ. However, typical functions p satisfying properties (a) and
alone are insufficient to guarantee effective self- eelf-gated activation due to the convergence limitation.
which happens if p has fixed upper-boune bound (see preliminary condition (2)):
Intuition 3.1 (Convergence limitation). Specifically, for any D; and xy, when their raw importance measures 2; and 3. are both sufficiently large, the difference in their gating weights p (āx) and (ā²) can become arbitrarily and trivially small. This indicates that even if a; contributes significanily
4

---

Published as a conference paper at ICLR 2026
MIL
Roc
EIG
C
(
MLP
20
e-9 Negatiee Value
hS
Feetum Stlecton
AF
Avrerge Ptoling
Safighe Punedion
Blenent Mtut
Flexible Scaling Unit
Operational illustration of FleS. Features from different channels distinguished by distinct color families, where darker shades within each color family indicate higher feature responses.
Figure 2:
more to the update of un than v p may still fail to assign sufficiently discriminative weights to them, thereby limiting the effective use of them for model learning.
We refer to this property as convergence limitation, which results in trivially discriminative gating weights problem, which we characterized as follows:
Theorem 3.1 (Convergence limitation), For any it and ž; corresponding respectively to I; and x; w.r. w. if 4000 (i) M 0, then, for any given O, there must exist threshold X such that for all 3,, Tÿ >> X, we have (p(T.) -(ā;)] <.
Cause of non-local tension. We clarify the non-local tension challenge based on the trivially discrim- inative gating weights phenomenon, which hinders the activation module from fully leveraging the context cues already modeled outside activation:
Intuition 3.2 (Trivially discriminative gating weights cause non-local tension). Transformer layers utilize the attention mechanism to capture non-local cues to enhance features. More generally, consider abstract self-gated activation process with inputs contained non-local cues, where token mixer casts dynamic translation a to modulate æ. and then inputs the adjusted features into neuron (leveraging and b) produce the finer feature H' for activation:
= (w. (xc + a)) + b; p p
(2) 0,āy: <w,z))+b> and the
Then, suppose we have Ti Fy such that āy: (w, x.) + adjustments introduced by and a; effectively push the finer outputs further away from zero, such that i, > Z; and H > T respectively. Suppose that Y and E are sufficiently large such that the differences between the weights assigned by p ((() and p (3t) become trivial (i.e., almost degrading ReLU's rigid masking process), the learning contributions introduced by a and aj can be significantly neutralized. This leads to a failure to fully exploit non non-local cues from informative features for model learning.
MODELING
Based on the theoretical awareness, we identify effectively resolving the non-local tension challenge as key avenue enhancing self- activation models in networks that leverage non-local token mixers. In particular, addressing the non-local tension challenge hinges on resolving the convergence limitation. These insights underpin our novel methodology: FleS- style adaptive scaling mechanism.
4.1 PROTOTYPE: PLES-PROTO
Overview, Drawing on the insights and conclusions in Sec. 3.2 we introduce FleS prototype:
à (ō) KveP (Koo 8.
where Kys and Kho denote the vertical and horizontal scaling coefficients, respectively.
New insights into scaling from non-local cues. We embody Kge and Kho based on set interdepen- dent heuristic insights:

---

Published as a conference paper at ICLR 2026
Intuition 4.1. (1) non-local tension is statistical effect: For given filter, subset of features that exhibit high importance, relative to the layer-wise feature space, may collectively trigger non-local tension, leading to their contributions being underutilized. (2) Based (1), any numerical modulation of an activation process for a feature (or its response) should consider its relative relation reference feature group. Independent modulation of individual activation input is thus inadequate for capturing the contextual nature of non-local tension. (3) A channel corresponds to the response of specific filter. Given that different filters may vary significantly in magniuue and direction, two implications follow: (a) Even for the same feature group. response magnitudes can vary across channels (e.g., larger-norm filters tend to produce larger-magnitude responses). With static p. such responses may be pushed into flatter regions of the galing curve, diminishing the discriminability of uctivation. (b) The triggering interval for non-loca, tension may differ across filters: the same response magnitude may correspond to different relative feature contributions under different filters. For example, if two filters share the same direction but differ significantly in norm, given feature will generally have greater gradient inffuence on the smaller-nowe nore filter. Hence, scaling strategies should differentiate between channels to account for such discrepancies. (4) Any (ideal) pattern can be represented by an order-sensitive sequence of filters. Object categories can be viewed as semantically meaningful abstractions of high-level patterns. In particular, we posit that object category information offers a meaningful basis for grouping effective responses and converting them into adaptive scaling coefficients.
In particular, insights I) (2) and collectivv suggest that: (i) different filters (i.e., channels) may exhibit different triggering zones to non-local tension w.r.t. their pre-activation aetiviiiiaiin (ii) non-local tension is primarily associated with high-importance features and is negligibly influenced by unimportant ones. Therefore, features should be utilized discriminatively according to their importance levels when extracting statistical (non-local) cues for generating appropriate scale coefficients. (iii) Furthermore, according to (i) different filters may adopt different criteria for "important features"' based on their responses. We posit that a feature is considered important if it yields at least a positive response. We formalize this insight via Intuition and Proposition
4.1 which play a key role in our methodology.
Intuition 4.2. For given filter w, consider set of features with positive responses. {x | "eli>00 0), and a set of features with negative responses, {x jē < 0j. Assume that: (1) the contributions of features are recalibrated by sigmoid-like function p (ã) Sigmioo (fhoT), where Bue. 3ho E IK+ are fixed positive values (note that commonly used p can be approximated by a sigmoid-like function); and (2) 8 N(u,o), a). which represents the random variable as a proxy for generating filter responses. Then, (i) positive features are more likely to have higher expected contributions than negative ones after recalibration. Moreover, the more even the distribution is, the more likely positive features are to dominate the overall contribution; (ii) particularly, for an extremely even distribution, the contribution of negative features is negligible compared to that of positive features.
Proposition 4.1 formalizes Intuition 4.2
Proposition 4.1 (Relative recalibration bias). For conditions assumed in Intuition 4.2 for any fixed B(p(š))==0)
ju E R, the conditional expectation ratio: R (u, B(p(a)(5=-0) satisfies limg--o R (M, σ) = O.
Further discussion and proofs of Intuition 4.2 are provided in Sec. B.
Modeling coupled scaling coefficients, We then follow the above insights model the FleS-style coupled vertical and horizontal scaling coefficients in a simple yet effective manner (more technical choices are discussed in Sec. [.12]:
Nog softplus (Avelt ({*=)) + Tue) Kho softplus (ahoff ((xt)) + Tho)
(4)
respectively, where,
+ = meansex, { 5 0)
(5)
denotes the mean filter response of non-negative features within the largest accessible set X. of channel-c (e.g., in ImageNet model training, X, represents the set of feature vectors in channel-c across the entire mini-batch). We refer to Æ as the effective mean response, which isolates the influence of negative features. preventing it from neutralizing the contribution of positive features (further elaborated in Sec. B.I.. Then, / ((#t)) }) defines the normalized effective mean response
6

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation FleS-Proto on ImageNet dataset (Deng al.
2009).
Backbone
Activation
#Shufftle
#Params.
FLOPs
Top-1(()f
GELU
21.1M
2.6G
78.7
Swin-Micro
85.2
FleS-Proto
21.1M
2.6G
77.3
Swin-Base
GELU
87.7M
15.1G
83.5
* The Swin-Micro (Liu et al. 2021) backhone is applied, where FleS activation function is compared with the GELOU (Hendrycks & Gimpel 2016) haseline.
across all the channels:
H C
(1
(6)
-1-1 2
Ove and aho. initialized small value (e.g., 1 10- are pair of learnable parameters that scales y ([#2 }) to introduce adaptability. Yee and Tho. initialized to fixed value (e.g.. 0.6, ensuring that Kye and Kho are initially close to 1.0), are learnable parameters stabilize training in the early stages. Notably, the Softplus functions are applied to impose a smooth positive constraint, as the importance levels measured by pre-activations are sign-sensitive.
In particular, we identify that this interpretable yet simple design of FleS-Proto can introduce incredible enhancements Transformer layers (using Swin- Transformer for example (Liu et al.) 202I)) ImageNet with the standard non-shuffle evaluation setting. Specifically, in the non-shuffle setting, images are arranged in the order of their categories, so that the largest clean channel-specific statistical range corresponds to the entire mini-batch. This provides highly valid channel-wise effective responses { for calculating five and Kho-
As shown in Tab. D using the standard 300 300-epoch Transformer-tailordd recipe (Tovvron et al. 2021
Liu al. 2021 without auxiliary training data, a small-size Swin-Micro variant (f.e., Swin- 1, 2, 2, 2 consisting of 9 blocks, requires only about 50% of the computational cost of Swin-T (Liu al. 2021)) achieves significant performance improvements only by replacing GELU with Prooo for activation. It outperforms Swin-B by a remarkable margin (85 (85.2% vs. 83.5%) while requiring only approximately 1/6 of the computational cost (2.6G FLOPs vs. 15.1G FLOPs). However, when the channel effective mean responses are calculated on a shuffled batch for evaluation, they can no longer provide clean class-specife statistics. As result, the Top-1 accuracy of FleS-Proto Swin-Micro drops 77.3%, performing even worse than the vanilla Swin-Micro baseline. These two phenomena motivate our practical design of FleS for broader applicability.
4.2 PRACTICAL MODEL: FLES
Practical modeling. Experimental evidence Tab. demonstrates the critical importance of the effective channel mean response in modeling the scaling coefficients. Building upon this awareness, we design FleS, applicable to scenarios where obtaining clean channel-specific statistics is challenging. Specifically, utilize lightweight MLP (with channel reduction ratio of 32 by default) channel attribute recorder to compute each scaling coefficient as follows:
Kee MLP
(x:)
Kho
MLP ho (x
Notably, for realistic recognition tasks, compute each effective channel mean response म. readily accessible region in practice, Xe, where x + RC represents the effective channel mean vector, and æ its e-th element. For example, on ImageNet, we set 7* meanae x, {ā F > 0}. where X, the e-th channel slice of the input feature map object detection). X, uses a finer neighborhood (e.g., a x 15 patch on COCO (Lin et al. 2014
H CCC For dense tasks (e.g.,
As the key to realizing adaptive scaling in realistic pattern recognition tasks, MLPs exhibit translation equivariance, allowing them to detect informative regularities in the effective channel mean vectors
ã R across the inputs with complex class distributions (e.g., shuffled single-class images or multi-class road scene images). These regularities are then adaptively converted into scale coefficients. The operational diagram of FleS is illustrated in Fig.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of different activation functions on ImageNet (Dng et al. 2009) with (left) Swin- Min (Liu et al. 2021) (Swin-[1. 1, 1] and (right) PoolFormer-ST ST2 Yuu et al. 2022 backbones.
Backbone
Swin-Min (Liu al.
2021
PoolFormer-Sh S12 (Yu et al.
2022
#Epechs
120
300
Activation
#Params.
FLOPs
Top-1 (%)1
#Params.
FLOPs
Top-1 (%)
GELU [Hendrycks al., 2016)
11.8M
1,6G
68.7
11.9M
1.8G
77.2
ReLU
Nair Hinton 2010
11.8M
1.6G
68.1
11.9M
1.8G
76.6
SiLU
Elfwing et al.
2018)
11.MM
1.6G
68.9
11.9M
1.8G
Mish
Misra 2020)
2020)
118M
1.6G
68.6
11.9M
1.8G
77.1
Pserf
Biswas et al.
2022a
118M
1.6G
69.0
11.9M
1.8G
NaN
SMU
Biswas
2022b
11.M
1.6G
68.9
11.9M
1.8G
77.3
IIEU Cai
2023)
13.4M
1.6G
69.5
14.3M
1.8G
78.6
AdaS CCai
2024a
13.7M
1.7G
69.7
15.1M
1.9G
78.2
StarRel.U
Yu et al. 2024)
11.8M
1.6G
69.1
11.9M
1.8G
76.8
Meta-ACON (Ma et al. 2021)
13.4M
1.6G
68.3
14.3M
1.8G
78.0
FleS (Ours)
13.0M
1.6G
71.4
13.8M
1.8G
79.4
FleS-AdaS
14.1M
1.7G
73.0
All competing methods are trained from scratch following the same recipe outlined in Implementation details.
##ppccs" denotes the epochs of training;" "NaN" denotes failed training; The baselines use GELU activation.
EXPERIMENT
We evaluate the effectiveness, versatility, and robustness our proposed FleS. Experiments are con- ducted on four major vision benchmarks: ImageNet (Deng et al. (0099 and CIFAR- 100 (Krizevskky) 2009) 4 for standard image classification, ImageNet-L2 (Liu et al.] 2019) for classification under long-tailed distributions [F) and COCO (Lin et al. 2014) for object detection (J). To further assess its generalizability beyond vision, we validate FleS on GLUE (Wang al. 2018) (C), the popular NLP benchmark. Moreover, by adapting FleS
sensstivee semantics in NLP, validate its extensibility by introducing Gatte,, stronger variant with markedly improved performance.
We evaluate FleS against widely used and SOTA activation functions. From lens, activation functions can be distinguished by the properties of p(-) (and thus the overall
decision-making
behavior of <(-)). We summarize the main competing activation methods as follows: (i) Monotonic
activation functions ((-) with discontinuous p(-): Softplus (Dugas et al., 2000); ReLU (Nair
& Hinton 2010); and StarReLU (Yu et al.) 2024). (ii) Static self-gated functions x(-) with
smooth, monotonic p(-): GELU (Hendrycks & Gimpel 2016); SiLU (Elfwing et al. 2018); and
Mish (Misral 2020). (iii) Dynamic self-gated functions o(-) with a modified p(-) integrating
adaptive components, Pserf (Biswas et al. 20: SMU (Biswas et al. (022b): Meta-ACON (Ma
al. 2021; IIEU (Cai) 2023); AdaShift (Cail 2004a); and FleS. Notably, FleS can be viewed as particular form of dynamic self-gated activation (category (iii) where p(-) is constructed from
sign-aware, channel-wist statistics of a reference feature group, and these statistics are then processed by small MLPs to produce a feature-importance-clibinnn
adaptive scaling of activations.
Furthermore, we provide methodological insights into the modeling FleS PeeS-Proo/FFleS through targeted ablation studies in Sec. 5.2 with further details available in Sec.
IMAGENET CLASSIFICATION
Implementation details. We evaluate FleS across two representative MetaFormer backbones to validate its effectiveness in alleviating the non-local tension challenge (that is, whether it effectively activates neural features that intrinsically contain non-local cues): (1) Swin-Transformer (Liu et al.) 2021), the most popular vision Transformer backbone, ranging from Swin-Min (i.e., the minimal Swin model, Swin-[1, 1, 1, IJ) to Swin-T (i.e., Swin-[2 - 2)); and (2) PoolFormer-SI1 Yu et al. 2022), efficient yet effective MetaFormer model for visual recognition. To further validate the generalizability of our insight and modeling, we also evaluate FleS with (3) ResNet (He et al. 2016). the most prevalent CNN backbone (using ResNet-50). Note that the baseline Soin-Traaffommrt and PoolFormer-S S12 use GELU (Hendrycks Gimpel, 2016) activation functions, and ResNets employ ReL.U (Nair & Hinton 2010), respectively. For fair comparisons, (1) we adopt the standard
8

---

Published as a conference paper at ICLR 2026
Table 3: (Left) Comparison of the FleS-enhanced and vanilla GELU Swin-M(icro) (i.e.. Swin-
[1. 2, 2],, Swin-T, and ViT-B/16 (Dosovitskiy et al. 2021) models on ImageNet (Deng et al. 2009).
(Right) Comparison of different activation functions on ImageNet using ResNet- backbone.
Activation Backbone
#Params.
FLOPs
Top- 10e-((())1
Activation Backhone
#Params.
FLOPs Top- I (
GELU
21.1M
2.6G
78.7
ReLU
25.6M
4.1G
77.2
SiLU
21.1M
2.6G
78.6
+SE-Net
28.1M
4.1G
77.8
SMU
Swin-M
21.1M
2.6G
78.8
PRcLU
25.6M
4.1G
77.1
MI-ACON
24.2M
2.6G
78.9
PWLU
N/A
N/A
77.8
FleS
23.5M
2.6G
80.3
SMU
25.6M
4.1G
77.5
SMU-1
25.6M
4.1G
76.9
GELU
28.3M
4.4G
81.3
FReLU
ResNet-50
25.7M
4.0G
77.6
SiLU
28.3M
4.4G
81.4
DY-ReLU
27.6M
N/A
77.2
SMU
Swin-T
28.3M
4.4G
81.4
ACON-C
25.6M
3,9G
76.8
Mt-ACON
32.7M
4.4G
81.5
Mt-ACON
25.8M
3.9G
78.0
FleS
31.7M
4.4G
82.3
HIEU
25.6M
4.2G
79.7
GELU
86.6M
16.9G
79.7
AdaS
25.6M
4.1G
79.9
ViT-B/16
FleS
97.4M
16.9G
80.7
FleS
28.1M
4.1G
80.1
Note: FleS with Noe and omitted is equivalent to SiLU (Elfwing al 2018
training-ever eeaiinngevalattio recipe (Touvron et al. 2021; Liu ct al. 2021) for Vision Transformers, except for (2) Swin-Min, for which we reduce the 300-epoch training to 120 epochs (due time and resource constraints); (3) For ResNets, we adopt the standard CNN training training-evaluation recipe (Zhou et al.
202T Ma al. 2021) (details are included Sec. G Experiments are conducted using four A6000 GPUs.
Experimental results. The comparative results of our activation function, FleS, and the SOTA/popular competing methods across various types and sizes of MetaFormer and ResNet back- bones are reported in Tabs. 2 and and Tab. 3 respectively, leading to four key observations: (1) On Transfrranfrrmer and PoolFormer-SI/ backbones, FleS demonstrates significant improvements over all existing popular and SOTA activation functions. In particular, the accuracy gains introduced by FleS over SOTA activation methods are of those methods over the GELU baseline. Notably, prior to the introduction of FleS, SOTA methods like Meta-ACON and SMU also exploited non-local information re-scale the bounds of the activation functions but fell short in enhancing the static self-gated baseline, GELU, for Transformer layers. This validates the critical importance of our interpretable methodological insights for addressing the non-local tension challenge. (2) Then, on ResNet backbones, FleS also demonstrates clear improvements over the prevailing and SOTA activation functions. Although FleS requires additional parameters to capture meaningful statistical cues, it brings only negligible computational cost (measured by FLOPss. (3) FleS exhibits remarkable scalability. It not only works effectively when applied independently but is also capable of boosting other SOTA activation functions. For example, incorporating FleS's flexible scaling scheme into AdaShift (Cail 2024a) improves Swin-Min. (4) The effectiveness of FleS's flexible scaling is consistently demonstrated across different network architectures and sizes. These results comprehensively validate our insights and practical designs for modeling discriminative neural activation methods. More experimental results and relevant analysis are provided the appendix.
more pronounced than the improvements
5.2 ABLATION STUDIES
We conduct extensive ablation studies on ImageNet to probe the theoretical and empirical insights underpinning FleS. Here, we present three representative studies and include more studies in Sec. E
On w/ or w/o feature statistics for flexible scaling. We elucidate and demonstrate the significance of channel effective mean responses (denoted by "channel indicators" in the following text for simplicity) in driving adaptive scaling for discriminative self-gated activation. Here, we validate this insight by comparing omits the channel indicators. Specifically, the vertical and horizontal scaling coefficients of FleS-Do DG defined fie softplus (Yve) and Kjo softplus (Yho), respectively. The comparative results are presented in Tab, leading to two key observations: (1) W/o leveraging statistical cues in channel indicators, FleS-DG exhibits significantly lower accuracy than FleS. (2) Despite this, FleS-DG still
original FleS with downgraded FleS variant (denoted as "FleS-DG") that

---

Published as a conference paper at ICLR 2026
Table 4: Ablation studies on (left) or w/o the channel effective mean intensities { for modeling
FleS coefficients; and (right) mining statistical cues within positive feature elements for FleS.
Activation Backhone
#Params.
FLOOPs
T
Activation
Backhone
#Params.
FL.PSs Top-1
GELU
11.8M
1.6G
68.7
GELU
1188
1.6G
68.7
FieS-DG Swin-Min
11.8M
1.6G
69.1
FleS-P&N Swin-Min
13.0M
1.6G
69.8
FleS
13.0M
1.6G
71.4
FleS
13.0M
1.6G
71.4
"FleS-DG" denotes the FleS variant omitting (# } in generating scaling coefficients.
TTe FleS variant "FleS-P&N" averages positive and negative features for calculation of channel indicators.
outperforms the GELU baseline. These findings indicate that (1) statistical cues provided by channel indicators critical guidance; and (2) scaling coefficients remain beneficial for self self-gated activation, even the absence of channel statistical cues. These validate our insights.
On mining statistical cues in positive features. We clucidate the necessity of treating positive and negative features differently in the modeling of non-local cues (in Sec. 4.1), aiming to prevent the neutralization effect induced by negative features on positive ones in adaptive scaling. Accordingly, we propose excluding negative responses when computing channel indicators as simple yet effective design strategy. We validate this insight by comparing our original FleS with tailored FleS variant (denoted as "FleS-P&N") that averages both positive and negative responses to calculate channel indicators. As reported in Tab. 4 FleS-P& improves upon the GELU baseline but yields clearly inferior results to the original FleS. This supports our theoretical analysis.
0.5
Pusitive: stage=
Positive: stage-2
Pusitive slage-
0.3
Positive stage-4
Negative: stage- 1
0.2
Negative stage=2
0.1
Negative stage-3
Negative: stage-1
10
20
30
40
50
60
70
80
90
100
110
Figure 3: Illustration positive versus negative gradient magnitudes over 120 training epochs. On empirical positive-negative gradient contributions. We conduct a tailored study on ImageNet with Swin-Min + FlcS and, for each epoch and stage, measuring the ratio between the mean gradient magnitudes at positions with positive versus negative FleS outputs. Averaged over training, gradients on positive responses are much larger than on negative ones: the mean ceaitive-tonnegatie ratios are about 5.3x, 7.9x, 12.7x, x, and 13.8x for stages 1-4, respectively, with deeper stages exhibiting stronger asymmetry. Over epochs, this gap widens from early to mid training and then remains high: ratios increase roughly from 3-9x (epochs 0-9) to 6-15x (epochs 40-79), and stay clearly above the carly-phase levels in the late phase (epochs 80-119) Thus, training proceeds, positive responses increasingly dominate the effective gradient budget, especially deeper stages. These align with our sign-aware indicator design: the dominant optimization signal already lies the positive side, so emphasizing in the indicator helps avoid cancellation effects. Fig. visualliz these trends.
CONCLUSION
In this work, we identified and formalized the convergence limitation inherent in self-gated neural activation, showing that it gives rise to the unstudied yet critical non-local tension challenge, which we found to hinder the potential activation functions enhancing modern neural networks. Grounded in mackinn principles and their encouraged insights, we derived the FleS-style dynamic scaling scheme that provided the first principled remedy to non-local tension. Comprehensive experiments various popular benchmarks, together with targeted ablation studies, verified its effectiveness, generalizability. robustness, and extensibility, indicating strong potential to advance interpretable activation modeling for pattern recognition.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was jointly supported by the Research Grants Council of Hong Kong (Nos. 25206524, 15212925); the National Natural Science Foundation of China (Nos. 42301 62302172); the Inno- vation and Technology Fund (No. PRP/068/23FX); the Platform Project of Unmanned Autonomous Systems Research Centre (No. P0049516) 6); the Seed Projects of Smart Cities Research Institute (Nos. POO5 1028, P0054511); ); JSPS KAKENHI (No. JP25K21207), JST CREST (No. JPMJCR22M2); and the Guangdong Basic and Applied Basic Research Foundation (Nos. 2025A1515010225,
2025A151506000)..
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06453 2016.
Koushik Biswas, Sandcep Kumar, Shilpak Banerjec, and Ashish Kumar Pandey. Erfact and Pserf: Non-monotonic smooth trainable activation functions. In Proc. AAAI Conference Artificial Intelligence (AAAI), 2022a.
Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey. Smooth maximum unit: Smooth activation function for deep networks using smoothing maximum technique. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022b.
Sudong Cai. lieu: Rethinking neural feature activation from decision-making. Proc. IEEE International Conference on Computer Vision (ICCV), pp. 5796-5806, 2023.
Sudong Cai. AdaShift: Learning discriminative self-gated neural feature activation with adaptive shift factor. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5947-5956, 2024a.
Sudong Cai. Learning Discriminative Neural Representations for Visual Recognition. PhD thesis, Kyoto University, 2024b.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprint arXiv: 900.00/50000 2019.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic ReLU, In Proc. European Conference on Computer Vision (ECCV), 2020.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 244)3303-314 1989.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In Proc. IEEE Conference Computer Vision and Pattern 2009.
Recognition (CVPR), 2009.
Ju-Long Deng. Control problems of grey systems. Systems & Control Letters, (():288-294) 1982.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICIR), 2021.
Charles Dugas, Yoshua Bengio, François
Bélisle, Claude Nadeau, and René Garcia. Incorporat- ing second-order functional knowledge for better option pricing, In Proc. Advances in Neural Information Processing Systems (NeurIPS). 2000.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
11

---

Published as a conference paper at ICLR 2026
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint orXiv:/706.02677 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. IEEE International Conference on Computer Vision (ICCV), pp. 1026-10344 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint 2016.
Kurt Hornik Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2): 251-257, 1991.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural Networks, 2(5):359-360 1989.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. gqueeze-and-Excinee Networks.
IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 42(8):2011-2033, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. International Conference Machine Learning (ICMI., pp. 448-456, 2015.
Deepa Joshi and Sanjay Kumar. Interval-valued intuitionistic hesitant fuzzy Choquet integral based TOPSIS method for multi-criteria group decision making. European Journal of Operational Research, 248( (18(1):18-119) 2016.
Minjoon Kouh. Toward a more biologically plausible model of object recognition. PhD thesis, MIT, 2007.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken, Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861-867, 1993.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective Kernel Networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 510-519, 2019.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects context, In Proc. European Conference on Computer Vision (ECCV), pp. 740-755, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, pp. 2999-3007, 2017.
Sifeng Liu. Grey systems analysis: Methods, models and applications. Springer Nature, 2025.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In Proc. IEEE International Conference on Computer Vision (ICCV), 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2026
Ningning Ma, Xiangyu Zhang, Ming Liu, and Jian Sun. Activate not: Learning customized activation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 8032-8042, 2021.
Andrew L Maas, Awni Y Hannun, Andrew Ng. et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ICML Workshop, 2013.
Diganta Misra. Mish: A self regularized non-monotonic neural activation function. In Proc. British Machine Vision Conference (BMVC), 2020.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In Proc. International Conference on Machine Learning (ICML), 2010.
Jindong Qin, Xinwang Liu, and Witold Pedryez. An extended TODIM multi-criteria group decision making method for green supplier selection in interval type-2 fuzzy environment. European
Journal of Operational Research, 258(2):626-66) 638, 2017.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. Proc. Workshop Track of the 6th International Conference on Learning Representations (ICLR), 2018.
Jafar Rezaei. Best-worst multi-criteria decision makinn method: Some properties and linear model. lannga-international Journal of Management Science, 64:126 2016.
T. Serre, M. Kouh, C. Cadieu, U. Knoblich, G. Kreiman, and T. Poggio. A theory of object recognition: Computations and circuits in the feedforward path of the ventral stream in primate visual cortex. Technical report, MIT, 2005.
Thomas Serre. Learning a dictionary of shape-components in visual cortex: Comparison with neurons, humans and machines. PhD thesis, MIT, 2006.
Thomas Serre, Aude Oliva, and Tomaso Poggio. feedforward architecture accounts for rapid categorization. PNAS, 104(15):6424 6429, 2007.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. In Proc. International Conference on Machine Learning (ICML), 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop on BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, 2018
Lei Wu. Learning single neuron for non-monotonic activation functions. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2022.
Ye Xu, Ye Li, Lijun Zheng, Liang Cui, Sha Li, Wei Li, and Yanpeng Cai. Site selection of wind farms using GIS and multi-criteria decision making method in Wafangdian, China. Energy, 207: 11222. 2020.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. MetaFormer is actually what you need for vision. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. MetaFormer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2):89-911,, 2024.
Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Hiuan Yang. Decoupled Dynamic Filter Networks, In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
13

---

Published as a conference paper at ICLR 2026
A DISCUSSION & PROOFS OF SECTION 3
A.1 RATIONALE BEHHIND FEATURE CONTRIBUTION RECALIBRATION FOR FILTER UPDATE IN NEURAL SELF-GATED ACTIVATION
In this appendix. we clarify the rationale behind treating the gating weights p(ž) as key factor in recalibrating the influence (i.e., contribution) of features to the update of filters. Note that this appendix inherits the preliminary settings and assumptions proposed in the main paper.
Motivation, Grounded in the decision- perspective, we revisit the physical meaning of neural activation from the viewpoint of back-propagation. Specifically, at a learning layer we consider the activation process, Φ (&), as a form of contribution recalibration: for a given input feature vector and filter vector w, the pre-activation (e.g., ã (w. æ),, where we omit the bias term b to simplify the following analysis) can be viewed as a similarity- importance of with respect p(ă) to modulate this raw score, effectively emphasizing or suppressing the influence of x on the update of w.
score that estimates the relative The activation function then accordingly assigns gating weight
Proxy objective. Then, consider recognition scenario and build upon several key behavioral properties of the cross-entropy (CE) loss:
(1) The CE gradient dominated by prediction error, producing a clear push-pull effect that penalizes incorrect classes and pulls toward the correct one;
(2) At the logit layer, the gradient can be simplified as the product a prediction deviation term and the input feature, exhibiting locally linear response;
(3) In early training or shallower layers, the back-propagated CE gradient approximately retains a first-order structure;
(4) Each sample's 's contribution to parameter updates can be modeled independently, without involving global interactions or higher-order coupling.
Based on these properties, we adopt the following proxy objective to approximate the CE gradient behavior at layer T:
1 (b
Crocx
2
Notably, we let
= (w" xx)
denoting the virtual ideal feature-filte response, as a reference to help simplify the representation of the effective loss. Here, w* denotes the ideal pattern (i.e., effective objective) with respect to the filter w.
Then, the gradient of groxy about w be calculated as:
a Proxx
( (ō) (ō*))
Grrixy
7 ( o (ā*))
dw
a PrONY
0o (ō)
a ( (7)
d (i+))
dw
8 Proxx
a (p (2) 8)
db
( (8) (i*))
ōw
(o (ō) Φ (5
(p'
i + p (ī)) T.
(10)
As the reference group, we consider the case where the activation is removed. this case, the gradient is calculated as:
V Cproxy (ī- 5z x.
(11)
Feature contribution recalibration.
14

---

Published as a conference paper at ICLR 2026
Intuition A.1. In particular, for significantly large Æ-that is, when the corresponding is of relatively high importance with respect to w (or w" in a physical sense)-unde the condition where expect the potential occurrence the non-local tension phenomenon. identify that the proxy gradient V w Crexxy can be further approximately simplified as follows:
V Uproyy = (Ø (ā) o (o" (T)) x,
(12)
i.c. the term p' (i)- - is negligible.
We formalize the conclusion in Intuition by Proposition A.1 Proposition A.1. For gele- activation function (ã) p (ā) i that satisfies:
(1) limg-- ( (3) 0;
(2) and + M> 0.
we have:
lim p (z) = =0
(13)
Proof. core insight. limg-++0 pf (i) - -0 d' (8) p
(+)) (8 - +00 (i.e the term } That is, if p(ā), where limı- p (क) M,
(3( is an infinitesimal of higher order than grows more slowly than in()) by an order (or orders) of magnitude as +00, then we have lim (z) 0.
Proof by contradiction. Our following proof is carried out using the Fundamental Theorem of Calculus, the Lagrange Mean Value Theorem, and contradiction.
Suppose 3g > 0 and an unbounded increasing sequence {ën) such that:
[ 'p' (fx) Vn,
(14)
Lee, suppose that the convergence of p' (8)) is not ensured (the contradictory case Proposition A.1 thus, we have:
Corollary A.2. 1p (ix)) g.
Further, assume that is differentiable on the interval
[ 20,,1 by using Corollary A.2 and Lagrange
Mean Value Theorem, we have: I[., E [Tn, 27.] Jnn >> 0 such that
p (2ū - (āā)) p' (En) (2ī - -in)== = (En) In.
(15)
therefore,
g ā,
G G
1e (22,) (ā..) p (sn) In
In
(16)
Sn
2
This leads contradictory conclusion the pre-assumed condition a convergent function).
(2) (i.e., lp (2ii..)) p (5n) is
In other words, for any given small value 0, there exists threshold x such that the inequality p'(T) ã < holds for all i > X. This conclusion is equivalent to lim pf (ø) - = O, and explains why we treat the term p' d()) - as negligible when is sufficiently large.
This completes the proof of Proposition A.1
Then, under the assumption that is sufficiently large, we have:
V n Aproxy
= (o (6) - o (ā*)) p(T) (p) f p(=)p x*) (ā)...
=Mp(T) -
-Mp(f) Vw Ww/ooo
(17)
That is, Mp(it) can be interpreted as the effect of recalibration. Furthermore, since M is a fixed value for each e. posit that p(ã)) acts the primary contributor to feature contribution recalibration through a self-gated activation process.
This supports the intuitions and insights proposed based feature contribution recalibration.
15

---

Published as a conference paper at ICLR 2026
A.2 PROOF OF THEOREM 3.1
In the main paper, we present Theorem 3.1 to formalize the problem of convergence limitation, which may underlie the non-local lension challenge and is inherently present in typical self-gated functions characterized by the general form:
+ (z)
(G) 8
(18)
where - (w, x) + b E R is given feature element (scalar), derived from the inner product of the filter and feature vectors along with a bias term b, and p R - IR assigns score p (œ) to weight T. Typically, the weighting function p is commonly required to satisfy that (Wul 2022):
lima--xx
0: and lim P ( - M
Retrospect. For ease of reference, restate Theorem 3.1 from the main text Theorem A.3 here.
Theorem A.3 (Convergence limitation: restatement of Theorem 3.1.. For any Ts and ă correspond-
ing respectively to æi and æ w.r.f. u, iflimz--*o P (ã) M U, then, for any given 0, there must exist threshold X such that for all ã,, .z X, we have lp(ã,) p(ā,)) <.
Proof. By the definition of limits, the given assumption:
lim p (ã) M, a 2020
(19)
where M O, implies that: for any 0, there exists sufficiently large scalar X such that for all I > X, we have:
l MI <
(20)
Therefore, Vi,, 83 > X, applying the above conclusion, we have:
(a. - MI و (0(T,) M <
(21)
Based on the triangle inequality, we have:
< 2
(22)
This completes the proof.
B DISCUSSION & PROOFS OF SECTION
B. DISCUSSION AND PROOFS OF INTUITION AND PROPOSITION 4.1
Overview, We propose Proposition 4.1 to help clarify our intuition that inspires the modeling of the non-local indicators ( } (Eq. (5) in the main text), i.e., posit that positive and negative features should be used in discriminative manner to produce the non-local indicators for inducing FleS-style adaptive scaling scheme, as prevent the contributions of positive features from being neutralized by negative ones. In particular, we identify that positive features tend to have relatively higher accumulated contributions than negative features, and this advantage becomes more pronounced when the input distribution is relatively flat.
Before proving Proposition
first discuss more general case clarify Intuition All discussions and proofs in this appendix inherit the assumptions and pre-conditions established therein.
Advantage of positive contribution after activation. Through self-gated activation, the contributions of positive and negative features are relatively emphasized and suppressed, respectively, thereby giving positive features a relative advantage over negative ones in their overall influence.
This finding can be clarified follows:
Proposition B.1. Let
16

---

Published as a conference paper at ICLR 2026
(1) (()) -Presigmoid (Bhoa) denote sigmoid-like function, where Bue, Bho > 0 and i RR;
(2) X ~ (N,, representing the random variable as a proxy for generating filter responses;
(3) ip (z; μ. σ) denote the Gaussian density with mean μ. and standard deviation o;
(4) f(, a) _0 p(@) ((;; (1,)) dë - S p(3) p( ps, () dă define the cumulative gap of gating weights (i.e., indicator of recalibrated contribution).
Then, for any given a > 0:
(i) du > 0, we have f 0) > 0;
(ii) 3po < 0 such that Vu (Ho, 0), f(,, 0) > 0.
(i) clearly holds, we then omit its detailed proof, as y (T; Iu20 symmetric about E = and p(t) > () for any _ > 0. This ensures 50 p(ž) 14, σ) dă _ p(ā) p(;; u, o) dē, ie.. f(u, o) <= 0.
Proof. First, Proposition
thereby focus on Proposition B.
ii in the following.
Fix any > 0. By the continuity of p( and co(i;; jr, o), and by dominated convergence on compact j--iterrvlss, the mapping ju - f(u, or) is continuous. Invoking Item (i) we have f(0, σr) > Hence, by continuity at particular, letting μo == -8 < 0 yields f(u, > 0 for all μ E (po. 0).
0, there exists = 8(o) > 0 such that | < implies f Ø)) 0. In
This completes the proof.
Remark B.1. Proposition B.1 indicates that, under a standard self-gated activation, the aggregate influence of positive features tends to be amplified relative negatives, thus tending to yield higher contributions to filter updates during the activation process.
Furthermore, we present an extension of Proposition B.I as follows. This conclusion helps clarify how the relative advantage of the contributions of positive features becomes more pronounced when the input distribution is relatively flat:
Proposition B.2. Consider f(u, that admits a negative interval (this condition is satisfiable, e.g.. for p (ō) sigmoid (ī) and = 1, then, numerically
verify that f f(u, o) lue-1< 0). Let No (a) denote the unique negative root of f(u, = 0. Then the function po (a) is strictly decreasing in 0.
Proof. We apply the implicit function theorem to the identity f(Po (a), = 0. Then
due 0f/0o
(23)
do 05/0/u
We compute the partial derivatives of f. First, note that
(-1
(8-1)) H)2
(6
do p( plii = ((z;11,0)
(24)
σ3
Hence,
0f
|0 (()))(ā;;po))
( g3
1
0
dā
p(o)
p(o),((;
dē
20
(25)
Because p(ū) monotonic and much larger on than on 0, and (5 - 12) grows quickly on I > 0, the first integral dominates. Thus, 2f 80
> 0.
Next, compute the derivative with respect μ as:
( - 1)
p(;
p(;; a).
(26)
17

---

Published as a conference paper at ICLR 2026
Therefore, have:
@f 0p
p(z) p.) p(z) p( -)p((; dž
(27)
When 1 < O, both integrals involve (Ø 14) 0, but the second is weighted by smaller p()). Hence the second term dominates, and Combining the above, we conclude that:
Pp < 0.
duo 05/00 0,
(28)
do 0f/0μ
and therefore (o(a) is strictly decreasing a.
This completes the proof.
Notably, Propositions B. 1 and B.2 underpin our intuition for Proposition 4.1 which can be regarded as an extreme case of Proposition B.T We now address Proposition 4.1 also clarify related conclusions (i.e., Lemma B.4 and Proposition
B.5)
to further support our intuition.
Retrospect. For ease of reference, we restate Proposition 4.1 from the main text as Proposition B.3 here.
Proposition B.3 (Relative contribution recalibration bias: restatement of Proposition For conditions assumed in Proposition B.T. for any given μ IR, the conditional expectation ratio: R (14,0) = {(p())(=<O) satisfies limg-a R. (1,, o) = O.
Proof. We first consider p (8) sigmoid (œ), i.e., Bye and 3ro for simplicity, and then generalize the conclusion to general sigmoid-lie functions.
Let C+ (1,, S0 p(0) ( pt dă and C_ (J,,0) S (()) (0;; 14, a) dă indicate the cumu- lative recalibrated contributions of positive and negative features, respectively.
core insight. As a increases, the Gaussian distribution o μ1 σ) becomes increasingly flat. In the limit 20, the contributions restricted to any finite interval [a, b] vanish:
C+ (ae(aa))
0, C- 0.
(29)
Thus, the total contributions C+ and C_ are entirely determined by the behavior over the tails of the distribution.
We focus on two semi-infinite regions: (M_,0) with M_ < 0, and (0, M4) with M, >> O, where
C_ C- C-I o0)"
C4 C+ lae(o.M.)
(30)
In this regime, the gating function satisfies:
p i
for गे (0, M4),
p(ū) 0 for ē (M-,0).
(31)
As result, the ratio of negative to positive cumulative contributions tends to zero:
BE(M-.0) 0,
(32)
C+ (+),
which implies:
C_ 0,
(33)
C+
since the excluded portions [M_ 0] and M,] are finite intervals whose contribution becomes negligible in the limit
Proof based on the core insight. Fix any > 0. To analyze the ratio C-/C+, as a - oc, we partition each integral into contributions over finite and infinite intervals.
18

---

Published as a conference paper at ICLR 2026
Let M_ < 0 < M+ be two constants. We split:
M_
(2: (( 0) dă + { JM_
C_
p( dē
(34)
1
M+
m ((
C+
(
(35)
M+
Finite region vanishing. For any & > O, since (( μl, p) converges uniformly to 0 on any compact interval as - sc, we can find E > 0 such that for all o > E.
finite <E,
I fuite < <E.
(36)
Asymptotic behavior on infinite tails. Note that on E (M,,0), P(T) - 0, and on T E (M+, oo).
p(ū) 1. Thus, for large enough a,
M- (i;; dē p(M-) I
M-
1 & sup (E(-oo,M))
p(ā)
(10;;1100 dā
(37)
--0
Similarly,
x I
1
inf xE(M..x)
p(t)
p(e;u,o) > p(+)
di.
(38)
MM
M.
Since p(M-) < 1 and p(M+) xe 1, and since pp μ, u,o) is normalized, we can always choose M_ and M, so that:
M_
()): dē >
((z:11,) dē for all large o.
(39)
M
Conclude the ratio vanishes. For all large o,
C_(,o)
1 finite
P(M-). S p(ō) dē + p(M+) Shr. ((ō) dē s
(40)
(14,0)
1 + Tumite
Hence, we have:
Lemma B.4. For the conditions assumed in Proposition B.3 we have: lim (v.r)
0.
C+ (u.o)
19

---

Published as a conference paper at ICLR 2026
Then, by using Lemma B.4
we have the following conclusion:
E
[p
( I x<0 0)
lim R (k, = lim a-+-0
p
0)
S- p()) ((;
F(
lim
+0
P[2=0) So (
di
dē
lim
0-+00
de
1-4
((0) p( 0) dē
lim
(. dë 2 o
[0]
lim
2 So ( o) dī
lim
p (2:
0,
(41)
where
① (2)
e 7
dt
(42)
00
denotes the standard normal cumulative distribution function (CDF).
Note that the above conclusions are applicable general sigmoid-like functions, since simultaneously changing See and Pro in both C+ (1,0) and C_ (μ, does not affect the results.
This completes the proof.
A variant case: for uniformly distributed inputs. Here, we generalize our Intuition 4 to the case where the inputs I obey a uniform distribution.
Proposition B.5. For x 11 (5-,5+) where 8- < and 8+ 0, we have the following conclusion:
P2 p())dž O.
lim
o(a)d
Proof. Because lim (() a O, we have: lim (
Further, since
4 p (T) M 0 == lim p (T), we have two derived conclusions:
(1) (() monotonically non-decreasing and p (7) > 0:
(2) 3V+ + such that ( > 0 for any ă A+
Without loss of generality, suppose p (i) K lazat, where K > 0 is a constant, we have:
8+
lim
p o dē
(ā))
lim 8 A
p (()) dē
lim
p (i) dz
lim
Kdē
+++
+00
(43)
Therefore, have: lims+ -0t P (8) dē +00.
20

---

to Published as a conference paper at ICLR 2026
Then, prove lima- So p (() dē is upper-bounded,
Because lim, p p()) ã 0, we have:
(
(
(
lim ( at
lim
lim
lim
0.
(44)
2-++0
Without loss of generality, let (*) p(-8), we have: g (ū) = o (22)
2 - +00
That is,
VE > 0 such that Vi > A we have:
g ( $ C
(45)
il+c
where C' > 0 and > 0 are constants.
Based on the derived conclusions above, we have:
+++
lim 8 + Jo
g (z) dē
9 (z) dē
lim
g (z)
8+
C1 lim
9 (2) dē.
(46)
+000
where C1 > 0 is a constant, and:
x- =C..c-c.0=C.. +00,
lim -
9 (i) dē & C lim
C
C.
++0
iltc
+0
(47)
where C2 a constant. This proves that limg---oo 18 (z) dē is upper- bounded
0
Therefore, we have the conclusion: lims --00,8 -000 R2 (6 ( This completes the proof.
C GENERALIZATION BEYOND VISION: GLUE EVALUATION
To further assess the generalizability of FleS beyond vision, we validate it on the GLUE bench- mark (Wang et 2018
Adapting vision FleS NLP. FleS aims to extract, encode, and inject task-relevant commonalities across a semantically meaningful group of inputs ("reference feature group"). In vision, class labels provide a natural grouping; in NLP, although token semantics are highly context-sensitive. task- dependent regularities still emerge (e.g., sentiment). Thus, when adapting FleS to NLP, the critical step the construction the indicator (clas--ggrupp-ellevan statistics) under contextual volatility, while the modulation mechanism (e.g., MLP- -baaedd scaling) can remain largely unchanged. We adapt the practical FleS from the vision domain to NLP based on these heuristic insights.
Practical models. Here, we introduce two NLP FleS variants:
FleS-NLP:
direct adaptation of vision FleS to sequences; replace hard positive selection with Softplus us to stably extract indicators on short sequences. Moreover, because each token already encodes condensed semanties and token meanings can vary substantially within sentence, we compute token-level "class" indicator rather than sentence-level mean.
lightweight enhancement that mimics state evolution via a depthwise ID conv inside a simple Sigmoid-based gaLe-MLP (with channel reduction ratio of 8 by default). This adds only ~6% parameters/FLOOs over the baseline model with GELU activation. Notably, this SeqGate indicator provides sequence-awand content-adaptive smoothing akin to lightweight state evolution, while remaining batch ordering.
Gate::
21

---

Published as a conference paper at ICLR 2026
Training protocol.
Backbone. BERT-Tiny.
Pretraining. BookCorpus + Wikipedia from tekniitttbbrrt--bee---eaaae max len=128, batch=4096, 1r-6x 10 epochs=50, warmup=0.1, wd=0.01, mixed precision.
Fine-tuning, GLUE (9 tasks): batch=128, 10-x10-1; epochs: CoLAA/STS-B=100, MRPC=10, others=4.
Table 5: Comparative results on GLUE benchmark (full-task evaluation).
Backbone
BERT-Tiny
BERT-Mini
BERT-Small
Activation
GELU
FleS-NLP
FleS-SeqGate
GELU
GELU
#Params.
4.38M
4.51M
4.66M
4.66M
11.20M
28.80M
#Num-Attn-Blocks
2
2
2
#Pretrained-Epp
50
50
14
50
50
50
CoLA
MCC
12.13
20.68
23.48
22.04
17.38
27.24
SST-2
Acc.
81.54
82.45
83.72
84.17
83.65
89.79
FI
77.78
81.75
83.20
83.56
76.35
87.10
MRPC
Acc.
69.61
71.32
73.77
75.98
76.68
81.13
FT
78.00
78.57
81.99
82.21
80.55
84.86
QQP
Acc.
83.21
83.25
86.50
86.48
85,80
88.73
STS-B Pearson
23.65
23.72
77.03
79.68
84.25
85.20
Spearman
23.72
22.88
76.37
79.14
84.33
85.13
-M (Acč.)
65.58
67.25
71.77
72.44
67.47
77.42
MNLI
-MM (Acc.)
65.66
68.31
72.25
72.94
67.72
77.97
QNLI Acc.
63.77
65.37
81.59
81.37
73.84
83.87
RTE
Acc.
51.26
55.60
57.04
55.96
57.04
58,84
WNLI
Acc.
50.70
56.34
56.34
56.34
49.30
45.07
SCORE
56.72
59.44
69.97
70.18
67.59
74.32
BERT- Tiny enhanced with FleS-NLP and FleS-SeqGate are compared against the GELU baseline and substantially larger models, i.e., BERT-Mini and BERT-Small. Note that "WNLI" (marked by "*") is excluded from the final score as recommended (Wang et al. 2018).
Findings. Our observations are fourfold:
FleS-NLP enjoys clear improvements over the GELU baseline with minimal overhead, aligning with the vision findings.
FleS-SeqGate delivers remarkably powerful gains over both GELU and FleS-NLP with only marginal additional cost.
Well-suited for fast pretraining: FleS- See-SeeGate achieves ncar-50-epoch performance after
just 14 epochs, suggesting fast convergence and favorable optimization behavior.
Challenging significantly larger models: FleS-SeçGate enables BERT-Timy to surpass much larger models, i.e., BERT-Mini (~ 2.5x in parameters and computational cost) by notable margin, while remaining competitive with BERT-Small (> 6 in parameters and computational cost).
These demonstrate the effectiveness, generalizability. and extensibility of our theoretical and heuristic insights, as well as the strong potential the FleS activation methodology inspired by them.
D FURTHER CLABORATION ON THE USE OF POSITIVE AND NEGATIVE PRE-ACTIVATIONS IN FLES INDICATOR MODELING
central design principle in FleS is to construct channel-wise indicators via a sign-aware re- calibration that monotonically emphasizes positive features and suppresses negative ones before
22

---

Published as a conference paper at ICLR 2026
summarizing pre- are-activations The positive-only indicator used in default configuration is the simplest instantiation of this idea: it explicitly distinguishes positive and negative responses and focuses the statistics on positively activated features.
We suggest choosing between the positive-only indicator and softer alternatives based on the applica- tion scenario. For ImageNet-style vision benchmarks, a positive-only design is typically sufficient: these datasets exhibit relatively mild semantic variation in appearance, and a hard separation between positive and negative responses already provides reliable per-channel statistics at very low compu- tational cost. In contrast, for NLP tasks with more abrupt token-level semantic changes, we adopt a Softplus-basee based indicator (Sec. C.. which offers numerically smoother and more stable behavior. More generally, in applications that demand numerically safer behavior-fol example, when negative values carry informative semantic meaning-we we recommend using the two additional instantiations described in Sec. F212 with the Softplus-basee bbasld variant being generally preferred.
To clarify how we interpret positive and negative pre-activation responses, consider a self-gated activation of the form (revisiting Eq. (
( p(T) p(z) 2 0,
(48)
where p(ī) is viewed as a soft importance score indicating how strongly a response i should be retained. In this perspective, positive and negative responses are not globally absolute concepts, but are defined relative to a given filter. Suppose a layer includes two filters w1, W2 E RC with C' 3 2, and let RC be an input feature vector. The corresponding output o R² has two channels, where ū = (æ, wi> and /2 = (æ, w2) (omitting bias terms for simplicity). The same æ may be strongly suppressed by wi (e.g., L1 0) while still being informative for w2 (e.g.. g.. 32 > 0). Being down-weijg wwin-weighted in one channel does not imply that feature is discarded by the layer: it may be emphasized in other channels whose filters are better aligned with it.
Based on this understanding, we propose that an effective principle for constructing channel-wise indicators in FleS is to introduce differential feature rectification that mitigates confusion caused by Dositive-negative cancellation. Concretely, we apply monotonic, sign-aware recalibration that emphasizes positive features and attenuates negative ones before summarizing pre-activation responses. This makes the resulting statistics more faithfully reflect genuine per-channel intensity, rather than confusing truly weak channels with channels that have strong but mixed- responses. In contrast, aggregating positive and negative responses a symmetric manner may incur "-1+ 111+++--stty styye cancellation effects, which increases the burden on the MIP that predicts scaling factors from the indicator, because its input no longer reliably encodes filter-level importance statistics (Tab. g (right) empirically supports this observation). Importantly, even if response is negative for the dominant filter of one channel, the same feature can still be positively emphasized in other channels; FleS only rectifies the statistics used for scaling, not the global availability of features.
From a decision-making perspective. the scenario where negative values carry informative semantic meaning can be viewed as follows: in sufficiently complex applications, before filters have been updated enough reliably indicate feature importance, the sign of a pre-activation response (positive versus negative) may not reliably reflect utility. Under this mechanistic view, it is often preferable to use a numerically safer indicator (e.g., the Softplus-bast variant) rather than aggressively maximizing efficiency via strictly positive-only design.
E COMPLEMENTARY ABLATION STUDIES FOR SECTION 5.2
We present further key ablation studies examining the insights underpinning FleS, complementing Sec. 5.2 Unless otherwise specified, each ablation experiment is conducted on ImageNet using the Swin-Min (Liu et al. 2021) or PoolFormer-S12 YYu et al. 2022) backbones.
E. ON THE SATURATION REGIME
We quantify saturation on ImageNet using the original Swin-Micro and PoolFormer- trained for 120 epochs. For each epoch and for every MLP block, measure the fraction of activations whose self-gating weight p(i) exceeds 0.9 (for GELU, 0 < p(ā) < 1), and aggregate these fractions per hierarchical stage.
23

---

Published as a conference paper at ICLR 2026
For Swin-Micro, saturation is clearly depth-dependent in early training: in the first few epochs, only about 0.1% of activations are saturated in stages 1-2, compared to roughly 0.6% in stage and 2.5%-3% in stage 4. As training progresses, stages 2-4 stabilize around 1%-2%. while stage 1 increases to about 4%-5%, rising from almost zero to a higher steady level. On average, stages 2-4 remain in moderate range (around 1%-2%), whereas stage 1 eventually accounts for a larger saturated fraction.
PoolFormer-S12 exhibits stronger saturation under the same protocol, especially in deeper stages. In early epochs, stages 3-4 already reach roughly 3% and 6%-7%, respectively. During mid training, saturation becomes more evenly distributed across depth: stage rises to about 4%-5%, while stages 2-4 settle around 2%-3%. After the middle stage, these ratios remain relatively steady. Overall, except for the very first few epochs, non-trivial fraction of features stays in the saturation regime throughout training.
Sugel
0.00
Sugel
0.05
0.00
Sdtrdionntin
Step2
Skap2
0.03
0.06
Stepe3
0.06
Supp3
0.02
0.02
000
Suger
Sugel
0.00
0.00
10
30
40
30
70
80
50
100
110
10
40
30
(2)
80
50
110
Epoch
Epoch
(a)
(b)
Figure 4: Fraction of activations in the positive-side saturation regime (p(ø) 0.9) for (a) Swin- Micro and (b) PoolFormer-SI1 on ImageNet. For each epoch and each hierarchical stage, we compute the proportion of MLP activations whose self-gating weight exceeds 0.9 and aggregate per stage. The curves show that (i) saturation is initially stronger in deeper stages, (ii) saturation levels gradually stabilize training proceeds, and (iii) except for the very first few epochs, non-trivial fraction of features remains in the saturation regime throughout training.
From the non-local training perspective, these patterns suggest that non-local effects tend to accumu- late in deeper blocks at the beginning of training: before the model has acquired stable knowledge, deeper blocks operate at higher semantic level, so non-local information aggregates there and drives subset of features into the high-as activation (and saturated) region. As training enters the mid late phase, the model converges to a more effective allocation of activation mass across stages: saturation ratios stabilize, fluctuations shrink, and each stage maintains characteristic saturation level that reflects its role the hierarchy. Figure visualizes the saturation dynamics for both architectures.
E.2 INITIALIZATION OF YYe AND "ho
FleS introduces learnable log-scale parameters, Tue and Thos which are mapped non-negative scaling factors Kye and Kho through a Softplus transform. In all main experiments, we set "eee Tho 0.6, which yields an "identity-safe" configuration where Kee and K&o are initialized close to 1.0. This keeps FleS near the baseline activation at initialization and avoids aggressive changes to the optimization landscape.
To quantify sensitivity to this initialization, we perform an ablation on ImageNet using the Swin- Min backbone, jointly varying the initialization of Tve and "ho in the range [-1.0, 2.0]. Table summarizes the results. For values around (-2, FleS is not sensitive to the exact initialization of y: all settings consistently yield substantial gains over the GELU baseline (from 68.7% to x 71% top-1), suggesting that AdamW with warmup absorb moderate changes in the initial gate steepness and range. A more aggressive negative initialization (r = -1.0, corresponding to k s 0.31) leads to a small but noticeable drop, likely because the very small initial scaling reduces effective gradients and mimics an overly small base learning rate in early training: even with Adam W and warmup, the optimizer then struggles to reach the best trajectory. Relatively large positive values slightly increase
24

---

Published as a conference paper at ICLR 2026
the initial steepness but only induce minor, non-systematic fluctuations in accuracy, indicating that the optimizer tolerates moderately over-scaled initial gates. Based on these observations, we adopt Tve "ho 0.6 the default setting.
Table 6: Sensitivity of FleS to the initialization of Yue and "ho Im- ageNet with the Swin-Min backbone. The parameters Kive and Kho are derived from Yve and Tho via Softplus transform.
Activation
Backbone
Init. Tve: Tho
Init. Biges Rio
Top-1 (%)
GELU
Swin-Min
68.7
FleS
Swin-Min
0.6 (default)
1.0375
71.4
0.0
R 0.6931
71.2
-1.
0.3133
70.8
1.0
~ 1.3133
71.4
FleS
Swin-Min
1.4
1.6204
71.2
1.7
1.8678
71.3
2.0
2.1269
71.1
E.3 EVOLUTION OF Kye AND Kho
We study how the FleS scaling factors evolve during training by running 120-epoch ImageNet experiment with Swin-Min + FleS and, at each epoch, recording for each of the four stages (i) the stage-wise mean of Kho and (ii) the mean of the top 10% Kie values.
9
K_ (ve): stage
8
k [ve]: stage
7
[ve): stage-
6
5
x (ve]: stago
4
(ho): stage=
3
k_ (ho): stage
2
(ho): stage
k (ho): stage4
0
0
10
20
30
40
50
60
70
80
90
100
110
Figure 5: Stage-wise evolution of FleS scaling factors ImageNet with the Swin-Min backbone. For each epoch and each stage, we record (1) the stage-wise mean of the horizontal scaling Khoi and (2) the mean of the top 10% vertical scaling values Kue. Both factors start near 1, rapidly become depth-dependent in early training, and eventually settle into regime where shallow and mid stages exhibit mild scaling, whereas the deepest stage maintains the strongest horizontal and vertical scaling, reflecting its role in non-local evidence aggregation.
At initialization (epoch 0), both Kho and Ree are close to 1 across all stages, i.e., FleS starts from an approximately identity-like scaling. Within the first few epochs, they quickly become depth- dependent. In stages 2-3, the vertical scaling factor Ave rises from 1 to about 2.4-3.1, and the horizontal factor Khe rises to around 5-7. Stage also strengthens but more moderately, with Kye peaking 3.1 and Kho at 2.6. Stage 4 shows the most aggressive early horizontal scaling: Kho grows rapidly from & 1.1 to about 7.4, while Kye increases more gradually over longer period, eventually peaking around 5.0.
After this early phase, stages 1-3 gradually relax to stable, mid-range values. By the end of training (e.g., epoch 119), stage-1 Kye has decreased from its peak 3.1 1.23, and Kho stabilizes around s 2.47. Stages 2-3 similarly converge to milder scaling, with K.. around a 1.06-1.31 and K around 2.85-3.77 3.78. In contrast, the deepest stage 4 consistently maintains the strongest scaling
25

---

Published as a conference paper at ICLR 2026
throughout training: epoch 119, it still has the largest values, with Kye 2.14 and Kho 4.68. The variances exhibit a similar depth- cepth-dependent pattern: deeper stages show larger dispersion in the early epochs (especially stage 4), which gradually contracts training converges.
Overall, this 120-epoch run reveals a clear depth-dependent gating pattern: mid nid-sshllow stages converge mild scaling, while the last stage maintains the most aggressive horizontal and vertical scaling. This is consistent with our non-local training perspective, where deeper blocks bear more of the non-local evidence aggregation and thus benefit from stronger, persistent gating. Figure visualizes the per-stage Kve and KAo dynamics over training.
E.4 STABILITY IN IRREGULAR SMALL-BATCH REGIMES
To assess the stability of FleS under non-standard small-batch constructions, we consider an irregular two-stage training schedule on ImageNet with Swin-Min. In the standard setting, we use a batch size of 1024. In the irregular setting, we train for 60 epochs with batch size 132 and learning rate 3 x 10-1, followed by 60 epochs with batch size 92 and learning rate 2 x 10-4. The results are summarized in 7
Table Performance of FleS and GELU under standard barge-batch regime and an irregular small-batch schedule on ImageNet with the Swin-Min backbone.
Activation
Backbone
Batch Setting
Top-1 (%)f
GELU
68.7
Swin-Min
1024
FleS
71.4
GELU
68.2
Swin-Min
132 + 92
FleS
70.3
Both the GELU baseline and FleS exhibit small drop in top-1 accuracy under the irregular batch schedule (from 68.7% 7% to 68.2% for GELU, and from 71.4% to 70.3% for FleS), with comparable degree of degradation. Importantly, FleS still delivers clear improvements over GELU in both regimes, indicating that FlcS maintains similar tolerance irregular/small-bacce training while achieving consistently higher accuracy.
E.5 INTERACTION WITH BATCH NORMALIZATION
We further examine how FleS interacts with batch normalization by replacing layer normalization (LN) with batch normalization (BN) in each Transformer block of Swin-Min and re-training on ImageNet with the same recipe for variants.
Table 8: Ablation study on interaction between FleS and normalization.
Activation
Backbone
Norm.
#Params
FLOPs
Top-1 (%)1
GELU
Swin-Min
LN
11.8M
1.6G
68.7
FleS
Swin-Min
LN
13.0M
1.6G
71.4
GELU
Swin-Min
BN
11.8M
1.6G
69.3
FleS
Swin-Min
BN
13.0M
1.6G
73.8
* Replacing layer normalization (LN) with batch normalization (BN) brings modest gain for GELU, but much larger gain when combined with FleS.
As shown in Table g BN yields modest but noticeable improvement for the GELU baseline (top-1: 68.7 69.3). When combined with FleS, however, the gain from BN is substantially larger (top-1: 71.4 - 73.8), which is notable given that FleS already operates in a higher-accuracy regime where
improvements typically harder to obtain. This suggests non-trivial interaction between FleS's
26

---

Published as a conference paper at ICLR 2026
activation scaling and batch-normalized feature statistics. A detailed analysis of this phenomenon is beyond the scope of this work, but points to an interesting direction for future investigation.
E.6 LINEAR LAYER VERSUS MLP IN THE INDICATOR HEAD
To assess the effect of replacing MLPs with linear layers, we conduct an ablation on ImageNet with the Swin-Min and Swin-Micro backbones. Specifically, we replace each FleS MLP with low-rank linear module (i.c., an MLP with reduction ratio but without nonlincarity), denoted as F-LS-LRL, under comparable parameter budget.
Table 9: Ablation of the indicator design: MLP vs. a low-rank linear (FleS-LRL) variant.
FLOPs
Throughput images/sec.
Top-1f (%)
Activation
Backbone
Ratio r
#Params
(G)
GELU
Swin-Min
11.8M
1.6
4207.2
68.7
FIeS-LRI.
32
13.0M
1.6
4016.7
71.0
FleS (original)
32
13.0M
1.6
4011.3
71.4
Swin-Min
FlcS-LRL
24
13.4M
1.6
4013.6
71.2
FleS (original)
24
13.4M
1.6
4001.0
71.5
GELU
Swin-Micro
21.1M
2.6
2775.6
78.7
FleS-LRL
32
23.5M
2.6
2633.8
79.7
Swin-Micro
FleS (original)
32
23.5M
2.6
2616.8
80.3
FleS-LRL replaces the MLP with two linear layers of the same reduction ratio r. FleS outperforms FleS-LRL at close computational cost.
On the Swin-Min backbone, evaluate reduction ratios (r 32, default, and 24 with slightly more parameters). In both cases, FleS-LRL yields a small but consistent drop in top-1 accuracy compared to the original FleS, while still clearly outperforming the GELU baseline. On the deeper Swin-Micro backbone, the gap between FleS-LRL and FleS becomes more pronounced, in regime where marginal gains in top-1 are harder to obtain. In terms of cost, FleS-LRL and FleS have almost identical FI.OPs and very similar throughput (single RTX 3090, torch comp i e), and both introduce only a light overhead over the baselines. Given the higher accuracy gains at close version the default configuration (Table 9)
cost, we adopt the MLP-with-reductie
Intuitive elaboration on using MLPs. Our decision-making interpretation suggests two key intuitions that guide the practical design of FleS indicators. First, extracting and translating importance descriptors is central: main challenge in mitigating non-local tension is construct descriptors that numerically reflect the inter-channel importance of pre-activations and then translate these descriptors into activation scales. In other words, the interpretive view reduces the problem to how build and exploit channel-wise statistics that genuinely encode feature importance for scaling. Second, sign-aware recalibration matters: in the affine-activation pipeline, the affine projection provides initial signal of how strongly each input feature should influence the update direction of a given filter, but this signal alone is insufficient. When features with positive and negative affine projections induce gradients of comparable magnitude but opposite effect, their contributions can partially cancel out, making harder for the filter follow an update trajectory dominated by informative features. This motivates sign-aware recalibration step before summarizing channel statistics for scaling.
To probe these intuitions, we first constructed a prototype variant, FleS-Proto. On ImageNet, when very clean, class-aligned channel-importane statistics are available at test time (Tab. [ in Sec. I.I.. simple linear mappings from these statistics activation scales already yield very strong gains. However, when the batch is fully shuffled or the batch size is reduced, these gains shrink markedly or even disappear. This suggests that realistic regimes- where clean and sufficiently rich channel statisties are rarely accessible-reffin central practical challenge. We therefore hypothesize that the mapping from channel-win wise rectified statistics to effective activation scales is generally more complex than single affine transform can capture. In FleS, the small MLP head acts as a lightweight universal approximator that refines these rectified statistics within its receptive field and transforms them into scales that better adapt the
and translating noisy channel-importance cues becomes the
27

---

Published as a conference paper at ICLR 2026
current activation process. Empirically, this design yields consistent accuracy benefit the low-rank linear variant FleS-LRL at essentially the same computational cost (Tab. 9p, supporting the of MLP-based indicator in the final algorithm.
E.7 ABLATION ON NEIGHBORHOOD SIZE FOR OBJECT DETECTION
In the vision instantiation of FleS for dense recognition tasks, a spatial neighhorhood (window) size needs to be chosen to compute the indicator statistics. In practice, select this window size empirically based on the imput resolution and the typical pixel extent of objects. For example, on COCO we adopt a 9 x 15 window: COCO images are typically resized to 800 1333, and small objects usually occupy more than 10 pixels, so 9 15 window can capture relatively clean class- relevant statisties. We then an MLP to aggregate cross to construct the indicators.
cross-window window cues by scanning over the image
Table 10: Ablation study of neighborhood size (realized by window) on COCO object detection using PoolFormer-S12 encoder with RetinaNet.
Activation
Window
mAP (%)1
APto(%)
AP7s(%)1
APs(%)
APx((%)
AP)(
GELU
35.5
55.5
37.5
19.5
38.7
46.3
FleS
x
36.2
57.0
38.1
20.7
40.1
46.8
FleS
9 9
36.1
56.6
37.9
20.5
40.0
46.8
FleS
x
35.8
57.0
37.5
20.0
39.6
46.7
FlcS remains robust when reducing the window from 9 15 to 9, while smaller window leads modest performance drop but still improves over the GELU baseline.
To investigate appropriate neighborhood settings, we vary the window size COCO object detection
with P0 as the encoder and RetinaNet as the detection head. As summarized in Tab. TO] FleS reasonably robust when reducing the window from 9 x 15 to x 9 (mAP 36.2 - 36.1), while smaller 5 5 window leads to modest drop, yet still outperforms the GELU baseline. These results suggest that using a window covering roughly 50 or more pixels is preferable, while still focusing compact local range facilitate the extraction of relatively simple class-relevant
E.8 SCOPE OF NON-LOCAL TENSION ACROSS ARCHITECTURES
On classical CNNs (e.g., ResNet), we observe that the performance gains of FleS are mainly driven by the general benefits of its adaptive scaling mechanism, rather than by non-local tension (NLT) effect of the same strength as in token-mixer-bee based architectures (e.g.. Swin and PoolFormer). In relatively deep blocks, a classical CNN also capture non-local information to extent through its enlarged effective receptive field, but the resulting non-local signals are more diffuse and less explicit than those in token-mixer-based blocks, the opportunities for strongly triggering NLT are comparatively rarer. This picture, where FleS primarily boosts CNN blocks via general adaptive scaling rather than fully developed NLT regime, consistent with our empirical observations.
As shown in Tab, II on ResNet-50 (as a representative classical CNN), two state-of-the-art activation functions, IIBU (Top-1: 79.7%; parameters: 25.6M; FLOPs: 4.2G) and AdaShift (Top-1: 79.9%;
parameters: 25.6M: FLOPs: 4. 4.).), achieve slightly lower accuracy than FleS (Top-1: 80.1%;
parameters: 28. 1M; FLOPs: 4.1G), while using fewer parameters and similar FLOPs. In contrast, on Swin and PoolFormer, FleS exhibits substantially larger margin over all competitive activation methods. This contrast suggests that, for classical CNNs, the main benefit of FleS stems from its adaptive scaling mechanism, whereas the NLT phenomenon is more pronounced in architectures that explicitly model non-local/token mixinn interactions.
Notably, we view the channel winneel-wiseeee indicator design-which models interpretable, lightweight non- local cues as a critical factor underlying the gains FleS on both CNNs and Transformers. To probe this, conduct additional ablation on ResNet-50 where remove the key component implied by our decision decision-making interpretation, namely the monotonic sign-aware recalibration that emphasizes positive and suppresses negative pre-activation responses. Without this component,
28

---

Published as a conference paper at ICLR 2026
Table 11: Investigation of non-local tension in ResNet.
Activation
Backhone
#Params
FLOPs
Top-1 (%)]
ReLU
ResNet-50
25.6M
4.1G
77.2
IIĒÚ
25.6M
4.2G
79.7
ResNet-50
AdaS
25.6M
4.1G
79.9
FleS
28.1M
4.1G
80.1
ResNet-50
FleS-P&N
28.1M
4.1G
79.4
"FleS-P&N" denotes a FleS variant that aggregates positive and negative pre-activations in balanced (symmetric) manner.
the Top-1 accuracy of FleS ResNet-50 drops from 80.1% to 79.4%, making it inferior IIEU and AdaShift. This supports the view that, on classical CNNs without explicit block-wise non- local modeling, the gains mainly arise from FleS's principled adaptive scaling mechanism, with the decision-nee maasiinn-maiiggeoo guided, sign-aware indicator modeling serving as key driver of the observed improvements.
E.9 ON POSITIVE CONSTRAINTS FOR SCALING COEFFICIENTS
We posit that positive constraints decisive for FleS-styl adaptive scaling in self-gated activation, based on our analysis in Sec. as each pre-activation is sign-sensitive importance measure of feature. That is, negative values of Kne and Kho will reverse p(t) and ã. thereby compromising their physical meaning in decision-making
Table 12: Ablation study positive constraints for vertical and horizontal scaling coefficients. "BSL" denotes "Baseline."
Activation
Backhone
Variant
#Params.
FLOPs
Top- I (%)
GELU (BSL)
11.8M
1.6G
68.7
-po
13.0M
1.6G
68.1
FleS Variant
Swin-Min
-PCRAO
13.0M
1.6G
71.1
[CC Mho)
13.0M
1.6G
67.9
FleS
Original
13.0M
1.6G
71.4
We validate this heuristic intuition by comparing the original FleS with ablated FleS variants that omit the positive constraints
(1) the vertical scaling coefficient Kye (denoted as "-po "),
(2) the horizontal scaling coefficient Kho (denoted "-PC "),
and (3) both Kye and Kūo (denoted as "=pC [ho)) "). Based on the comparative results in Tab. T2 observe the following: (1) Omitting the positive constraints
Kee results in significant drop in accuracy. (2) Omitting the positive constraints on Kho leads to a clear decrease in accuracy, but not as severe as (1), because the base function (i.e., Sigmoid) itself provides positive constraint on the overall scaled inputs. (3) Omitting the positive constraints on both Kue and Kho results in the worst performance. These phenomena support our intuition.
E. ON INDEPENDENT USE OF THE VERTICAL/HORIZONTAAL SCALING COEFFICIENT
We model Kee and Kho scaling coefficients for activation functions, responsible for controlling the bounds and steepness, respectively, thus enabling targeted modulation of different aspects of the activation shape.
We compare the original FleS with two ablated variants, omitting (1) the vertical scaling coefficient Kee (denoted as "-Kne") and (2) the horizontal scaling coefficient Kka (denoted as "-Kka" "), to
29

---

Published as a conference paper at ICLR 2026
Table 13: Ablation study on the independent use of vertical or horizontal scaling.
Activation
Backbone
Variant
#Params.
FLOPs
Top- 1
GELU (BSL)
11.8M
1.6G
68.7
Kye
12.6M
1.6G
69.6
FleS Variant
Swin-Min
Kho
12.6M
1.6G
70.8
Roves Kho
12.6M
1.6G
68.9
FleS
Original
13.0M
1.6G
71.4
Note that the FleS variant "- (Koee Khoo" I" is equivalent to SiLU (Elfwing et al. 2018) activation function. "BSL" denotes "Baseline."
examine their individual contributions. As shown in Tab. 13 our key observations are as follows: (1) Both control groups. "-Kue" and -Koo" lead to decrease in accuracy, with "-Kuee demonstrating a more severe decrease than "~ " (2) Despite the performance degradation, both "-Koze and Kho" still enjoy significant improvements over the GELU baseline. These observations support our insights.
E.11 GENERILIZABILITY ACROSS VARIOUS WEIGHTING FUNCTIONS
In this work, we apply the Sigmoid function as the default weighting function Since our assumptions generalize across different forms of self- self-gated functions, we posit that the FieS-style adaptive scaling scheme is applicable to various choices of
Table 14: Ablation study on the generalizability of FleS scaling scheme using different p.
Prototype d (ō)
Activation
Backhone
weighting (#)
Top-1($8)1
( i Kov. (KannT)
SiLU (Ellwing et al. 2018)
68.9
Swin-Min
sigmoid (#)
FleS (Original)
71.4
GELU (Hendrycks et al., 2016) GELU-Fle
/
68.7
Swin-Min
0.5 erf ($/v2)
/
71.2
Mish (Misral 2020
/
68.6
Swin-Min
tanh (softplus (?))
Mish-FleS
/
70.9
TanhGate (Cai 2024a)
68.7
Swin-Min
0.5 (tanh (2) +
TanhGate-Fles
71.1
* Each activation function with the suffix "-FleS" refers a FleS augmented variant, where corresponding p is applied with FleS-style scaling.
To investigate the generalizability of FleS across various weighting functions, we conduct a tailored ablation study. Specifically, we validate four FleS variants, each of which employs different function. Each FleS variant is compared with its baseline counterpart (the reference group), which uses the same p but excludes the FleS-style scaling augmentation. The references and their corresponding MeSSauummented variants include: (1) bascline SiLU (Elfwing al. 2018)) and FleS, using Sigmoid pt (2) baseline GELU (Hendrycks & Gimpel (016)) and GELU-FleS, using ERF- based p; (3) baseline Mish MMisra 2020) and Mish-FleS with
tanh (softplus (:)); and (4) (Cai 2024a and TanhGate-Fles, where
TanhGate (a simple Tanh-based function suggested
(+) = 0.5 (tanh
1) 121+
As reported in Tab. 14 each FleS variant demonstrates significant improvements over its baseline counterpart. This validates the generalizability of FleS's methodology.
30

---

Published as a conference paper at ICLR 2026
E.12 ON POLARITY-SELECTTI INDICATOR: DEFAULT & ALTERNATIVES
In Sec. 4.1 (see Eq. (5) in the main paper and the corresponding clarification), we introduce our default of positive and negative features in modeling the non-local indicator that drives FleS-style scaling scheme. Notably, Eq. (5) (main paper) is motivated by Intuition 4.2 and alternative technical formulations can be used in its place.
method for implementing the discriminative
Table 15: Ablation study on technical choices for implementing selective use of positive and negative features in FleS. "BSL" denotes "Baseline."
Activation
Backbone
Variant
#Params.
FLOPs
Top-1(%))
GELU (BSL)
11.8M
1.6G
68.7
Softplus-based
13.0M
1.6G
71.4
FleS Variant
Swin-Min
Quasi-linee
13.0M
1.6G
71.3
FleS
Original
13.0M
1.6G
71.4
To investigate the modeling of non-local indicator (denoted as x² ) conduct targeted ablation study comparing our default method with two functionally similar variants, used as reference groups:
(1) The Softplus-bud based variant:
softplus (7.ā + BBc) laEXe
(49)
where V/ and Be are learnable factors to dynamically adjust the shape of Softplus function; (2) The quasi-linear variant:
T
if i
T
-1)
(50)
11 exp
if <イ
where 1) is a small threshold (e.g., 0.1 in this ablation study). Note that Eq. (50) defines differentiable function on R.
As reported in Tab. [5] these three variants achieve similar improvements over the baseline, while the default method attains these improvements with relatively minimal technical effort.
E. FROM FLES-PROTO TO PRACTICAL FLES: WHAT BATCH-CLASS COHERENCE REVEALS
Setup. Here we present an extended ablation study that helps clarify our findings on FleS-Prote and justifies the FleS design they inspire.
To this end, we use Category-Bie Blookk Shuffled Evaluation Protocol on ImageNet:
group validation images by ground-truth labels (50 per class);
form each class group into a contiguous block;
randomly permute the order of class blocks;
(4) uniformly shuffle image order within each block. Sequential sampling thus yields evaluation batches that typically contain same-class samples, while the global class order is disrupted.
We denote this as #CClss--Block Block-Shuffle" and vary the evaluation batch size. We also consider "#ootal Batch Shuffle," which mixes classes across all batches.
Observations. (I) When batches remain class-coherent (CBS) and the evaluation batch size is moderate (> 8), FleS-Proto shows stable, strong performance. (2) As the batch size shrinks (e.g.. & 4), FleS-Proto degrades, indicating under-representative bat-h statistics. (3) FleS and GELU insensitive both class-orde and global shuffling, they do not rely inter-sample batch statistics. These phenomena support the view that the drop with FleS- -Proto arises from insufficiently representative statistics in small mixed batches.
31

---

Published as a conference paper at ICLR 2026
Table 16: Effect of evaluation batch size (BS) under the Class-Block Shuffle protocol (ImageNet).
Activation
Backbone
#CB-Shuffle
#TB-Shuffle
#Eval. BS
#Params.
FLOPs
Top-1(%))
256
21.1M
2.6G
85.2
256
21.IM
2.6G
85.2
128
21.IM
2.6G
85,4
64
21.IM
2.6G
85.5
32
21.1M
2.6G
84.9
FIcS-Proto
Swin-Micro
16
21.IM
2.6G
83.1
8
21.M
2.6G
80.1
4
21.IM
2.6G
74.9
21.IM
2.6G
66.7
21.1M
2.6G
54.1
256
21.1M
2.6G
77.3
256
21.IM
2.6G
78.7
GELU (BSL)
Swin-Micro
256
21.1M
2.6G
78.7
21.IM
2.6G
78.7
256
23.5M
2.6G
80.3
FleS
Swin-Micro
256
23.5M
2.6G
80.3
23.5M
2.6G
80.3
"#Class-Block (CB)-Shuffle" indicates whether the block protocol is applied; "#Total- MatthllBachhhh (TB)-Shuffle" shuffles samples globally across batches.
From prototype to practice. FleS-Proto is a proof of concept serving expose modeling cues on how to extract and aggregate class-specific statistics. Guided by these cues, FleS replaces hand- crafted batch statistics with learned transformations that infer robust class-relevant indicators even under shuffled or noisy batches.
On the role of ML.ss in FleS (capturing shared regularities through translation equivariance). Core principle. Effective adaptive activation derlinngdesiinedd mitigate non-local tension that hampers neural selectiitt-shoultee extract, encode, and inject common statistical characteristics among samples under a task-relevant grouping rule (cf. the "reference feature group"). Such shared characteristics obey statistical regularities: the more samples, the more representative the statistics; in image classification, the class identity naturally yields meaningful grouping.
Why MLPs? Generalizing to shuffled batches. In fully or heavily shuffled batches. clean class-specift statisties unavailable; when the number of same-class samples drops to 4, direct aggregation becomes under-represd redeeettpreee due to large intra-class variance in C-dimensional descriptors. In the extreme case of full shuffling, we cannot infer any grouping beyond single image.
Hypothesis. Although descriptors from the same class may differ in the original space, their channel- wise distributions may follow a latent pattern learnable via transformation.
Goal. Learn transformation that maps noisy, sample-level descriptors to meaningful scaling coefficients, recovering shared structure even when class-coherent batches are absent.
Properties needed.
(i) Universal approximability: MLPs approximate continuous functions, making them suitable for mapping noisy statistics to scaling coefficients with compact parameterization.
(ii) Permutation equivariance across tokens and batches: MLPs apply the same pointwise transformation to every token, independent of position. As result, moving token to different position yields the same value for that token; changing the token order merely reorders the outputs one-to-one. This apsition-agnostic tokenwise permutation equivariance makes the layer robust to token and batch ordering.
Instantiation. We (a) extract preliminary descriptors (e.g., token/patch-lewe mean responses per sample); (b) feed them ML. that projects them into shared space where class-commonalities
32

---

Published as a conference paper at ICLR 2026
become salient; (c) use the outputs to guide adaptive activation scaling, restoring discriminative signals lost under batch shuffling.
Beyond shuffled classification: object detection. The design extends to detection, where image contains multiple object classes and background. Semantic objects are typically spatially continuous; for non- non-boundary pixel I,, small neighborhood B(Iz, c) contains same-class pixels. One first compute fine-grained local statistics (e.g., pooled or masked aggregation) and then apply the same MLP transformation to drive adaptive scaling in mixed-class settings.
Takeaway. Batch-class coherence diagnostics reveal why FleS-Proto succeeds under class batches yet drops under heavy shuffling, and how these phenomena motivate FleS: learn transforma- tions that infer robust, class-relevant indicators without relying on inter-sample batch statistics.
E.14 ON ROBUSTNESS TO BATCH SIZE
Setup. We conduct a controlled ablation on ImageNet with the Swin-Micro backbone, varying the global batch size B E (256, 1024, 2048) for FleS while keeping all other training settings fixed; the learning rate scaled approximately linearly with B. GELU baseline B=1024 is included for reference.
Table 17: Ablation of training batch size ImageNet.
Activation
Backbone
#Params.
FLOPs
Batch Size
Top-1(%)
GELU
Swin-Micro
21.1M
2.6G
1024
1x10-3
78.7
256
2.5x10-4
80.1
FleS
Swin-Micro
23.5M
2.6G
1024
1x10
80.3
2048
2x10
80.3
* With LR scaled approximately linearly with batch size by following (Goyal al. 2017) maintains stable Top-1 across a wide range of batch sizes.
FleS
Results. As shown in Tab. 17 FleS exhibits strong stability across an 8x range batch sizes (80.1-80 100.3% Top-1). At test time, FleS is batch-size invariant; evaluating with batch size 1 yields 80.1% Top-1, consistent with the design that avoids inter-sample batch statistics at inference.
E. CHANNEL REDUCTION RATIO OF LIGHTWEIGHT MLP IN FLES
Setup. We analyze the sensitivity of FleS to the channel reduction ratio r, which directly controls the hidden size of the two lightweight MLP heads: H = [D/r] ] for input/output scaling (all other settings fixed). Unless stated otherwise, the backbone is Swin-Miero ImageNet; we set 32 by default.
Table 18: Ablation on the channel reduction ratio r.
Activation
Backbone
#Params.
Ratio
Top-1(%)
GELU
Swin-Micro
21.1M
78.7
22.8M
48
79.7
FleS
Swin-Micro
23.5M
32
80.3
25.9M
16
80.4
Larger MLPs (smaller r) improve accuracy but with diminishing returns relative to parameter growth.
Findings. As r decreases (i.e., the MLP size increases), accuracy improves, but the marginal gain becomes small compared with the considerable additional parameters. Overly small MLPs (r=48) lead to noticeable performance degradation. Overall, 3= strikes a robust balance between accuracy and model size; therefore adopt it the default.
33

---

Published as a conference paper at ICLR 2026
E.16 ROBUSTNESS TO INITIALIZATION
Setup. We assess initialization sensitivity on PoolFormer-S12, training with identical settings while varying the random seed (42, 0, 31415, 2025). GELU baseline (seed 42) is included for reference.
Table 19: Robustness to random initialization.
Activation
Backbone
#Sced
Top-1(%))
GELU
PoolFormer-SI1
42
77.2
42
79.4
79.4
FleS
PoolFormer-SIl
31415
79.3
2025
79.5
FleS demonstrates robustness across seeds and consistently outperforms GELU.
Findings. Across seeds, FleS remains stable (79.3-79.5% Top-1 acc.) and consistently exceeds the GELU baseline by significant margin, indicating robustness to initialization.
F ROBUSTNESS UNDER LONG-TAILE DISTRIBUTIONS
Setup. We evaluate FleS on ImageNet-LT (long-tailed distribution) using PoolFormer-SIl To isolate the intrinsic effect of activation strategies, we do not apply any specialized long-tail techniques deferred re- re-balancing). Training follows common long-tail practice: 300 epochs, 20 warm-up epochs, cosine LR, base LR 5x 10-1, AdamW, weight decay 0.05.
(e.g., re- wwwwhhing/re-sampppaappppe
Table 20: Comparative evaluation on ImageNet-LT.
Activation
Backbone
#params.
FLOPs
Top-1(8)1
GELU
PoolFormer-SI
11.9M
1.8G
37.1
FleS
PoolFormer S12
13.8M
1.8G
40.6
No specialized long-tail tail mitigating techniques applied.
Findings. FleS substantially outperforms GELU under class-imbalance, indicating stronger adapt- ability to long-tailed distributions even without any imbalance-speciffe heuristics.
G IMPLEMENTATION RECIPES FOR IMAGENET EXPERIMENTS
For fair comparisons, (1) we adopt the standard
recipe (Touvron et 202
Liu et al. 2 for Vision Transformers, except for (2) Swin Min, where we reduce the 300-epoch training to 120 epochs (due to time and resource constraints); (3) For ResNets, we adopt the standard
CNN maining-evaluutte recipe (Zhou et al. 2021 Ma et al. 2021. The implementation protocols are detailed as follows:
1. For Swin-Micro (Liu al. 2021), Swin-T T (Liu et al. 2021 , and PoolFormer- $12 Yu et 2022), we adopt the standard data augmentation suggested in (Touvron et al.) 202 1 Liu et al. 2021) and widely used AdamW optimizer (Loshchilov & Hutter] 2019) to train each implemented model with the standard cosine scheduler through 300 epochs (including 20 linear warm-up epochs). The learning rate starts from 10 with an effective batch size of 1024 by default and decays to 1 10 66 smoothly. The weight decay is set to 0.05 and aabol-smoothing of O.I. We follow the common practice to stabilize the model weights by 10 cool-down epochs with the minimum learning rate x 10-6 after the main epochs.
34

---

Published as a conference paper at ICLR 2026
2. For Swin-Min, retain most of the recipes from the above configuration | except that reduce the training epochs from 300 to 120 to shorten the training duration, as we train our model and the competing models from scratch.
3. For ResNets (He et al., 2016), we adopt the common data augmentation strategy (Zhou et al. ) and the standard SGD optimizer train each model for 120 epochs, including
5 linearly increasing warm-up epochs. The learning rate starts at 0.1 with a batch size of 10-5. The momentum and weight decay are set to 0.9 and 10-1 respectively. We follow the common practice to stabilize the model weights by 10 cool-down epochs with the minimum learning rate x 10-5 after the main epochs.
256 and decays
Following the common practice, we (1) train and test all models with an image size of 224 224; (2) report the results of our models and the official results for the baseline methods in terms of Top-1 Accuracy, rounded to one decimal place.
H CONVERGENCE ATTRIBUTE
70
65
Midh
Mch
SMU
5.5
SMU
80
SheFol
StefRled
IEU
DEU
20
355
Mr-Aam
MbAvon
R68
R66
90
120
150
180
210
240
200
300
180 Epoch
210
240
Epoch
Figure 6: The accuracy curve (left) and loss curve (right) of PoolFormer-S12 Yu et al. 2022 backbone with different activation models.
We show the convergence curves of models built on the PoolFormer-S11 backbone (Yu al. 2022 and equipped with FleS and other competing baseline/popullarSSOAA activation functions. Note that each model is trained by the standard 300-epoch recipe (suggested (Touvron et al., 2021 Liu et al.
2021 and introduced in the training configuration I from scratch to convergence.
Fig. illustrates the convergence trends Top-1 accuracy (higher is better) and training loss (lower better) of the PoolFormer-S/ S12(s) models equipped with FleS and other activation functions. The baseline PoolFormer-Sh employs the GELU activation function. FleS achieves the highest Top-I accuracy and the lowest loss across epochs, demonstrating its favorable convergence properties.
I CIFAR-100 CLASSIFICATION
Implementation details. We further compare our FleS with other SOTA/popular activation functions on CIFAR-100 (Krizhevsky) 2009), using CIFAR-Swin-T as the backbone. CIFAR-Swin-T is modified version of the original Swin-T (Liu et al. 2021), which was designed for ImageNet (Deng al. 2009) and downstream tasks. Specifically, we reduce the base embedding dimension from 96 to 24 to prevent redundant parameters, as CIFAR-100 contains far fewer images than ImageNet and has significantly lower image resolution.
To ensure fair comparisons, all models are trained from scratch using the same standard training- evaluation recipe. To construct this recipe, we adopt most of the training protocols and data augmen- tations suggested in (Li et al.) 2019), with slight modifications to fit the Transformer-based backbone. Specifically, each model is trained for 350 epochs with a batch size of 256, by an Adam optimizer with weight decay of 0.05. The learning rate starts from 10 and decreases to 1 10 by following the standard cosine learning rate schedule. All the input images are fixed to the size of 32 32 by following the common practice.
35

---

brute-fort approach to calculate channel indicators Published as a conference paper at ICLR 2026
Experimental results. Comparative results shown in Tab. 21 where FleS improves upon the SOTA and popular competing methods by a significant margin. This observation is consistent with observations on ImageNet, further supporting the adaptability of FleS across datasets of different scales.
Table 21: Comparison of different activation functions on CIFAR-100 benchmark dataset.
Backbone
Method
GELU
SILU
Mish
Pserf
TEC
AdaS
FlrS
Top I(S)
66.7 0..7.10.3
65.8.10.22 1.8
65.7.10..3 18M
66.0.10.3
66.7 7...
67.0.10.2 2.0M
68.9.10.3
CIFAR-Swin-T
#Params.
1.8
1.8M
2.0M
2.0M
Each model is trained times, and the mean and standard deviation of its Top-1 accurncy are reported.
J MS COCO OBJECT DETECTION
Implementation details. In this Appendix, we further validate the versatility and generalizability of our activation function FleS on MS COCO (Lin et al. 2014) object detection. We evaluate FleS by comparing it with a series popular and SOTA self-gated activation functions, including (1) GELU (Hendrycks & Gimpel| 2016) (the most widely used activation function in Transformers), (2) SMU (Biswas et al. 2022b). and (3) Meta-ACON (Ma et al. 2021). Meta-ACON generalizes lightweight channel attention (Hu al. 2020 to perform context-aware dynamic scaling on input features. Thus, it is functionally relevant to FleS and has similar number of additional parameters, but with fundamentally different motivation, philosophy. and methodological insights. We conduct the experiment using the popular PoolFormer-Sh (Yu et al. 2022) backbone and RetinaNet detector (Lin et al. 2017).
For fair comparisons, we train each RetinaNet equipped with different activation models from scratch using the same implementation configuration constructed on the 1x schedule in MMDetection toolbox (Chen et al. 2019). To better suit Transformer layers, we replace the default SGD optimizer the 1x schedule with Adam optimizer. The learning rate starts at x 10 and decays to 1 x 10 gradually. Following common practice, the weight decay is set to 0.05. We report the results using standard evaluation metrics, i.e., mAP as the primary metric for average precision and A APts. APs. APM, AP, as specific APs for different scales. Each PoolFormer-S12 (Yu et al.] 2022) backbone using a different activation function is initialized with its corresponding ImageNet pre-trained weights. To ensure reproducibility, maintain deterministic mode in each implementation.
Table 22: Comparative evaluation MS COCO (Lin al. 2014) object detection. The PoolFormer- S12 Yu et al. 2022)> is applied as the encoder with the popular RetinaNet detector (Lin et al. 2017
Activation
Encoder
m.P (%)1
APoo(%)
APro(%))
APs(%)
AP2(%) APa(
GELU
35.5
55.5
37.5
19.5
38.7
46.3
SMU
35.4
55.2
37.1
20.5
38.6
47.2
PoolFormer-S11
Mt-ACON
35.5
55.7
37.6
19.7
39.1
47.6
FleS (Ours)
36.2
57.0
38.1
20.7
40.1
46.8
Experimental results. The comparative results are reported in Tab. 22] where FleS achieves clear improvements over all the competing popular and SOTA activation models across almost all evaluation metries, especially in APs, which measures performance on challenging small objects. This further validates the generalizability of FleS.
Note that as discussed Section 4.2 our main paper, for dense recognition tasks such as object detection, each image contains multiple semantic classes of objects, which requires computing the channel indicators ø * with finer ranges to mitigate class information confusion (e.g., image patches). Due to time and resouree constraints, for each feature patch of size x 15, and leave the investigation of the optimal patch size for future exploration.
36

---

Published as a conference paper at ICLR 2026
K ELABORATION ON INTUITION BEHIND FLES-NLP & FLES-SEQGATE
For NLP tasks, our design motivated by the fact that token-level semantics are highly context- dependent and can change abruptly along the sequence. In this setting, naïve sequence-level mean tends to wash out local, context-specific cues. Therefore, instead of using global indicator as in the vision setting, FleS-NLP and FleS-SeqGate construct token-level indicators, that each position is modulated by statistics adapted to its own semantic neighborhood; the integration of channel-wise cues is largely delegated to the FleS MLP in this design.
In FleS-SeqGate, to make these indicators expressive yet lightweight, we further use depthwise ID convolution along the sequence as low-cost way for each channel to aggregate information from nearby tokens. This provides more suitable mechanism than a simple mean for estimating context-aware per-channel importance, while keeping the computational overhead small. The design is partially inspired by scan-style state-space models (e.g.. Mamba), but we adopt ID depthwise convolutions as a much cheaper proxy, while preserving the core FleS principle of monotonic, sign-aware recalibration of pre-activations.
More broadly, presenting both FlcS-NLP and FleS-SeqGate serves to illustrate that the decision- making perspective is not only useful for interpreting existing nonlinear mechanisms, but can also guide the design new ones. In particular, viewing nonlinear activations through this lens naturally leads token-level indicators and lightweight sequence-wise aggregation as principled ways to handle context-dependen semantics in NLP, analogous to how attention has inspired a variety of subsequent token-mixing architectures.
L POTENTIAL LIMITATIONS
We note that practical limitation of the current vision instantiation of FleS for dense recognition tasks is that a spatial window size needs to be chosen to compute the indicator statistics. In practice, we select this neighborhood (window) size empirically based on the input resolution and the typical pixel extent of objects. For example, on COCO we adopt 9 15 window, since images are typically resized 800 1333 and small objects usually occupy more than 102 pixels, so such window can capture relatively valid class-relevant statistics. We then use MLPs to integrate cross- cross-window cues by scanning over the image to construct the indicators. This additional design choice slightly increases the deployment burden.
M POTENTIAL EXTENSIONS
As possible direction for future work, we plan to explore FleS variants with improved adaptability, in particular unified and adaptive schemes for extracting valid, fine-grained rectified statistics on pre-activations to construct more effective indicators.
LLM USAGE
ChatGPT was used to aid in polishing the writing. Specifically, it was employed to correct grammar, improve readability, and refine the clarity of sentences.
37


Reviews:
[{"review_id": "tkYg6DEAsm", "reviewer": "Reviewer_rpDy", "summary": "The paper identifies a limitation in self-gated activation, which is argued to be the reason for limited effectiveness of self-gated activation in transformers due to saturation of gating components. This introduces what the authors call convergence limitation specifically in high-importance features wrt a filter where the difference between the importance become negligible and result in the tendency to lose discriminability wrt contributions of features. In transformers, this collapse of gating discriminability causes activation to neutralize contextual cues the architecture tries to capture.\n\nThe authors proposed flexible scaling in self-gated activation using horizontal and vertical dynamic scaling. Horizontal scaling shifts or stretches the gating curve to avoid saturation and vertical scaling increases the discriminability by increasing the range of gating values. The scaling coefficients are conditioned on channel-wise statistics of positively contributing features. In practice, the proposed activation, called FleS, computes per-channel effective responses, feeds them through lightweight MLPs, and outputs the two scaling coeffs. The authors benchmarked the proposed activation across ImageNet, CIFAR-100, long-tailed recognition, COCO detection, and GLUE where FleS consistently outperformed SOTA activations. Particularly notable are results in Swin-Transformer models.", "questions": "1. Can the authors quantify how often real Transformer activations fall into the saturation regime?\n\n2. How do $\\kappa_h$ $\\kappa_o$ evolve during training? Please include training curves of these scalars and variance across layers. \n\n3. Is FleS stable in small-batch regimes? Also, how does FleS interact with batch normalization.\n\n4. The paper gives an intuition but more follow-ups on why to exclude negative values: What happens in architectures where negative values carry semantic meaning? Have the authors visualized the gradient contributions from negative vs positive responses?\n\n5. Is there any other costs other than Flops to be discussed and compared? This becomes important especially when impact of MLP size is also discussed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear identification of an important issue \n\nThe underlying cause of non-local tension problem was clearly discussed, something that previous works have not articulated. \n\n2. The logical framing of the problem, its cause and the proposed approach\n\nThe paper provides an intuitive interpretation of activations as \"importance modulators\". Then clearly identifies the harm of saturation in self-gating activations and draws a logical connection from convergence limitation, trivially discriminative gating weights phenomenon, and non-local tension problem. \n\n3. Practical design of activation with strong empirical results across different models\n\nFleS is simple, seems to be lightweight, and can easily be dropped into modern architectures. Performance improvements especially in Swin-Micro and Swin-T are substantial. The improvements in experiments in Metaformers, CNNs, detection backbones, and long-tailed classification are promising. This broad applicability suggests that the identified problem is real and not confined to a narrow architecture.", "weaknesses": "1. Some theoretical claims rely on partially informal assumptions with more room for quantification/formalization:\n\nAlthough the results generally support the narrative, but some justification or quantification can show if attention-enhanced features regularly fall into the saturation regime. Also, the explanation for why positive-only feature responses should dominate importance is intuitive, but remains heuristic; the decision-theoretic interpretation could be formalized more rigorously.\n\n2. Insufficient analysis of optimization stability and dynamics\n\nThe paper mentions initializing \\gamma values but doesn't quantify sensitivity to initialization. Given that activations can strongly shape optimization trajectories, this lack of investigation is a methodological gap\n\n3. Dependence on channel-level statistics require more investigation/illustration of failure scenarios\nThe batch-dependence in channel statistics brings up questions about microbatch regimes, distributed training, highly-multimodal batches. \n\n4. Discussion of potential failures/limitations or stress-tests:\nIn addition to gains in performance, it's worth discussion more about limitations and scenarios that the proposed activation may fail to be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oxTAi3t8IA", "reviewer": "Reviewer_Ewa3", "summary": "This paper introduces a new gating mechanism. It builds on the intuition that having a fixed gating function can be problematic: If all features are salient, then a classical gating function like GELU will saturate and not meaningfully discriminate between these features. To address this, the authors introduce a new gating function which works as follows:\n\nFirst, for each channel they calculate the mean of the positive features. This is a measure of how likely this feature is going to saturate the activation function. Then, they feed these statistics to two small MLPs. These MLPs provide two scaling factors (of the inputs and the outputs).\n\nThe authors argue that this is particularly important for transformer networks, since the attention mechanism in transformers is likely to lead to having many salient activations within a channel. The reason the authors use an MLP rather than deriving scaling factors directly from the inputs is because it is (1) not possible to derive class-specific statistics at test time when the class is unknown, and (2) in shuffled batches the amount of information per-class can be very noisy. Hence, an MLP is a more appropriate way of estimating appropriate scaling factors given the small amount of noisy information found in a single batch.\n\nThe empirical results in the paper are very encouraging.", "questions": "* Did you run any ablation studies over the MLP? Does a single linear layer suffice or is it essential to have non-linearities in the MLP?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* Strong empirical results\n* A very flexible method that applies to many networks/architectures\n* Robust method that seems relatively insensitive to hyperparameters", "weaknesses": "* A bit heuristic (e.g. using positive-only means)\n* The connection between the theory and the practical algorithm is tenuous (given the use of MLPs in the final algorithm). Although it is interesting to see how the algorithm was motivated by the authors, it does end up feeling a lot like a post-hoc justification. I would prefer it if the authors approach their work as purely experimental and use this space in the paper for more exhaustive empirical validation.\n* Subpar presentation: I found the text needlessly filled with jargon, non-standard terminology and abbreviations, drawing spurious connections to other theories, too densely written, etc.\n  * For example, abbreviating \"activation\" to \"Act\" and referring to pre-activations as \"projected responses\" really doesn't help with readability.\n  * Then there is the list of newly introduced terms and accompanying abbreviations: non-local tension (NLT), convergence limitation (CL), trivially discriminative gating weights (TDGW), etc.\n  * Connections to decision-making and neuronal stimulus-response mechanisms.\n  * For example: Figure 1 should be a clear explanation of the problem this paper tries to tackle, in a way that readers can grok it after just reading the abstract. Instead, readers are presented with the following sentence, which is barely comprehensible (and contains several grammar mistakes): \"[...] the origin of the NLT and the key insights behind FleS shows how CL triggers TDGW problem, which in turn neutralizes the influence of external non-local cues through Act. and show two qualitative insights into addressing non-local tension: vertical and horizontal dynamic scaling strategies.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vPjKkessIu", "reviewer": "Reviewer_HdQ3", "summary": "This paper proposes a novel self-gated activation function, FleS (Flexible Scaling for Self-gated activation), aimed at addressing the Non-Local Tension (NLT) issue in Transformer architectures. From a decision-theoretic perspective, the authors argue that conventional activation functions (e.g., GELU, SiLU) exhibit convergence limitations when handling high-response features, reducing the efficiency of non-local information utilization. FleS adaptively adjusts the activation function’s boundary and steepness via dynamic vertical and horizontal scaling factors (κ_ve, κ_ho). Experiments on ImageNet, CIFAR-100, and COCO benchmarks demonstrate significant performance improvements. The work combines solid theoretical analysis, novel design, and comprehensive experiments, providing a new interpretable perspective for activation function modeling.", "questions": "# **Questions**\n\n1. **Interaction with Normalization Layers:**  \n   FleS does not consider the effects of BN or LN, but these normalization layers are widely used in modern networks. Could LN, which normalizes along the channel dimension, interfere with FleS’s computation of channel-wise statistics?\n\n2. **Scope of Non-Local Tension (NLT):**  \n   FleS is also effective on CNNs such as ResNet. Does this indicate that the NLT problem exists in modern CNNs as well, or is the observed performance improvement mainly due to the general benefits of the adaptive scaling mechanism?\n\n3. **Implementation Details:**  \n   In the COCO experiments, how was the 9×15 neighborhood size chosen? Were other sizes tested? For NLP tasks, why do FleS-NLP and FleS-SeqGate use a *token-level indicator* and *depthwise separable 1D convolution*? Could the authors provide intuition or rationale for these design choices?\n\n4. **Applicability to Large Models:**  \n   Are there any numerical stability or training bottlenecks when applying FleS to large models such as ViT-L, LLaMA, or T5?\n\n5. **Version Consistency:**  \n   The paper presents multiple FleS variants (FleS-Proto, FleS, FleS-NLP, FleS-SeqGate). Is it possible to provide a unified version that supports multiple tasks without task-specific adaptations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "# **Strengths**\n\n1. **Originality and Significance:** The paper identifies and clearly defines a novel and important problem, *Non-Local Tension (NLT)*. It explains why many advanced activation functions that perform well on CNNs fail to provide similar improvements in Transformer architectures. This is not only a technical innovation but also a conceptual contribution, offering a new perspective for understanding and improving activation mechanisms in modern neural networks.\n\n2. **Theoretical Quality:** The paper provides solid theoretical support for both the problem and the proposed solution. The authors construct a clear logical chain: from NLT to *Trivially Discernible Gating Weights (TDGW)*, and then to the root cause, *Convergence Limitation (CL)*. The analysis is rigorous, accompanied by intuitive explanations and formal theorems. FleS is tightly designed around this theory, using *effective average response* (considering only positive responses) to generate scaling factors, which is both a clever and theoretically justified heuristic.\n\n3. **Comprehensive Experimental Validation:** The experiments are thorough and well-designed, covering multiple tasks such as image classification, object detection, and natural language processing. Various network architectures are evaluated, including Swin Transformer, PoolFormer, and ResNet. FleS consistently demonstrates significant performance gains across all settings, strongly supporting its effectiveness, generalization, and robustness.\n\n4. **Clarity:** The paper is well-written, clearly structured, and logically coherent. Each part—from problem formulation, theoretical analysis, method design, to experimental validation—is clearly presented. Figures (e.g., Figures 1 and 2) help readers grasp core concepts and understand FleS’s operational mechanism, making complex theory and methodology accessible.", "weaknesses": "# **Weaknesses**\n\n1. **Lack of a Unified Analysis Framework:** The paper mainly compares FleS with GELU, Meta-ACON, and other methods. Explaining the relationship between these approaches and FleS from a unified theoretical perspective would strengthen the academic rigor and interpretability of the work.\n\n2. **Complexity and Computational Overhead:** The practical version of FleS introduces an additional MLP, increasing the parameter count (e.g., Swin-Min from 11.9M to 13.8M), approximately a 10% increase that is not entirely negligible. The paper does not report inference speed (FPS); it is recommended to quantify the trade-off between computational overhead and performance improvement.\n\n3. **Hyperparameter Sensitivity:** FleS introduces new hyperparameters, such as the MLP channel reduction rate and the neighborhood size (9×15) for computing statistical measures. These settings are somewhat empirical, and further analysis is needed to understand the sensitivity of model performance to these parameters.\n\n4. **Insufficient Elaboration of the Decision-Theoretic Perspective:** Although the paper claims to be inspired by decision theory, the connection between the theoretical principles and the design of FleS remains high-level. Clearly specifying which decision-theoretic principles directly guided the FleS design would strengthen the motivation and theoretical foundation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)