Expressive yet Efficient Feature Expansion with Adaptive Cross-Hadamard Products
Authors: Xuyang Zhang
Xi Zhang
Liang Chen
Hao Shi
Qingshan Guo
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Deep Learning
Neural Architecture Search
Hadamard-based Operators
Feature Expansion and Reuse
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose the Adaptive Cross-Hadamard (ACH) module to enable efficient, non-linear feature expansion via learnable Hadamard products and dynamic softsign normalization.

Abstract: Recent theoretical advances reveal that the Hadamard product induces nonlinear representations and implicit high-dimensional mappings for the field of deep learning, yet their practical deployment in resource-constrained vision models remains largely unexplored. To address this gap, we introduce the Adaptive Cross-Hadamard (ACH) module, a novel operator that embeds learnability through differentiable discrete sampling and dynamic softsign normalization. This facilitates highly efficient feature reuse without incurring additional convolutional parameters, while ensuring stable gradient flow. Integrated into Hadaptive-Net (Hadamard Adaptive Network) via neural architecture search, our approach achieves unprecedented efficiency. Comprehensive experiments demonstrate state-of-the-art accuracy/speed trade-offs on image classification tasks, establishing Hadamard operations as specific building blocks for efficient vision models.

Published as a conference paper at ICLR 2026
EXPRESSIVE YET EFFICIENT FEATURE EXPANSION WITH ADAPTIVE CROSS- PRODUCTS
Xuyang Zhang!? Xi Zhang", Liang Chen" Hao Shil?, Qingshan Guo
'Beijing Institute of Technology. Beijing, China *Chongqing Innovation Center, Beijing Institute of Technology, Chongqing, China
ABSTRACT
Recent theoretical advances reveal that the Hadamard product induces nonlin- car representations and implicit high-dimensional mappings for the field of deep learning, yet their practical deployment in resource-constrained vision mod- els remains largely unexplored. To address this gap, we introduce the Adap- tive Cross-Hadamard (ACH) module, a novel operator that embeds learnabil- ity through differentiable discrete sampling and dynamic softsign normaliza- tion. This facilitates highly efficient feature reuse without incurring additional convolutional parameters, while ensuring stable gradient flow. Integrated into Hadaptive-Net (Hadamard Adaptive Network) via neural architecture search, our approach achieves unprecedented efficiency. Comprehensive experiments demon- strate trae--ff-the-ar accuracy/speed trade-offs on image classification tasks, es- tablishing Hadamard operations as specific building blocks for efficient vision models. The source code is available at lttps:: //gttgituu com/ acelych/ hadaptivenet
INTRODUCTION
Since AlexNet revolutionized computer vision (Krizhevsky et al. 2012), deep convolutional neural networks (CNNs) have advanced rapidly. Subsequent innovations mitigated gradient explosion via residual connections (He et al., (016) and integrated self-attention into vision architectures (Doso- vitskiy et al., 2020), gradually shifting model design toward greater depth for performance gains.
Conversely, lightweight networks (Howard et al.) 2017 Zhang et al. 2018 Ma et al. 2018 Han et al. 2020) pursued efficiency. These models widely adopted the inverted bottleneck structure (e.g., MobileNets (Howard et al. 20177 Sandler et al.) 2018 Howard al.) 2019 Qin et al., 2024), ConvNext (Liu et al., [0022] Woo et al. 2023)), which expands channel dimensions within blocks rather than compressing them. This design enables residual operations in lower- dinee-dimesioonll spaces, reducing computation while mitigating representational redundancy in high dimensions.
However, the inverted residual structure's dependency repeated channel expansion/redduction operators inevitably introduces computational redundancy. Although effective, its dimension ex- pansion phase requires significant convolution operations to project features into high-dimensional spaces, where substantial similarity exists across newly generated channels. GhostNet (Han et al.) 2020 Tang et al., 2002; Liu et al., (204) reveals this critical inefficiency, demonstrating that large portion of expanded channels exhibit high linear correlations, and thus can be inexpensively syn- thesized via learned linear transformations of primary features rather than redundant convolutions. This breakthrough established the first generalized framework for feature reuse, bypassing costly dimension- sppeiffcc operations. Subsequent works like FasterNet (Chen al.. 2023) further refined this paradigm, implementing feature reuse via partial convolution operators that selectively merge spatially neighboring features.
Our work revisits feature recombination efficiency from an orthogonal perspective: instead of gen- crating or filtering features, we exploit the intrinsic nonlinear representational capacity of learnable Hadamard products to achieve ultra-efficient feature fusion. The Hadamard product (a.k.a. element- wise multiplication), as a highly practical method, has long garnered significant attention in the fields
"Corresponding author. (chen1 @bi e.edu.cc)

---

Published as a conference paper at ICLR 2026
ImugeNes- top-1 Accumcc (19)
ImageNer- 1k: Top- Accumey (%%)
Emager
MabeCco
PLOPS (M)
Lntency (ms)
Figure 1: The trade-off between FLOPs/latency and top-1 accuracy. These diagrams compare the efficiency among different sfte-of-the-an models with ours Hadaptive-Nei in image classifica-
tion task. Detailed experimental configurations are provided in section 5.3
of deep learning. Recently, it became a new learning paradigm in the field of lightweight network design owing effective performance and concise computation. Its principle is straightforward, for two identical matrices A, B:
C = A A Ar, Bi,
Recent theoretical advances reveal that stacked Hadamard products can induce nonlinear representa- tions and implicitly high-dimensional mappings when deeply cascaded (Ma al. 2024). Capitaliz- ing on these insights, we propose the Adaptive Cross-Hadamard (ACH) module. This novel operator transcends conventional Hadamard usage by embedding learnability through two key mechanisms: (i) channel attention invention-gudedd guided feature gating, and (ii) differentiable discrete sampling. Thus, ACH es- tablishes Hadamard products as foundational deep learning operators while enabling parameter-free feature reuse.
To effectively deploy the ACH module, we construct Hadaptive- (Hadamard Adaptive Network) through differentiable neural architecture search (NAS). jointly optimizing model topology and ACH integration points. For efficient on-device execution, we further develop tailored GPU acceleration strategies addressing computation scheduling challenges. In comparative experiments. Hadaptive- Net outperforms sfate-of-thear efficient models, achieving higher accuracy with lower computa- tional costs (fig. Ip.
RELATED WORK
This section reviews two types of previous studies related to this work: the application of Hadamard product and efficient model design.
2.1 RESEARCHES IN HADAMARD
be learned from Chrysos al. (2025) that the taxonomy for applying the Hadamard product in deep learning is divided into four categories; high-order interactions, multimodal fusion, adaptive modulation, and efficient operators. Ma al. (2024) and Chen et al. (2022a) reveal its ability to implicitly induce high-order nonlinear mappings. As an example of multimodal fusion, Kim et al. (2077) uses the Hadamard product to achieve low-rank bilinear pooling as an approximation of full bilinear pooling. Adaptive modulation-ale also referred to as the gating mechanism, such as in LSTMs (Hochreiter Schmidhuber 1997) -is a widely adopted application the Hadamard product. For instance, HAda (Wang et al. 2024) employs it to scale weights generated by a hypernetwork in multi-view learning scenarios, while HiRA (Huang et al. (025) applies it to construct high-rank weight updates during the fine-tuning of large language models. MogaNet (Li et al., (024) also uses the Hadamard product to adaptively focus on informative features by fusing multi-scale depthwise separable convolutions with varying dilation rates.
2

---

Published as a conference paper at ICLR 2026
AvgPool
Crozs-Hnlomard Product
(1itt))
M
Manton
BySan
Cumbol
Indicos Outte
Concal
BarrkNerm
BCA
St Sis
PAB Cars (o-wd)
Iickrrand Charael Contitation
X &
Z, C3.+
Yeal
Figure 2: Illustration of the ACH module. Input features X undergo linear transformation and batch normalization. An ECA module generates channel-wise scores, with Gumbel-Topk sampling (training) or top-k selection (inference) determining active channels. Selected features Z undergo cross-Hadamard product, normalized by dynamic softsign, then concatenated with original features.
The forms of efficient operators are quite diverse. To mitigate the O( complexity of Trans- formers. some approaches replace matrix multiplications in attention mechanisms with Hadamard products, as seen FocalNet (Yang et al. 2022) and HorNet (Rao et al. 2022). Gu Dao (2023) Znd[Zhu et al. (2024) adopt the Hadamard product as core operator for ultra-effiiient feature expan- sion and nonlinear fusion via channel-wise cross-products, However, existing methods suffer from critical limitations: fixed combination rules (inter- intra-channel) restrict optimization flexibility, and predefined operations limit interpretability, We therefore propose enhancing Hadamard prod- uets with learnable channel expansion capabilities, transforming them into dedicated deep learning operators that leverage inherent nonlinearity while overcoming previous rigidity.
2.2 EFFICIENT MODEL. DESIGN
The pursuit efficient architectures has driven continuous innovation: from SqueezeNet's pioneer- ing use of pointwise convolutions (Tandola et al.) 2016), to MobileNetVI's depthwise separables (Howard et al. 2017). MobileNetV2's inverted bottlenecks (Sandler et al. 2018), and ShuffleNet's channel shuffling (Zhang et al. 2018 et al. 2018). Neural Architecture Search (NAS) further ad- vanced efficiency in MnasNet (Tan et al. 2019). EfficientNet (Tan & Le 2019). and MobileNetV3 (Howard al. 2019) culminating in Nebils universal inverted bottlenecks (Qin et al. 2024). Concurrently, vision transformers inspired hybrid designs like Mobile-Former (Chen et al, 2022b) and Edge ViT (Pan et 2022).
Feature reuse mechanisms provide complementary efficiency: GhostNet revealed channel-wise re- dundancies in conventional convolutions, replacing redundant features via linear transformations (Han et al. 2020 Tang et al. 2022). FasterNet constrained convolution ranges (Chen et al. 2023), while GhostNetV3 (Liu al.
2024) and MobileOne (Vasu et al. 2022) adopted RepVGG's repa-
rameterization (Ding et al. 2021 to merge parallel branches.
METHODOLOGGY
This section establishes a hierarchical framework for the ACH module, progressing from mathe- matical foundations to architectural deployment. First, we formalize the Hadamard product's 's role in channel expansion. Second, we introduce differentiable discrete sampling via Gumbel-TopK with adaptive temperature annealing, enabling end-to-end channel selection. Third, to stabilize dy- namically generated features, DySoft normalization replaces statistical normalization with bounded sigmoidal activation. Finally, integrate ACH into Hadaptive-Nor through gradient-based NAS.
3

---

Published as a conference paper at ICLR 2026
3.1 HADAMARD FOR CHANNEL EXPANSION
Inspired by the properties of high-dimensional mapping and non-linearity, we observe that the Hadamard product aligns well with the characteristic of neural networks that gradually increase channel dimensions while reducing spatial dimensions. This suggests that the Hadamard product is particularly suitable for channel expansion.
Specifically, we compute the Hadamard product for pairwise combinations of input channels while retaining the original feature maps. This can be expressed as:
Y + [X O X, ((1,)) E (1,2..... C), j}} C(C+1), xxxW
s.t. XE RC Y E IR
where X represents the input feature map. X; and Xj denote the i-th and j-th channels of X, O denotes Hadamard product, and @ denotes channel-wise concatenation, respectively. This approach can be seen as putting the initial features X and the features after transformation into the same feature space. More specifically, the stitched feature vector can be understood high-dimensional vector, and the original feature space can be regarded as a set of bases, providing interpretability for the composite features that earry implicit high-dimensional information.
Based on these insights, we designed the Adaptive Cross- Hadamard module, which is illustrated as fig. The design details and learnable methods of the module will be discussed in the following sections.
3.2 DIFFERENTIABLE DISCRETE SAMPLING
As feature maps propagate through deep networks, their channel dimensions expand dramatically, causing the number of possible channel interactions to grow quadratically. This combinatorial ex- plosion makes exhaustive pairwise computation prohibitively expensive. Even for modest channel counts, practical implementations require selecting a fixed subset of channels for efficient process- ing. We thus reformulate eq. || as:
= X 1Z Zy ((ij) E (1,2,..., (,).,,,,, i1}
CarC -1)
2 +C xHxW
s.t. X XERCXHX YER
s.t. Z= (x+ K
where S represents sequence of chosen channels' indexes and C(s) indicates the amount chosen channels.
However, this selection process is inherently discrete, posing challenge for gradient-based opti-
mization. Thus, introduced Gumbel-Toph trick (Gumbee] (954) for selecting procedure. For- mally, we donate scores of each channels as a vector ç. which is obtain from an ECA module (Wang al. 2020
ç = ECA = P(X) * W +
where denotes adaptive average pooling, denotes a ID convolution operation. Then calculate the probability distribution as below:
{e + Og 7
exp
M.
CEC
Ee + 0c
(4)
Σg exp
s.t.
0, = log(
- -log(u)),
Unif [0,
4

---

Published as a conference paper at ICLR 2026
where o, are i.idd sampled from Gumbel distribution, M denotes probability distribution vector resulted from softmax, and т denotes temperature parameter that controls the smoothness of the softmax output, respectively. The Gumbel-distrbuted perturbations o; inject controlled stochasticity into the discrete selection process, ensuring channels temporarily still receive gradient feedback. This prevents over-reliance initial channel selections while maintaining alignment with the ECA's distribution across forward passes. The temperature parameter governs output sharpness: higher values yield softer selections, while 7 produces one-hot behavior.
While M is continuous and differentiable, it leads to a discrete and nondifferentiable vector M With straight through estimator (STE) technique (Bengio 2013):
M = [s(c), top-k(M,, (() CE[L,C] Oc
forward
(5)
OC
0M
OC ²M
@solmmax(E/T) DE
backward
(6)
0g
DE
where (x) denotes the indicator function, discrete M could conduct data stream during training and gradient could skip through M M during backpropagation. Since hyperparameter mod- ulates the intensity of continuous values influence the selection of discrete values through softmax, Consequently, the adjustment of should be responsive to gradient variations. Instead of relying on a global parameter scheduler, the ACH module employs adapting r dynamically based on the norm of historical gradients, thereby preserving the end-to-ent training characteristics:
CLAMP (r (1 + a sign(llgrad|2 - -hist)), 0.01, 4.0) ||grad|2
Thist
This design specifically addresses layer-wise heterogeneity through dynamic responsiveness: T in- creases when current gradient norms exceed historical values (enhancing exploration for diverse features), while decreasing when gradients diminish (accelerating semantic-specific convergence). Refer appendix A..I the detailed procedure applied during each training epoch. To continuesly maintain gradient propagation, following steps require matrix operations:
M' = 5(c, Ss)
Vs E [1, Col,cc [1,
(8)
MI. = M'.
M
C
(9)
where MI denotes a one-hot mapping matrix from input channels to selected channels, 8(a, b) denotes Kronecker delta function. Given eq. (2) and eq. (9). we can finally obtain Y in eq. with gradient computation graph:
Y = X + Z @ Zj (0.5) E (122....o)).t7
(10)
s.t.
Z = M. X
For inference stage, it directly takes the first few bits of the output of the ECA module and uses this the index to extract the channels that need to be calculated, saving unnecessary calculation.
3.3 DYSOFT NORMALIZATION
The mrdds-Had product creates input-adaptive channel combinations that enhance nonlinearity but produce unstable output distributions. Unlike conventional convolutions that rely on statistical normalization, this dynamic behavior renders batch normalization (loffe & Szegedy 2015) ineffec- tive and risks gradient explosion. Inspired by recent success of activation-baser normalization in Transformers (Zhu al. (202)), we propose DySoft, dynamic softsign normalization that intrinsi- cally bounds outputs while maintaining hardware efficiency:
OLE
W + b
(II)
5

---

from Published as a conference paper at ICLR 2026
Table 1: Comparison of dynamic sigmoidal curves. The experimental conditions are the ex- perimental results of replacing the normalized layers of all cross Hadamard products of the small model finally determined in section 5
Table Neural Architecture Search Confi- dence Distribution. Showing selection confi- dence between Ghost and ACH variants. Un- derlined channels indicate downsampling layers. Fixed layers (no search) marked with hyphens. Values below 0.01% are indicated as <0.01 %. DySig, DyAlge represent dynamic sigmoid and algebraic sigmoid, as the abbreviation DySoft, respectively.
Sigmoid
Softsign
Alge. Sigmoid
Formula
21
I+
V1422
(%)
73.14
73.57
72.80
Channels
Ghost Conf.
ACH Variants Conf.
Performance Comparison of ACH Module Replacement on MobileNetV3, There a total of 1I Inverted Bottleneck modules in the network, with indices starting from O in the table. Several modules were selected for the ab- lation experiment. The first row of the table rep- resents the replaced layer(s), and the second row
Table 2:
DySoft
DySig
DyAlge
32
48
32
64
99.97 %
<0.01%
<0.01%
<0.01%
64
99.99 %
<0.01%
<0.01%
<0.01%
99.87 %
<0.01%
<0.01%
<0.01%
96
99.69 %
0.24%
<0.01%
<0.01%
represents the Top1 accuracy (%). ' denotes
96
99.48% %
0.18%
0.05%
0.23%
the original unmodified MobileNetV3-S, 'IB' de- notes Inverted Bottleneck.
96
98.40% %
0.80%
0.15%
0.51%
96
97.37% %
0.63%
0.63%
1.10%
96
2.92%
62.02%
6.58%
28. 28.18%
128
12.78%
43.72%
23.87%
19.62%
18"
IB'
IB'
1B
1B
128
0.34%
70.05 %
25.34%
4.24%
128
1.68%
34.56%
29.40%
34.30%
70.01
69.74
69.74
69.89
70.38
71.03
71.58
960
where a, w, denote learnable factors of an affine transform. Empirical comparisons table [ show softsign outperforms tanh and algebraic sigmoid variants in stability and computational efficiency, making it ideal for mobile deployment. The indispensability of DySoft discussed in appendix A.2
3.4 HADAMARD ADAPTIVE NETWORK
To systematically validate the efficacy, implementability and architectural compatibility of the pro- posed ACH module, we construct Hadaptive-Net (Hadamard Adaptive Network), a network family that serves as a testbed for ACH module. We employ gradient-based Neural Architecture Search (NAS) (Dong & Yang. 2019) not produce single, static architecture, but principled method- ology to discover the optimal integration of ACH modules within a modern, efficient backbone. This approach allows for its deployment, mitigating the biases of manual heuristic design.
objectively evaluate ACH's performance and unearth general design principles
Our search is informed by a preliminary analysis revealing that ACH is depth-dependent, perform- ing best in late-stage layers (table 2).. We thus designed search space co-integrating ACH with GhostNet-style modules, enabling NAS select the optimal operator per layer. The search re- sults (table 3 confirm our hypothesis: ACH is preferentially selected over Ghost modules in high- dimensional spaces. The finalized Hadaptive-Net architectures are derived from these discovered principles, with full specifications in the appendix A.3
4 IMPLEMENTATION
Prior to additional experimentation, must ensure the crdss-Had product, novel opera- tor, attains its theoretical efficiency on CPU/GPU and other hardware.
4.1 COMPUTATIONAL COMPLEXITY ANALYSIS
The computational complexities for expanding channel dimension with a feature map of f x f m dimensions with & x k convolution are analyzed as below. For inverted bottleneck:
O Omn- (2)
pointwise cony
(12)

---

Published as a conference paper at ICLR 2026
For Ghost module, which partially replaces the expensive pointwise convolution with more effi- cient strategy:
( f pointwwis conv + O((n s)- -k² cheap
(13)
Our method preserves the pointwise convolution while delegating channel expansion to Hadamard product operations:
Omm² fd) pointwise conv O((n m) hadamard
(14)
Since m < n, The computational complexity of Ghost module is reduced to of inverted bot- tleneck convolution, while our ACH module achieves approximately 1 the inverted bottleneck convolution's 's complexity in channel expansion. Derivations shown in appendix A.5] Remark- ably, each Hadamard-derived feature map requires only f² FLOPs, achieving superior efficiency compared to conventional approaches.
The emphasis on FLOPs over latency is driven by the necessity to maintain cross- pooss-platfomm com- patibility and approximate theoretical performance limits. While such a prioritization can be read- ily implemented and validated in serial processing architectures, heterogeneous computing systems present significant challenges that necessitate extensive optimization efforts.
4.2 GPU ACCELERATION
While lower theoretical computational complexity typically suggests faster inference speed, the ac- tual GPU execution involves intricate scheduling by the CPU. The sophisticated channel mapping process in ACH module often gets decomposed into multiple sub-operations by inference frame- works, manifesting as frequent CPU-GPU synchronization and repeated kernel launches. The trian- gular computation pattern of C2 combinations for tross-Hadamare products necessitates specialized operator design, for which we propose two optimization approaches:
1. Direct- Inreet-nndxxng: Each thread block exclusively handles one Hadamard product. The closed-form mapping from pairing index p to matrix indices (i.j) is:
[(2n 1)
(2n 1)2 8p]
2
= - i (2n 1 - i)
2
where denotes number of candidate channels.
2. Parity-Balanced: Assign e thread blocks (c: input channels), evenly distributing irregu- lar computations via iterative indexing algorithm inverse formula:
2 then compute pairing indices with an
i i2n - - 1) +
2
Table 4 demonstrates the acceleration ef- fects of different optimization approaches on the Had-ptive-Net-L. which confirms indispensability of optimization the
Table Acceleration performance.
(Native)
Direct-Indexing
Parity- Bally-Balacced
Latency (ms)
12.40
7.21
7.13
step of implementation. To approach ex- treme performance, we analyzed the performance the two algorithms under different character- istic scales. Details of the experiments and Harity-Balanced algorithm are shown in appendix [A.5] Resulting in practical inference scenarios, it's recommended to employ the parity-balancee approach for gigh-channee/smallHHWV tensors, while considering direct-indexing for spatial dimensions 32x multiples. For gerformance-critic applications, custom compilation of tilling strategies match- ing factors of specific spatial dimensions may be warranted.
EXPERIMENT
This section demonstrates the applied scenarios of proposed method. All experiments ACH module are modified from the configuration image classification experiment. All latency bench- marks conducted within the ONNX Runtime (developers) 2021) framework. To demonstrate

---

Published as a conference paper at ICLR 2026
ECA Malake
P.W.Conv
ECA
Learnable
DySoft
Top-1
69.27
PAW,
ACH
69.12
V
v
71.96
Fixed Selection
73.57
(X. C.l
Remove
(4) Reploce BanchNorm
Figure 3: Component-wise ablation. Illustration of component component-wise wise ablation variations with component-accuree table. (1) and (2) represent removal of pointwise convolution and ECA module, respectively. (3) represents the replacement of learnable selection with fixed channel combinations, and (4) represents the substitution of cross- coss-Haddmard normalization with standard batch normaliza- tion.
Table Replacements of ACH module on ef- ficient models. We replace the last two layers of each model. For instance, replacing last two universal inverted bottleneck modules for Mo- bileNetV4.
Table Performance of Hadaptive-Net on ob- ject detection. We employ the SSD object de- tector different scales of Hadaptive-Net and baseline models with COCO (Lin et al. 2014) dataset.
Model
Top-1 (%)
Params (M)
FLOPs (M)
Backbone
mAP@0.5:0.95
mIOU
MobileNetV3-SS
21.7
71.2
MobileNeiV3-S
70.01
1.61
123
MobileNetV3-S (repl.)
71.58
1.55]
114]
MobileNetV2-1100
21.9
70.5
MobileNeiV4-S
73.15
2.62
385
GhostNetV3-1.0
22.7
72.8
MobileNetV44S (repl.)
72.19L
2.981
3811
Hadaptive-Nere
22.1
72.4
ShuffteNelV2-1.0
65.89
1,36
303
Hadaptive-Net.a
22.9
73.0
ShuffleNelV2-1.0 (repl.)
71.68*
1,28.
291
Hadaptive-Net-L
23.2
73.4
StarNet-S1
71.84
2.68
854
StarNet-S1 (repl.)
72.07
2.561
810]
the practical optimization potential of our proposed ACH module, we implemented it as a custom CUDA operator. We stress that this result is presented showcase the high optimizability of the ACH operator. should not be interpreted as strict, head-to-head speed comparison against base- line models, which utilize the standard, framework-providded operators native ONNX Runtime.
5.1 ABLATION ON ACH MODULE
This experiment evaluates the contribution of each ACH component through controlled ablations:
(1) Whether to keep pointwise convolution layer. (2) Whether to keep ECA module. (3) Learnable selection or fixed combinations. (4) Dynamic softsign or batch normalization. The baseline model of this set experiments obtained from the best model of the previous set of experiments. Fig. 3 illustrates the variations of the ablation experiments with presenting the quantitative results, vealing that all the components serve their respective functions. The pointwise convolution provides fundamental channel-wise information exchange, while the ECA module enables the assessment of channel importance. These two components establish the essential foundation for the module's learnability. Disabling this learnability nearly renders the module ineffective, demonstrating that the discrete differentiation mechanism can properly provide gradients for the former components. The employment of dynamic softsign effectively cireumvents gradient explosion risks, consequently ex- hibiting markedly better performance than batch normalization in experimental trials.
5.2 PLUG-AND-PLAN VERSATILITY OF ACH MODULE
The ACH module's 's distinct mechanism enhances semantic feature representation, making it ideal for standalone integration. We validate this by replacing the final two layers of four oftte-----h-rrr effi-
8

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of efficient models. This table presents parameter counts, computational
complexity (FLOPs), and latency measurements obtained from the CIFAR-100 (Krizhevsky 2009) dataset.
Latency
Top-1
Accuracy
Model
Params
FLOPs
GPU
CPU
Mobile
CIFAR-100
Image Net-le
(M)
(M)
(ms)
(ms)
(ms)
(%)
(%)
MobileNerV3-S (Howard al.
2019
4.98
33.21
70.01
67.42
MobileOne-SO Vasu
2.10
275
3.48
30.70
69.70
71.42
Hadaptive-Net-S
2.10
131
3.41
29.45
4.28
73.57
73,96
ShufflcNetV2-1.0 (Ma al
2018
2.28
146
3.58
40,60
4.55
65.89
69.40
MobileNetV4--S (Qin ar
4024
2.62
185
4,46
24.68
4,31
73.15
73.80
StarNet-S1 [Ma ét al. 2024
2.68
422
6.00
82.73
7.96
71.84
73.50
Hadaptive-Nct-Al (ours)
3.09
339
5.26
39.81
6.47
74.10
78.07
StarNet-S2 Maa al. 2024
3.43
544
7.30
94.50
8.41
67.70
74.78
GhostNet-1.. Han et al. Z020
4.03
140
9.35
87.59
10.02
72.01
73.21
MobileNerV3-1. Hoowr
2019
4.33
215
6.09
56.41
6.66
72.81
75.20
MobileOne-S11 (aasu
2022
4.82
825
7.57
40.76
7.98
72.97
75.90
StarNet-S3 (Ma al. 2024
5.49
754
8,60
112.7
9.87
68.27
77.26
Iadaptive-Net-LL (ours)
6.11
669
7.13
57.62
9.11
74.73
80.79
StarNet-S4 (Ma al. 2024
7.22
1050
9.20
134.0
12.24
68.97
78.12
MobileOne-SZ7 Vasu et al
2022
7.80
1299
9.77
61.19
10.23
73.25
77.41
GhostNetV3-1.0 (Luu (L
al.
2024
8.13
404
13.91
180.54
19.07
73.20
73.92
MobileNetV44MM (Qin
2034
8.42
47.93
9.36
74.66
MobileOne-S3 Vasu 2022
10.15
1896
10.02
81.28
10.36
73.80
78.09
MobileNelV4-L Qui 2024
31.44
2170
10.75
79.94
11.71
74.38
82.92
cient networks: MobileNetV3 (Howard et al. 2019), MobileNetV4 (Qin et al. 2024). ShuffleNetV2 (Ma et al. 2018), and StarNet (Ma et al., 2024). with ACH. As table shows ACH improves accu- racy in all networks except MobileNetV4 while reducing computational costs, confirming its gener- alizability as plug-and-play performance enhancer. See appendix A.6 for deeper analysis.
5.3 IMAGE CLASSIFICATION
We evaluate the performance of Hadaptive-Net on image classification (CIFAR-106 100 (Krizhevsky, 2009), ImageNet-IK (Deng et 2009)), conducting comprehensive comparisons with other state- of-the-art efficient models. Our experiments use PyTorch with Adam optimizer (Ir=0.001, momentum=0.9, weight decay=le-4) and CrossEntropyLoss. Training employs cosine annealing with 5% linear warmup over 200 epochs (batch=64, 224x224 inputs). We conducted the experi- ments both on CIFAR- 100 (Krizhevsky 2009) and ImageNet- (Deng al. 2009). Latency tests use ONNX-converted models (batch=1) 500-run average (Hardware details in appendix A.4).
Result: According to table 7 Hadaptive-Ne achieves superior accuracy in the first two groups while maintaining relatively low computational requirements. Although MobileNetV4 demonstrates the best performance in the largest parameter group, this comes at cost of significantly higher computational overhead.
5.4 OBJECT DETECTION
To validate the generalization capability of HadaptiveNet as backbone network across different downstream tasks, we conduct object detection experiments using the SSD (Liu et al.) 2013) frame- work. All models are trained COCO train2017 (Lin et al. 2014) with fixed input resolution of 320x320 for 120 epochs, employing synchronized SGD optimization (momentum=0.9, weight decay=5e-4) and cosine learning rate decay initialized at 0.02. The detection head utilizes focal loss (? 2200 for classification and smooth LI loss for bounding box regression. Evaluation follows the standard COCO protocol reporting mAP@@ [0.5:0.95] va12017. For implementation details appendix A.4
Result: As shown in table Hadaptive-No Net continues the high-level performance of image classi- fication tasks in the extended task of object detection. This proves that Hadaptive-Ner has a more general feature extraction ability.

---

Published as a conference paper at ICLR 2026
Justification:
Maximizing the efficacy of the ACH module likely requires an end-to-end co-design. We believe the more profound opportunity presented by this study lies in leveraging the principles of structured, lightweight cross-channel interaction embodied by ACH to redesign bottleneck com- ponents like Feature Pyramid Networks (FPNs), focusing on efficiently fusing multi-scale feature information. This represents highly promising direction for breaking the efficiency bottleneck of current detectors.
5.5 GENERALITY ON TRANSFORMER
The Multi Heald--Headdd Self-Attention (MHSA) mechanism Transformer (Vaswani al. 2017) focuses on the N-dimension, i.e., the relationships between tokens, while the Feed-Forward Network (FFN) operates on the C-dimension, integrating semantic information carried and aggregated within indi- vidual tokens. The FFN typically follows classic inverted bottleneck structure, where the ACH module can effectively play role in computational compression.
To maintain research consistency, focusing on computer vision tasks and lightweight design, we have decided to supplement our experiments with improvements on the MobileViT (Mehta Rastegari 2022) model. Specifically, replaced the FFN layers of the middle four Transformer encoders in Mobile with the ACH module.
The comparative results are shown in table 8 which demonstrate that replacing half of the FFN layers with the ACH module yielded significant im- provements. Notably, this en- MobileViT-x-small
Result:
Table 8: Replacements ACH module on Mobile
Model
Params(M)
GFLOPs
Topl-Ace
Top5-Ace
MobileViT-small (Replaced)
4.55
2.879
71.70
92.16
4.40
2.822
72.42
92.20
1.80
1.559
69.98
91.43
hancement was achieved with- (Replaced)
1.74
1.537
70.48
91.71
out increasing the number of pa- MobileViT-xx-smal
0.88
0,588
67.62
90.19
rameters or computational com- (Replaced)
0.85
0.579
67.42
90.32
plexity (FI.Os), leading to su- perior performance on the CIFAR-100 dataset compared to the baseline. This experiment substanti- ates that the ACH module exhibits promising level of generalizability within deep learning, partic- ularly for the role of a channel feature extractor. For natural language processing related attempts, please refer appendix A.6
6 CONCLUSION
This work systematically transforms Hadamard products from auxiliary operations into specific deep learning primitives, culminating in the development the novel Adaptive Cross-Hadamard (ACH) module and its integration into Hadaptive-Net. Theoretical and empirical analyses show ACH's superiority over depthwise separable convolutions in computational efficiency and representational capacity. Lastly, this work establishes Hadamard cient deep learning architectures, offers insights for integrating novel mathematical operations into neural network design.
baasedddd--beddd based operations valuable direction for effi-
REPRODUCIBILITY STATEMENT
For theoretical verification, refer to appendix A.5 for computational complexity analysis. For im- plementation reproducibility, section discusses the whole principle of engineering acceleration algorithm kits. For training details, appendix shows the training configuration of hyperparame- ters and hardware set. The code implement the module and models in this paper has been open source.
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. abs/1607.0644 064500 2016.
CoRR,
1607. 607.06450
10

---

Published as a conference paper at ICLR 2026
Yoshua Bengio. Estimating propagating gradients through stochastic neurons. arXiv preprint arXiv:1305.2982, 2013.
Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo, Song Wen, Chul-Ho Lee, and S.-H. Gary Chan. Run, don't walk: Chasing higher flops for faster neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12021-12031, June 2023.
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European conference on computer vision, pp. 17-33. Springer, 2022a.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengehen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5270-5279, June 2022b.
Grigorios G. Chrysos, Yongtao Wu, Razvan Pascanu, Philip H. S. Torr, and Volkan Cevher. Hadamard product in deep learning: Introduction, advances and challenges. IEEE Trans. Pat- tern Anal. Mach. Intell., 47(8):6531-633 2025. doi: 10. 1000TTPAM..0025556660
URL
httpss///oiin org/ 1109
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
ONNX Runtime developers. Onnx runtime. https://onnxruntingg ai/ 2021. Version: 1.20.1.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/V 186118833311
URL
https:://oi.org///
1423
Xiaohan Ding, X. Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Mak- ing vgg-style convnets great again. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13728 13737, 2021. URL https://pp..emmtiiicco..... semanticscholarr.
org//orrpsII: ID: 221572:3325299
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1761-1770, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752,. 2023. doi: 1044550/ARXIV.2312.0773 URL htttps://doi org/11...... 48550/ 201. 00752
Emil Julius Gumbel. Statistical theory of extreme valuse and some practical applications. Nat. Bur. Standards Appl. Math. Ser. 33, 1954.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 1580-1589, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2026
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 000.//4044..1111. 2017.
Qiushi Huang. Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: parameter-efficiee hadamard high-rank adaptation for large language models. In The Thirteenth International Con- ference on Learning Representations, 2025.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 11mb model size. ArXiv, abs/1602.07360, 2016. URL TRLLhttps:/// /tttpps://pp.... http://ppp..........e.eem. semante icscolarr.....oo..o orrg/
CorpusID: 14136028
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Jin Kim, Kyoung Woon On, Woosang Lim, Jeonghee Kim, Jung-Wor Woo Ha, and Byoung- Tak Zhang. Hadamard product for low-rank bilinear pooling. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. htlltttts:///peeeeeeet
net htte/formmm?oo
rl #1rWWnkkg
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Siyuan Li, Zedong Wang. Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan 7.. Li. Moganet: Multi-order gated aggregation network. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. net/ffoum/ http://oooommdxxxgggggeeee dd=hhWgjgCCVV
OpenReview.net, 2024. httlhttpps//openeeeeee
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/ 10w.0400033,, 2014. URL httppp//eeee http::://axxi..rrt wwtp:iiirrg/aas../oo $405.0312
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: Single Shot MultiBox Detector. arXiv e-prints, art. arXiv:1512.0232. December 2015. doi:
Zhenhua Liu, Zhiwei Hao, Kai Han, Yehui Tang, and Yunhe Wang. Ghostnetv3: Exploring the training strategies for compact models. ArXiv, abs/2404. 11222, 2024. URL
semanticscholast htt.:/ooaw..eeecoorssssa:dd./... 2tt.:oopppuu.1222111775175
Zhuang Liu, Hanzi Mao, Chao-Yusa Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-1 11986, 2022.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Xu Ma. Xiyang Dai, Yue Bai, Yizhou Wang, and Yun Fu. Rewrite the stars. In Proceedings of the IEFF/VVF Conference Computer Vision and Pattern Recognition, pp. 5694-5703, 2024.
12

---

Published as a conference paper at ICLR 2026
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight ceneral-purpost, and mobile- friendly vision transformer. In The Tenth International Conference on Learning Representa- tions, ICLR 2022. Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https:
//openreviess ew. net. formm d DsUt. B
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martínez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In European Conference on Computer Vision, 2022. URL https:
7/api. saaii.smmanicsccool.. ortg/Corpus1::2445577777 2485722122285720000
Danfeng Qin, Chas Leichner, Manolis Delakis, Marco Fornoni, Shixin Luo, Fan Yang, Weijun Wang, Colby Banbury, Chengxi Ye, Berkin Akin, et al. Mobilenetv4: universal models for the mobile ecosystem. In European Conference on Computer Vision, pp. 78 18-96. Springer, 2024.
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu, Hornet: Efficient high-order spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems, 35:10333-10366, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, meeting of SIGDAT, a Special Interest Group of the ACL, pp. 163 164221114.... ACL, 2013. doi: 2L11865/V1/DD13- 117000 ARI.https: /ttp:://doi... wariiorg//ie 1 1ttp1/000001553////1311111
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-awand neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2820-2828, 2019.
Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Chaoting Xu, and Yunhe Wang. Ghostnetv2: attention. ArXiv, abs/2211 12990, 2022. URLhttps:
Enhance cheap operation with http://orpuusD.o 20tprpp0 253801665
//api. semant i cacholas a org/
Pavan Kumar Anasosalu Vasu, James Gregory Gabriel, J. Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7907-7917, 2022.
semanticscholi
Corp//orrssssia.ooo 2578800112
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Éukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net: Ef- ficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Shiye Wang, Changsheng Li, Zeyu Yan, Wanjun Liang, Ye Yuan, and Guoren Wang. Hada: Hyper- adaptive marameter-efficien learning for multi-view convnets. IEFE Transactions on Image Pro- cessing, 2024.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies. NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018 Volume 1 (Long Papers), pp. 1112-1122. As- doi: 10.18653/VI/N18. 1101. URL https:
sociation for Computational Linguistics, 2018.
100 11t.11...115311111m 8
13

---

Published as a conference paper at ICLR 2026
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16133- 16142, 2023.
Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. Advances in Neural Information Processing Systems, 35:4203-4217, 2022.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018.
Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without nor- malization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 14901-14911, June 2025.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. net/ wttp://oommmmmmodoeeeee http://wwww.....ooooooooooooe/r.
OpenReview.net, 2024. URL https:/
A APPENDIX
A.1 TRAINING MECHANISM
T Adjustment: We implement distinct temperature control mechanisms for ACH modules versus NAS due to fundamental differences in their training paradigms. For ACH modules distributed across network layers, which process heterogeneous features and semantics, we deliberately design a adaptive regulation algorithm based on gradient norm trends:
Algorithm Adjustment via Gradient Norm Tracking
Input: Current gradient tensor grad, scaling factor a 0.01 Parameter: Historical gradient norm Thist, current temperature 7
Output: Updated temperature T
1: if Thist * O grad * NULL then
[1 if [|grad]]2 2 Thist
2:
-11 otherwise
3: Tneo f- T. (1 a A)
4:
T + CLAMP( (nnow, 0.01
5: end if
6: if grad + NULL then
7: Thist + ||grad||
8: end if
9: return
For differentiable NAS (GDAS), which operates under fundamentally different optimization con- straints, we retain the GDAS framework's 's global annealing strategy:
Training protocol: Architecture parameters undergo periodic updates separate from model weights, with gradient clipping (|VV < 1.0) ensuring stable convergence.
Temperature scheduler:
Implements predefined decay strategies:
Tmax
(Tmax
Tmin): F
Te
Tiax
Tmin
Tmax
Tiin + 0.5(Tmax [Tax Tmin) [1 cos (I
14

---

Published as a conference paper at ICLR 2026
The three formulas represent linear, exponential, cosine annealing, respectively. Consistent with GDAS methodology, T remains within [0.1,4.0] throughout training.
NAS Specifics: The neural architecture search process employs a dual-optimizer framework with distinct settings for model parameters and architecture parameters. Training executes over 250 epochs with a global batch size of 64, utilizing CUDA acceleration on a single GPU device (ID 0). The primary model optimizer is AdamW with base learning rate 0.003, momentum 0.9, and weight decay le - 4, coupled with CosiaeAneellingll tecture parameters undergo separate optimization via AdamW with specialized learning rate 3e to accommodate their distinct gradient distributions.
scheduling for learning rate decay. Archi-
Cem
simee aatiteee
Table 9: Hadaptive-Net-S architecture details.
Stage
ConvBNAct
lan snndeedderaatt-laade
Layer
Module
Arguments
ConvBNAct
CNA
32, 2, (BN, None
CNA
[32, 48, 2, 2] (BN, HS
Stage
Adap.Betl.. Glost
CNA
[48, 32, 1, 1] (BN, HS]
Adap. Ghost
bandurtidee
AB
[32, 64, 'Ghost', 4.0, 2,
#2
4
AB
[64, 64, 'Ghost', 2.0,
Stage
Dep.Wise Coav
AB
[64, 96, 'Ghost', 4.0,
Adap.Bott. Glost
P.Wise Conv
6
AB
[96, 96, 'Hada', 16, 1]
AB
[96. 96, 'Hada', 16,
Meet.B Hada
AB
[96. 96, 'Ghost', 2.0,5,
Adag Batt Mluda
Medule
AB
196, 96, 'Hada', 16,
10
AB
196, 96, 'Hada', 16, 16,
Stage 4
DepeWise Conv
[96, 128. 'Ghost', 6.0, 2,
Adap.Bott. Gibstt
12
AB
[128, 128, 'Hada', 32,
P.Wlse Conv
13
AB
[128, 128, 'Hada', 32, 7, [128, 960, 1, 1] (BN, HS} [960, 100, 1280, 0.3]
Adaa..Bott.
14
CNA
15
FN
Head
Figure 4: Hadaptive-Ne architecture overview.
A.2 DYSOFT INDISPENSABILITY
Quantitative experiments can not explain the indispensability of DySoft since the model without DySoft training is extremely unstable and has no representative experimental data. However, these phenomena can illustrate a problem that DySoft is empirically necessary, which could explained by probability theory.
Problem Tracing: Traditional normalization methods, such as BatchNorm [Ifffe Szegedy 2015) and LayerNorm (Ba et al.) 2016), have priori assumption that the statistical mean and statistical variance of the tensors they receive are knowable and traceable, which constitutes the basis of model convergence. In the process of ACH training and reasoning, will involve standard 7,; ④ Z,j cross Hadamard product calculation. In previous machine learning methods, the use of Hadamard product usually self referential, that Z Z 3 Z. In this case, can easily infer the mean value of Z Z O from the mean and variance o² of Z
Var 2 = B[ - -1) = -E|))) E = μt +
Since tensor Z was processed by normalization from above layer, which approximately satisfies 7 N(n, According the fourth moment formula of normal distribution:
15

---

Published as a conference paper at ICLR 2026
E Var Z²)
1² + 6u²0 2 30
E|Z4| - (E[Z2))2
(nt + 6 + 300t) (1² + 02Q2
(2p +
If the self referring Hadamard product is deformed, for example d1 (Z) @ dh (Z), Let here be linear transformation operator, the corresponding matrix form X1, X2 (X E [mxx), bias vectors b,, b2 (b I,,, then:
Σ
X1.5 + B(b]
B[ø(2)) = - +0 = m
m
For variance, since Z can approximate normal distribution, here we assume that its elements are i..d, then there are:
1 m
A.j 0² 0². |
Var( (Z))
Σ Var Var( (a(()))))))))))))))))))))))))(
Suppose @ is nonlinear transformation operator, which does directly exist the predictability of analytical solutions. However, the purpose of normalization method is not to accurately track the statistical representation tensors, but to ensure that the statistical representation of tensors remains stable in the reasoning process.
Let the mapping Ty R. x R550 +R X be: (2, + (p',o'²). If unbounded, that is, there a sequence (ux, such that [ ofll - oc as long as layer accidentally reaches the state (such as disturbance. initialization deviation), the next layer of statistics will be unstable; If Ty is discontinuous the derivative is unbounded (e.g. f(z) 1.>0 at μ = 0), small disturbance can lead to n d² upheaval, resulting in unstable training.
BatchNorm is generally considered CV tasks. BN independently estimates the mean and variance of l: for each channel:
A = [x
o = Var=~5(TR)
And perform channel by channel affine transformation:
2/4
7
+ 8p
o
+e
This operation does not force statistical consistency between channels, but allows or even encourages significant statistical heterogeneity between channels:
ii s.t.
p4
+
0² + 0²
This property is consistent with the inductive bias of "channel division" in convolutional networks different channels can professionally respond different semantic patterns (such as edge, texture, color), which is the key basis for its high representation efficiency in visual tasks. In contrast, LN is normalized in the sample dimension:
x' xzy. +
1 == C Σ Ik. σ²
Σ
The implicit priori is that all channels at the same spatial location should have the same statistical scale, which drives statistical convergence between channels. This assumption is compatible with the inductive bias of "all tokens comparable" in the global attention mechanism (such ViT),
16

---

Published as a conference paper at ICLR 2026
but in CNN dominated by local receptive fields, it will weaken the channel specific characterization ability and lead to performance degradation.
Let us consider Vy x; @ zj, its output statisties depend on the joint second moment of the imput channel. Under the heterogeneity distribution induced by BN, let z; ~ NMs,, of), 2j NN,,, 03) and i..d, then:
E [yy] Paaj = 16,03
11307 { +
oo,o3
When the channel statistics differ significantly (e.g. l >> |0yl or >>) oÿ), the variance shows a multiplicative amplification effect, which is far beyond the single channel scale range. The affine parameters of BN are only channel specific, which can not effectively correct the new statistical off- set caused by such cross-channel coupling. Otherwise, the pairing process of i. j is obtained by the nonlinear transformation of each input, which makes it impossible for the statistical representation iterative map T'y to find the fixed point.
Although LN normalization may be used inside ACH module, it is very important to understand the heterogeneity between channels in CV tasks. There is usually typical CNN trunk containing BN upstream of the module, so the whole feature learning process has been dominated by the hetero- geneity of BN priori. The model's understanding of image semanties will evolve spontaneously towards the direction of "channel specialization". At this time, a strong cross-channel nonlinear module with implicit homogeneity assumption is inserted into the reasoning chain, it will lead to priori conflict.
Solution: The DySoft introduced is essentially variant of the softsign activation function:
OCE
C.C
y
+ b,
lim
+1
larl
+ a22
Due to the boundedness of softsign, no matter how large the input variance o² is, the output variance is rigidly limited in the (0, 1) range: When the input is small, it shows approximate linearity and maintains the characteristics of the signal. The parameter a can dynamically balance the expression and compression of the layer.
When the cross-Hadamard Vij = xy @ Ij has variance like 0g >>> 1, Var(U:)) increased by (aat). After accessing DySoft, this trend can significantly compressed and given boundedness. In addition, DySoft is also designed based on the hypothesis of channel heterogeneity, which is a priori compatible with the heterogeneity of BN. Its w. parameters are channel specific, and can independently learn the scale and offset for each Praas--a product channel. At the same time, does not destroy the channel professional representation established by the upstream BN, and only makes local intervention on the "danger signal", thus realizing the organic unity of characterization and stability.
In summary, DySoft learnable statistical compression gating (SCG) module, which achieves hard variance clamping for high square error input through bounded nonlinear mapping S(u) = u/(1 + [u]) and restores the characterization capacity in combination with channel specific affine transformation. Without violating the heterogeneity prior of batch normalization, the design ef- fectively inhibits the growth of multiplicative variance caused by cross channel nonlinearity (such as cross Hadamard product), and makes the statistical map T (1, r²) - (n',a") bounded and smooth, so as to meet the core condition of "knowability",, providing stable and convergent statis- tical target for the normalization layer.
A.3 HADAPTIVE- CONFIGURATION
This section mainly shows the results of three groups of NAS experiments and the decision of final Hadaptive-Net structure. See tables 10 to 12 for NAS experiments details.
Hadaptive-Net adopts a hierarchical backbone architecture comprising stem followed by four dis- tinct stages, as shown in fig. 4 To implement Ghost and ACH module with adaptability, we design the Adaptive Bottleneck that can decide the expansion layer of the bottleneck manually. The net-
17

---

Published as a conference paper at ICLR 2026
Table 10: Neural Architecture Search Result (a). Compared with different kernel sizes. Reaching 67.55% topl-acc as result.
Channels
Ghost Conf.
ACH Conf.
Blank
3
3
32
4X
32
64
54%
46%
24%
76%
96
52%
48%
96
27%
14%
9%
50%
25%
21%
28%
26%
96
20%
17%
23%
17%
23%
96
19%
19%
23%
17%
23%
20%
20%
20%
18%
22%
128
37%
28%
17%
17%
128
35%
33%
33%
128
60%
27%
13%
128
1%
15
1%
96%
1
15
96%
128
1%
2%
1%
96%
960
Table 11: Neural Architecture Search Re- sult (c). Shows the distribution of ACH con- figurations across different channel sizes. Val- ues represent percentage confidence (rounded to nearest integer).
Table 12: Neural Architecture Search Result (b). Shows the distribution of ACH configura- tions across different channel sizes. Values rep- resent percentage confidence (rounded near- est integer). indicates layers that were not searched. Since training ACH modules requires lot of iterations to be effective, the network tends to skip them during training. Reaching 66.23% topl lopl-acc result.
indicates layers that were not searched. Format: ACH-[chosen dim]- [sienelssize. Reaching 67.73% topl lapl-acc as re-
sult.
Channels
ACH Conf.
Channels
ACH Conf.
Blank
16-3
16-5
32-3
48-3
16
32
48
32
48
32
32
48
64
64
96
68%
12%
20%
96
44%
13%
43%
14%
9%
68%
96
47%
16%
37%
96
14%
13%
65%
96
35%
20%
45%
96
16%
13%
62%
96
45%
15%
39%
96
13%
15%
61%
128
96
13%
14%
60%
128
19%
52%
29%
128
128
28%
32%
40%
128
1%
19
1%
98%
128
36%
41%
22%
128
1%
1%
98%
128
40%
30%
30%
128
1%
1%
1%
98%
51%
23%
26%
1%
98%
960
128
1%
1%
1%
98%
960
work begins with linear convolutional layer as the stem, followed by fixed two conventional con- volutional layers in Stage for initial feature extraction. Stage 2 incorporates two fixed Adaptive Bottlenecks utilizing Ghost module expansion layers, enabling rapid downsampling. Stages and employ Ghost Ada.Bott. for downsampling layers and Hadamard Ada.Bot for repeated resid- ual blocks, with particular emphasis parameter concentration in Stage 3, following ConvNeXt's
18

---

Published as a conference paper at ICLR 2026
design philosophy. The kernel sizes progressively increase across stages, with non-downsampling layers configured as x 1, x 3, x 5, and respectively.
Refer to table 9 for detailed description of layer level architecture configuration. CNA denotes combinition of convolution, normalization and activation layers. AB denotes adaptive bottleneck, which could be subdivided into Ghost module ACH (Adaptive Cross-Hadamard) module. Hada denotes the ACH module. BN denotes batch normalization. HS denotes hardswish activation. FN denotes full connection layer. All first arguments represent input/output channel. All last two arguments represents kernel size and stride size, respectively.
A.4 EXPERIMENTS DETAILS
The following is a detailed description of the experimental part of this paper.
Hardware Configurations: Latency tests conducted on:
Desktop GPU: NVIDIA RTX TITAN (24GB GDDR6, CUDA 11.6)
Server CPU: Intel Xeon Gold 5218 (2.3GHz, 16C/32T)
Mobile SoC: Qualcomm Snapdragon 870 (4xConex-AA7722.GHH. 1xA77@3.2GHz,
Adreno 650)
All tests used ONNX Runtime 1.16.0 with default execution providers.
Object Detection Training Protocol:
The base learning rate 0.02 corresponds a batch size of 64 distributed across 5 GPUs, scaled linearly according to the batch size. We apply 3-epoch linear warmup and reduce the learning rate le-5 via cosine scheduling. Data augmentation includes random HSV color jittering with hue delta limited to 18 degrees and saturation scaling between 0.5-1.5, followed by random canvas expansion up to 2x original size and loU-based cropping with thresholds sampled from [0.1,,.3.0.5).
Object Detection Architecture Specifications The SSD detector generates 6 default boxes per feature map location with aspect ratios spanning [1:1, 1:2, 1:3, 3:1]. Feature maps are extracted from five backbone stages with strides of [8, 16,32,64, 128] pixels respectively, corresponding to spatial dimensions from 38x38 down to 1x1. During focal loss computation we set the o-balancing parameter 0.25 after empirical validation across the range [0.1,0.5].
A.5 IMPLEMENTATION DETAILS
This section will supplement the derivation of previous computational complexity analysis and im- plementation details of GPU acceleration algorithm mentioned in the original text.
we can derive the ratio of Ghost Module complexity to that of the standard pointwise convolution:
With eqs.
m 8 f2 (n 8) n f2
8 71
122
RatiOGhou
(15)
Since is often chosen as fraction of n (c.g., s = n22),, the term n=8 is approximately a constant (c.g., 1/2). Since A² is small (c.g., 9 for 3 x kernel) and m can be relatively large, the term is often negligible. Under the condition m & n, the simplified complexity ratio is:
&a
RatioGhost
(16)
n
Similarly, we calculate the efficiency ratio by comparing ACH module complexity to the standard pointwise convolution:
19

---

Published as a conference paper at ICLR 2026
Giga- FLOOPs va. Input Channels (Ci) (Ration4)
Gigz MACs vs. Input Channels (C) (Ratiom4)
Tube Tye
Conpuaiooll Compleii, (Gip-
Coryleedty
Chonnes
Impar Clanads
Figure 5: Comparison of computational efficiency under different input channel sizes
Parts
Figure 6: Comparison of computational efficiency under different expansion ratios
RatioACH m= f2 (n m) m. n. f²
f2 m²² + no
1
(17)
771 7
77
171
11
Given the constraint n, < considered negligible. Thus, the simplified complexity ratio for the ACH module is:
the terms and 1 become very small and can be
m
RatiOACH
(18)
For this theoretical analysis, have carried out several groups of measured data different input channel sizes and amplification ratios to confirm that ACH module has very strong reasoning speed compared with the standard point-by-point convolution and ghost module.
The experiment carried out for two specific situations: fixed 4-fold scaling ratio, 16-512 different input channel sizes; Fixed 64 input channel size, 4-24 times scaling ratio (224*224 per frame). We counted the Multiply-Aee Coltiply-ACCumlaan ACCumulate Operations (MACs) and Floating Point Operations (FLOPs) of
two groups of experiments as illustrated figs. 5 andd
This ratio can be shown intuitively in the above experimental results. From the experimental results, the computational complexity ratio under different channel sizes is relatively fixed, while different scaling ratios, which is m/n, show a linear relationship.
Previous section 4.2 presented two GPU acceleration algorithms for ACH operators. One algorithm.
the Direct-Indexing, is implemented as the name suggests. Another algorithm, the could be written as algorithm?
To systematically evaluate these methods under varying tensor configurations (batch/channel di- mensions versus spatial sizes), we conducted comparative experiments using square matrices (same sized height & width). See fig. for the experiment details and results.
Both algorithms demonstrate relatively stable performance across varying batch sizes, indicating comparable parallelism in aagosti-egnostie scenarios. Despite both expanding dimensional- ity, the -blaaee approach exhibits superior optimization for high-channel tasks compared to hith-batch scenarios, owing its specialized load balancing for channel-dense tensors.
20

---

Published as a conference paper at ICLR 2026
size
size
61
channel
batch
41-
64
66
(b)
Figure 7: Normalized difference heatmap of optimization approaches runtime. Color-coded vi- sualization of relative performance between direct- -ideexin (A) and parity- (B) approaches using A-B A+ where red indicates A is slower (B more efficient) and blue indicates the opposite. (a) Batch size versus spatial dimensions scaling. (b) Channel count versus spatial dimensions scaling.
For feature maps with smaller spatial dimensions, the parity- ballnced approach significantly outper- forms sirret-indexing due to: (1) The balanced approach's input tensor reuse pattern enhances L1/L2 cache hit rates in GPU global memory, reducing memory access latency while increasing arithmetic intensity per thread block through reduced thread block maintenance. (2) While appearing to intro- duce serialization, the balanced method effectively concentrates inevitable serial processes within individual thread blocks, as GPU core counts cannot simultancously satisfy all computational de- mands for dimensionally dense small tensors, therehy avoiding sontext-switching overhead. (3) Direct-indexing requires separate thread block allocation per matrix computation, leading to under- utilized warp resources when small matrices cannot fill the thread block size.
When spatial dimensions approach integer multiples of 32 (thread block dimension), direct-indexing prevails due thread blocks achieve near-saturation load conditions with peak artificial intensity, and the method's d's end-to-end processing better aligns with hardware scheduling optimizations.
Algorithm Parity-Balanced Indexing Strategy
Input: Channel count
Parameter:
Thread block group ID id
Output: Choosen channels j,
1: for it + 0 to c 1 do
2:
if it id A -(id it)) mod 2) then
3:
i < j +- id
4:
else if it > id A (id - it) mod 2 then
5:
i +- id,j +- it
6:
else
7:
continue end if
8:
9:
Compute Hadamard product for matrices and
10: end for
A.6 EXTENDED EXPERIMENTS
Grad-CAM: To further elucidate the role of the ACH module in enhancing the model's represen- tational capacity, designed two sets of comparative experiments using Grad-CAM visualization
21

---

Published as a conference paper at ICLR 2026
Convolation
GhossModale
Original
ACH Module
Hadaptive-Ne -Net
ACI Modale
GhostModull
ACH Module
GhusiMocule
ACH Module
Convolution
Convolution
GbostModule
Original
Baseline
Convolution
GhostModile
Figure 8: Network visualization via Grad-CAM across layers (1). Simple scenario: ladybug.
Downward arrows denote downsampling layers.
Convolution
Ghosi
Original
ACH Module
-ett Net
Hadaptive-
ACH Modale
GhossModdle
GhossModull
Convolution
ACH Module
ACH Module
Convolution
GhostModule
Original
Baseline
GhosMooulee
Figure 9: Network visualization via Grad-CAM across layers (2). Complex scenario: mushroom. Downward arrows denote downsampling layers.
to examine the changes brought by the ACH module compared to a conventional convolutional net- work. For clearer and more intuitive comparison, adopted the baseline modified version of Hadaptive-Net-S in which all ACH modules were replaced with Ghost modules, in order to demon- strate the feature extraction pattern under purely linear transformations.
22

---

Published as a conference paper at ICLR 2026
Table 13: Replacements of ACH module on BERT.
Layers
Modify
In-Out-Channs
Mid-Channel
SST-2
MNLI-10%
BERT-6
FFN (fixed channel)
764
764
64.7
36.7
BERT-6
ACH Module
764
2043
76.2
41.8
BERT-6
FFN
764
2048
80.3
46.7
BERT-12
FFN (fixed channel)
764
764
53.1
34.3
BERT-12
ACH Module
764
2043
75.9
BERT-12
FFN
764
2048
80.6
45.5
The first experiment, which is shown fig. involves a simple scenario, where ladybug is clearly distinguishable from the background. The baseline model exhibits standard processing pattern that progresses from texture analysis to focal emphasis. In contrast, Hadaptive- not only extracts tex- ture more accurately, but also achieves target focus with fewer layers, while performing more precise edge segmentation. After the final downsampling step, the baseline model continues attempting to focus on the main subject, whereas Hadaptive-Nee begins to attend to the edges of withered leaves, suggesting an attempt capture higher-level semantic correlations.
The second experiment, which is shown as fig. 9 presents a more complex situation, where a mush- room exhibits some color overlap with the background. Compared to the baseline, Hadaptive-Net transitions more rapidly from the edge extraction phase to the target focusing phase, and explores larger spatial area, indicating larger effective receptive field.
In summary, the introduction of the ACH module not only reduces computational complexity but also endows the model with more powerful semantic representation capabilities.
We tried to extend ACH module to NLP. We conducted comparative test on the 6-layers and 12-layers BERT (Devlin et al. 2019). The accuracy of the BERT model using differ- ent channel feature extractors (FFN, ACH module and standard FFN with unchanged middle layer dimension) was tested SST-2 (Socher et al. 2013) and 10% MNLI datasets (Williams et al., 2018).
NLP Attempt:
The models were evaluated on two standard natural language understanding benchmarks: the Stan- ford Sentiment Treebank (SST-2) for binary sentiment classification and Multi-Genre Natural Language Inference (MNLI) dataset for textual entailment. For SST-2, the model trained and evaluated on the full dataset. assess performance in low-rcsource setting, the model was trained 10% stratified subset of the MNLI training set and evaluated the full matched validation set.
The training configuration was consistent across both tasks. Models were trained for 3 epochs with global batch size of 32 and evaluated with batch size of 64. The optimization used learning rate of 2e-5 with a linear warmup over the first 10% of the training steps and weight decay of 0.01. The models, which followed BERT-base architecture (12 layers, 12 attention heads, 768-dimensional hidden states), were initialized with random weights. Input sequences were tokenized using the barttbbse-uncasedd tokenizer with maximum length of 128 tokens. The sole evaluation metric was classification accuracy, calculated as the percentage of correctly predicted labels against the ground truth. All experiments were run on single GPU without mixed- precissin training.
As a result in table [3] the performance of ACH module was not stunning enough to exceed the BERT baseline. However, from the perspective of the motivation compressing the calculation scale, ACH module still plays a big role. Only adding small amount of Hadamad product in one step can approach the FFN without channel depth to hetter level, revealing its potential as unique algorithm in the field of NLP.
As we are mainly engaged in CV related work and lack relevant experience NLP field, the ex- perimental setup may be little rough. If there are any problems, readers are welcome to correct them.
B THE USE OF LARGE LANGUAGE MODELS
In the preparation of this work, the author(s) utilized a Large Language Model (LLM) primarily to aid in polishing and refining the writing. The tool was used for purposes such improving gram-
23

---

Published as a conference paper at ICLR 2026
matical correctness, enhancing sentence fluency, and rephrasing for clarity. All ideas, theoretical analyses, experimental designs, results, and conclusions remain entirely those of the author(s). The final manuscript has been thoroughly reviewed and edited by the author(s), who take full responsi- bility for all content presented herein.
24


Reviews:
[{"review_id": "US6khrpXkg", "reviewer": "Reviewer_qj3w", "summary": "The authors introduce the Adaptive Cross-Hadamard (ACH) module, utilizing channel attention-guided feature gating and differentiable discrete sampling along with Hadamard (element wise). This module, when stacked can efficiently and effectively model nonlinear representations and high-dimensional relationships. The authors develop a novel and effective training methodology utilizing the Gumbel TopK trick to limit the compute of the model across channels as well as a novel hardware-efficient normalization scheme due to the nature and requirements of this new model. The authors use architecture search to demonstrate that this module is preferred over existing methods and demonstrate its computational efficiency. Finally, especially at small scale, the authors Hadaptive-Net models clearly outperform competing, less efficient models on top1 accuracy for the CIFAR-100 and ImageNet1K datasets. As a result, the authors have obtained a more efficient, more effective model.", "questions": "Please provide a clear and concise mathematical description of the algorithm, preferably with descriptions of the input and output dimensions of the individual operators. More precise definitions and clear descriptions of the algorithm will make this paper much stronger.\n\n Additionally, do the authors believe this method would be useful for tasks beyond classification? If so, please provide some justification or evidence.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Through the introduction of the Adaptive Cross-Hadamard (ACH) module these results develop a novel method for training pairwise Hadamard products with very few parameters and utilize Hadamard operations as building blocks for efficient vision models, which is relatively unexplored. Moreover the authors' module offers parameter-free feature reuse, reducing computational cost. The ACH module has strong accuracy/speed trade-offs, appealing for real-world and edge deployment and a hardware friendly implementation. Finally, in evaluation the ACH based classification models offer state of the art accuracy/speed tradeoffs.", "weaknesses": "I think it would be very helpful to have a precise description of the end to end method listed as an algorithm at some point. In particular, equation 10 is somewhat ambiguous to me. It is not very clear how the tensor X is added to the cross Hadamard product (e.g. what is the $\\oplus$ operator doing?) It is also  somewhat confusing at first blush what components of the model are trainable and which aren't. Finally the dysoft operator is somewhat ambiguously applied to the output of the pairwise Hadamard computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s2jtDBAjR1", "reviewer": "Reviewer_ZaiK", "summary": "This submission aims to achieve efficient and expressive feature recombination for lightweight vision models. It proposes Adaptive Cross-Hadamard (ACH) module, which leverages (i) learnable Hadamard (element-wise) products, (ii) discrete channel sampling via Gumbel-TopK, and (iii) dynamic softsign normalization for efficient feature expansion and recombination within CNNs. The optimal placement of ACH blocks alongside standard GhostNet-style linear expansion blocks is is discovered via differentiable neural architecture search to build the Hadaptive-Net.\n\nExperiments are conducted on image classification (CIFAR-100, ImageNet-1K) and object detection (COCO) tasks. The results show Hadaptive-Net’s competitive accuracy-latency trade-offs compared to some baselines. Empirical analysis and theoretical complexity comparison such as ablation studies, architecture search, Grad-CAM visualizations are also conducted. Practically, it also presents engineering optimizations for GPU execution.", "questions": "Most of my major concerns and related recommendations have been stated in the Weaknesses section. I encourage the authors to focus their efforts on addressing those points, as they are critical for strengthening the manuscript in the rebuttal stage.\n\nThe following are more specific, minor questions to help the authors think more deeply about certain design choices and experiment setups, which I hope might be helpful for this and future work:\n\n**(Q1)** Could the authors provide theoretical analysis or insights into why dySoft normalization outperforms others in the ACH setting? Perhaps via variance/convergence analysis or more direct connections to gradient stability?\n\n**(Q2)** Is there empirical evidence that ACH modules offer gains outside vision, such as language, audio? How do they perform in settings with heavy distribution shift? IMHO, such explorations may get closer to the heart of the question.\n\n**(Q3)** It seems that the NAS leaves most early layers as Ghost modules and prefers ACH only at high dimensions. Do the authors see this as a limitation or a natural consequence of Hadamard product behavior? Is hybridization always preferable to pure ACH insertion? I believe this worths deeper investigation.\n\n\n---\n\n## Justification:\n\nThis submission presents strengths in motivation, technical soundness, thorough experiments, and practical engineering contributions. However, several concerns exist such as the lack of clarification for some technical details, limited literature review, and more. Overall, I am leaning towards acceptance and first give a rating of 6. I would be glad to raise my rating if thoughtful responses and improvements are provided. Conversely, if most of the concerns remain unaddressed, I may also lower my score. I am also open to follow-up discussions with the authors to help further strengthen this work.\n\nI hope these comments help my fellow reviewers and ACs understand the basis of my recommendation.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**(S1)** Clear drive and originality. This work identifies redundancy in existing lightweight vision models like GhostNet and inverted bottlenecks, which is a topic of practical signifance. It presents a mathematically grounded alternative in Sec. 3.1, leveraging the non-linear benefits of Hadamard products. This moves beyond previous linear cheap operations (e.g., GhostNet, gating-based models) and explores a new, parameter-free path to feature learning.\n\n**(S2)** Technical soundness of method. The ACH module is well-engineered to address practical learning and stability constraints, including discrete channel selection via Gumbel-TopK sampling in Sec. 3.2, dynamic softsign normalization in Sec. 3.3, and NAS integration strategy in Sec. 3.4. In particular, I appreciate the ablation study in Fig. 3 which provides evidence for the design choices. The dramatic accuracy drop (73.57% to 64.39%) when replacing DySoft with vanilla BatchNorm is striking and supports the claims of stability. Appendix presents hyper-parameters, hardware, training schedules, and code release, which aids reproducibility.\n\n**(S3)** Focus on deployment and GPU optimization. Section 4.2 shows efforts to bridge theoretical and real-world inference latency, such as custom CUDA kernels (direct-indexing and parity-balanced) and algorithmic choices for ACH operation. Fig. 5 visualizes runtime differences under varying tensor shapes. This focus on implementation details is useful for helping with real-world deployment in resource-constrained scenarios. \n\n**(S4)** Broad experiments and competitive results. Ablation studies in Fig. 3 and Tab. 1, 2, 5 break down the contribution of each module component and validate plug-and-play integration on multiple efficient models. NAS-derived architecture choice is also examined in Tab. 3 and Appendix. Tab. 7 provides direct comparisons on CIFAR-100 and ImageNet-1K, considering both accuracy and efficiency metrics (latency, FLOPs). Notably, it achieves higher accuracy than recent MobileNetV4-L (74.73% vs. 74.38%) with less than 1/3 of FLOPs (669M vs. 2170M).\n\n**(S5)** Visualization and interpretability. Grad-CAM in Fig, 6 and 7 show that ACH enables more semantically meaningful feature extraction and achieves improved early focusing compared to linear baselines. Diagrams like Fig. 2 (ACH pipeline), Tab. 1 (activation comparisons) are also well-constructed and informative.", "weaknesses": "**(W1)** In Appendix A.4, complexity ratio for Ghost/ACH modules is handled via “given $m \\ll n$” simplifications, but experiment settings do not clarify the $m:n$ ratio in real deployments. This may hinder the reproduction or subsequent improvements in the community. I encourage the authors to provide more technical details on this point in revision.\n\n**(W2)** For object detection in Tab. 6, the gains of Hadaptive-Net-L over GhostNetV3 or MobileNetV4 are a bit marginal. This raises the question whether it actually overcomes the efficiency wall or simply offers a modest incremental boost within a specific operator. I recommend the authors add direct clarification or discussion on this point in related section. This would significantly alleviate readers' concerns.\n\n**(W3)** Seems no evidence is offered as to robustness under distribution/drift, input permutation, adversarial queries, or transfer to other domains (e.g., audio, language). This limits the “deep learning primitive” claim in Sec. 6. I recommend the authors consider slightly tempering this claim in the revised manuscript.\n\n**(W4)** Incomplete literature review. Several important prior work [1] [2] [3] [4] [5] that applies Hadamard products in efficient vision models, attention/bilinear pooling, or edge detection are not included and discussed. I recommend the authors incorporate related discussions in the revised manuscript, which would not only improve the completeness but help emphasize the contributions and novelty of ACH within existing studies. It would be better to show the direct comparisons with [4] and [5].\n\n**(W5)** The fine control of hyper-parameter $\\tau$ is managed solely through heuristics and lacks sensitivity analysis experiments to support the choices. \n\n---\n\n## Reference\n\n[1] Hadamard Product in Deep Learning: Introduction, Advances and Challenges, TPAMI 2025. This survey covers the use of Hadamard products in deep learning, especially for efficiency and nonlinear fusion. \n\n[2] Hadamard Product for Low-rank Bilinear Pooling, ICLR 2017. It proposes efficient attention musing Hadamard products relevant to feature recombination methods. \n\n[3] Unmixing Convolutional Features for Crisp Edge Detection, TPAMI 2021. It explores element-wise feature fusion similar to Hadamard approaches, directly related to efficient feature expansion.\n\n[4] MogaNet: Multi-order Gated Aggregation Network, ICLR 2024. It is a representative efficient CNN model that uses gated Hadamard products for multi-order feature aggregation.\n\n[5] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model, ICML 2024. It first utilizes SiLU-gated Hadamard products within bidirectional state-space blocks to enable selective feature propagation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JKCvW3fstv", "reviewer": "Reviewer_3mhu", "summary": "This paper introduces the Adaptive Cross-Hadamard (ACH) module, an operator that embeds learnability through differentiable discrete sampling and dynamic softsign normalization. The ACH module is designed to enable parameter-free feature reuse while stabilizing gradient propagation. It is integrated into the Hadaptive-Net (Hadamard Adaptive Network) using neural architecture search to achieve high efficiency.", "questions": "(1) In the sections on \"HADAMARD FOR CHANNEL EXPANSION\" and \"DIFFERENTIABLE DISCRETE SAMPLING,\" the paper primarily describes how these techniques are applied but lacks a detailed explanation of why they were chosen. Could the authors elaborate on the underlying motivation and principles for using these specific technologies? The current description in this area feels somewhat underdeveloped.\n\n(2) Regarding \"DIFFERENTIABLE DISCRETE SAMPLING,\" it seems that several of the components, such as the \"Gumbel-Topk trick\" and the \"ECA module,\" are existing techniques. Could the authors clarify the novelty of their approach in this context and how these existing tricks are combined or adapted in a non-trivial way?\n\n(3) The experiments primarily use CNN-based backbones. Have the authors evaluated or considered the applicability of their method to efficient Transformer architectures? The lack of such experiments may limit the perceived scope of the contribution.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper's approach of leveraging the Cross-Hadamard Product from the perspective of feature expansion to enhance model efficiency is novel and interesting.\n\n(2) The paper is well-written, providing a comprehensive description from its methodology to the implementation details.", "weaknesses": "(1) Insufficient Motivation for Technical Choices: The paper lacks a clear and in-depth discussion of the motivation behind its core technical components. For instance, in the sections on \"HADAMARD FOR CHANNEL EXPANSION\" and \"DIFFERENTIABLE DISCRETE SAMPLING,\" the paper focuses more on the implementation details (how) rather than the fundamental reasons (why) these specific techniques are suitable or advantageous. This makes it difficult for readers to fully appreciate the design principles of the proposed method.\n\n(2) Potentially Limited Novelty in Sub-components: The novelty of some technical components appears to be incremental. The section on \"DIFFERENTIABLE DISCRETE SAMPLING,\" for example, seems to rely heavily on the combination of existing techniques like the Gumbel-Topk trick and the ECA module. While combining existing ideas can be a valid contribution, the paper does not sufficiently articulate what makes this specific combination novel or non-trivial.\n\n(3) Limited Experimental Scope: The experimental evaluation is confined to CNN-based architectures. In the current deep learning landscape, demonstrating effectiveness on Transformer-based models is crucial for showing broad applicability, especially for a method focused on efficiency. The absence of such experiments makes the generalizability of the proposed Hadaptive-Net unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)