Composable Sparse Subnetworks via Maximum-Entropy Principle
Authors: Francesco Caso
Samuele Fonio
Simone Monaco
Nicola Saccomanno
Fabrizio Silvestri
Venue: ICLR 2026 Poster (2026)
Keywords: Maximum Entropy Principle; Iterative Magnitude Pruning; Circuits; Modular Neural Networks
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We present a technique to extract class-specific subnetworks behaving as reusable functional modules, which can be composed by summing their weights or averaging their logits.

Abstract: Neural networks implicitly learn class-specific functional modules. In this work, we ask: Can such modules be isolated and recombined? We introduce a method for training sparse networks that accurately classify only a designated subset of classes while remaining deliberately uncertain on all others, functioning as class-specific subnetworks. A novel KL-divergence-based loss trains only the functional module for the assigned set, and an iterative magnitude pruning procedure removes irrelevant weights. Across multiple datasets (MNIST, FMNIST, CIFAR-10, CIFAR-100, tabular and text classification data) and architectures (MLPs, CNNs, ResNet, VGG), we show that these subnetworks achieve high accuracy on their target classes with minimal leakage to others. When combined via weight summation or logit averaging, these specialized subnetworks act as functional modules of a composite model that often recovers generalist performance. For simpler models and datasets, we experimentally confirm that the resulting modules are mode-connected, which justifies summing their weights. Our approach offers a new pathway toward building modular, composable deep networks with interpretable functional structure.

Published as a conference paper at ICLR 2026
COMPOSABLE SPARSE MAXIMUM-ENTROPE PRINCIPLE
SUBNETWORKS
VIA
Francesco Caso +,1.2, Samuele F Simone Monaco
Nicola Saccomanno"s +:5
Fabrizio Silvestri'
'Sapienza University of Rome, University of Cambridge, & University of Turin,
"Politecnico di Torino, University of Udine
ABSTRACT
Neural networks implicitly learn class- seasss-pecific functional modules. this work, we ask: Can such modules be isolated and recombined? We introduce method for training sparse networks that accurately classify only designated subset classes while remaining deliberately uncertain all others, functioning class- specific subnetworks. A novel KL-divergence--aaee based loss trains only the functional module for the assigned set, and an iterative magnitude pruning procedure re- moves irrelevant weights. Across multiple datasets (MNIST, FMNIST, CIFAR- 10, CIFAR-100, tabular and text classification data) and architectures (MLPs, CNNs. ResNet, VGG), we show that these subnetworks achieve high accuracy on their target classes with minimal leakage to others. When combined via weight summa- tion or logit averaging, these specialized subnetworks act as functional modules of a composite model that often recovers generalist performance. For simpler mod- els and datasets, experimentally confirm that the resulting modules mode- connected, which justifies summing their weights. Our approach offers a new pathway toward building modular, composable deep networks with interpretable functional structure.
INTRODUCTION
Modern neural networks (NNs) implicitly develop internal subgraphs of neurons and connections tuned to respond to specific classes. These structures, sometimes referred to circuits (Olah et al.) 2020 O Neill and Bui 2024). emerge during training but are difficult to isolate, reuse, or compose. Representations for different classes are often entangled. resulting in shared neurons features phenomenon known as superposition (Mu and Andreas 2020 Saphra and Wiegreffe 2024 which makes clean modularity elusive.
This lack of class-level modularity limits our ability to understand, edit, or compose networks. we could reliably extract functional module-a subnetwork that specializes in recognizing one class (or a small subset) while ignoring others-we we could provide new tools both for modular NNs and for mechanistic interpretability. Crucially, such modules must not only function in isolation, but also be designed to compose smoothly, without fine-tuning or alignment (Ilharco et al. 2023 Hazimeh al. 2024). In this work, we want answer the following question:
Can we train sparse, class-specialized subnetworks that remain ignorant outside their domain, and compose into accurate, generalist models?
We answer affirmatively, proposing methodology that leverages key concepts: the Maximum- Entropy (MaxEnt) principle and sparse training. The MaxEnt principle (Jaynes 1957) advocates choosing the most uninformative distribution consistent with known constraints. Originally applied
in statistical mechanics, it has become central to inference theory (Kuić 2016 De Martino and
De Martino 2018
We claim that MaxEnt can guide functional isolation: modules are trained to make confident pre- dictions only for their class, and uniform predictions otherwise. This differs from standard entropy regularization used in calibration or selective prediction (Marzzak et al. 2024 Pereyra et al. 2017)
Alphahetical order. See Author Contributions section for details. Email: fc5800 aaaam.ac.u uk

---

Published as a conference paper at ICLR 2026
On the other hand, sparsity plays crucial role in this context. Through pruning, the network discards weights that are irrelevant to the task. possibly mitigating interference during weight-space merging. We demonstrate its practical benefits by leveraging the Lottery Ticket Hypothesis (LTH). In general, LTH claims that sparse subnetworks can match dense models if trained with the right initialization (Frankle and Carbin 20188) Zhoo et. 2019 Liu et al. (202)) In our scenario, pruning is essential for exposing the circuit and sparse subnetworks show improved modular compatibility.
2019
In light of this, we propose framework based on two main components: a novel Maximum- Entropy loss (MB) coupled with an Iterative Magnitude Pruning (IMP) strategy (Frankle and Carbin 2018), and a model merging step to achieve generalist models. The goal of the ME is to en- force confident predictions on a subnetwork's 's rewarded class set and uniform predictions otherwise, while the IMP removes spurious capacity and reinforces specialization through sparsity (Burkholz) 2022 Girish et al. (021). We call the resulting components subnetwork modules: sparse networks that specialize in class or class subset, and can be combined to form more generalist models.
To the best of our knowledge, this is the first work that trains NN by isolation and merging. Our results suggest a new approach to NN construction: rather than learning entangled solutions end-to- end, we can build systems from independently trained, antropy-regulariize modules. To summarize, our contribution is threefold: (i) we introduce a novel MaxEnt loss able to train specialized modules; (ii) we propose a new training pipeline to achieve better isolation and model merging to get generalist models; (iii) we validate our findings through extensive experiments on several architectures and datasets.
2 RELATED WORK
In this section, due to lack of space, we put only the most relevant related work. An extended version of related work involving model merging. pruning methods, modular training and MaxEnt principle is available in Appendix A
Modular Neural Networks. Several works have explored modular neural architectures that decom- pose computation across tasks (Kirsch et al. 2018 Han et al. 2021 Salem et al. (023).. Notably, Kirsch et al.) (2018) introduced end-to-end learning of both modules and their composition via a controller. They rely subsequent modules for learning broader tasks, while using expectation- maximization for learning a specific module. While sharing a similar motivation to our work, we build class-specific functional modules, trained to focus on single class while producing high- entropy on different classes. This specific goal is hindered by their structure of subsequent modules, that are chosen by a controller and reused across tasks, making it difficult isolate class-specific functionality. Additionally, our approach relies on model merging, rather than leveraging subsequent modules. Interestingly, Malakarjun Patil et al. (2023) highlight the importance of pruning methods in unveiling the hicrarchical structure of NNs. In particular, they propose an iterative approach for hierarchy detection, where the hierarchy is the underlying hierarchy of sub-functions in a specific task, delivering interesting network analysis. Our work partially shares the goal, but focuses more on a complete training pipeline and on submodules isolation without seeking hierarchical structure, rather than network analysis about submodules.
Interpretability. Our work inevitably aligns with the goal of producing interpretable net- works. Among the methods that tackle this challenge, we can distinguish between post-hoc and siceppeetableby-ddeet methods. While post-hoc interpretability methods (Ribeiro et al. 2016 Lundberg and Leel 2017) are constrained to work with trained models, our methodology is ap- plied during training. yielding units with clear semantics and compositional behavior. In particular, approach aligns with mechanistic interpretability methods (Saphra and Wiegreffe 2024 Olah let al. 2020), where subnetworks (or circuits) are studied as functional entities. Interestingly, al. (2023) propose brain-inspired modular training procedure, relying on network embedding into geometric spaces, and neurons swapping. Instead of leveraging pruning methodologies, they rely mainly on regularization. However, they mainly use synthetic datasets and never scale to CNN, ResNet, or VGG, while we show that our technique is applicable also to a broader range of tasks. Finally, our work also complements neuron-level analyses like Mu and Andreas (2020), which char- acterize the concepts encoded by individual neurons from a compositional point of view. Rather than probing what individual neurons encode post-hoc, we directly train subnetworks to be functionally specialized by construction, obtaining modular units with unambiguous class semantics. Our work is also conceptually related to Marchetti et al. (2024), where symmetry constraints shape functional structure our modules may reflect class- symmetry components in such framework.
2

---

Published as a conference paper at ICLR 2026
3 LEARNING COMPOSABLE
SUBNETWORK MODULES
We propose a method for constructing compos- able subnetwork modules, each specialized in classifying a specific subset of classes while re- maining unresponsive to others. These mod- ules are trained to produce peaked predictions on their target classes and uniform predictions otherwise, in line with the MaxEnt principle. Our proposed procedure consists of two key el- ements that we are going to discuss: a custom loss function and IMP schedule. Our final goal is to merge trained modules efficiently, re- sulting in a more powerful model, built by sim- ple merging. Figure provides an high level overview our method and its components.
8a
TNp
IMP
Algo. applied for each class
NE
ME
0
0c
Class class-specific
Example of Complete Merge
functional modules
Merged models
0p.2-c
3.1 MAXIMUM-ENTRON LOSS
Figure 1: Overview of our method. Starting from common initialization 6g we find a specialized subnetwork for each class in C, leveraging IMP and our ME loss (see Algorithm
Let be the full set of classes and R S the set of rewarded classes for given subnetwork module. For a training sample (z. y), where y E C. define target distribution ÿ R as:
di=y C
if y R otherwise
ÿ1
(I)
For example, if C 2, and R = {0} we will use 7 (1, 0, for class 0, 7 (0.33, 0.33, 0.33) for classes and 2. Let ÿ roftmax(fo(x))) be the predicted class probability distribution produced by the model. The MaxEnt loss (ME) is then defined as the Kullback-Leibler (KL) divergence between the target distribution ÿ and the predicted distribution ÿ:
1C
U
(ME(x,)) = Kī (1 =
Σ
D: log
(2)
This formulation encourages the model to output peaked predictions (low entropy) for samples in rewarded classes and uniform predictions (high entropy) for non-rewarded ones. In doing so, it promotes functional isolation, ensuring that each subnetwork module specializes only its intended class subset and remains maximally uncertain elsewhere.
Unlike one-vs-all formulations, our approach enforces neuron-permutatiin invariance on non- rewarded samples-a crucial property for composability. In one-vs-all setup, training a module for class would implicitly push other output neurons (e.g., neuron 1) to represent "not class 0." However, if another module uses neuron to represent "class 1," summing the two leads to seman- tie collision: the same neuron would encode both "class 1" and "not class 0", which includes many other classes.
3.2 ITERATIVE MAGNITUDE PRUNING
To enhance specialization and remove spurious capacity, apply Iterative Magnitude Pruning (IMP) (Frankle and Carbin 2018) as in Algorithm [ At each iteration, we train with ME, prune fixed percentage of low- magnitude weights, and reinitialize the remaining ones. After several rounds, final training phase is applied to the resulting pruned subnetwork. This process encourages class-specific specialization.

---

Published as a conference paper at ICLR 2026
Algorithm Iterative Magnitude Pruning with Maximum-Entropy Loss
Require:
Initial weights On. training data D, rewarded classes R. number of pruning iterations N,
pruning percentage P, epochs per iteration E
1: 8 <- 0g
2: for = to do
3:
Train model fo on D using ceniiimum-nntrrope loss with rewarded classes R for E epochs
4: Prune K% of weights in 0 with smallest absolute value, with K 1 P)
5: Reset remaining weights in to their initial values from Op
6: end for
7: Train the final pruned subnetwork on D using maximum-entrope loss with rewarded classes R
for E epochs
3.3 MODEL MERGING
We investigate two strategies for composing a generalist model from class-specific submodules iden- tified by ME training and eventually pruned through IMP. These strategies operate in weight-space and logit-space, respectively, and reflect two distinct regimes revealed by our experiments.
3.3.1 VIA WEIGHT SUMMATION
The first approach consists of composing a generalist model by summing submodules' weights: 8 merged = =; 8,. This operation is enabled by our design: submodules are trained to specialize on disjoint subsets of classes and to behave identically-via uniform predictions-on all others, minimizing interference.
We consider both pairwise merges (01 02) and complete merge E 02), and evaluate composition- ality using the same metries used for individual modules. To understand when and why summation preserves performance, we analyze the loss landscape through the lens of mode connectivity. Cru- cially, unlike typical mode connectivity studies, where 0 and 0g solve the same task, our submodules are specialized for different sets of classes. The merged model is intended to solve the union task, and is evaluated using the MuxEnt loss over the combined rewarded classes This distinction makes it essential to assess whether the merged weights lie in a low-loss region for the composite task.
Following Frankle et al. (2020) and Lubana al. 2023), we say that 0₃ and 0 are mode connected along a path 7(t) if:
VI E 10, ((T(o(()) S ((1-1) Ll,, (D)) - L (D), ++).
(3)
where is small margin, set 2% of the first term on the r.h.s. following Frankle al. (0220) and C is our ME loss evaluated on a dataset DD with labels restricted to R2 UR2, the union of the rewarded classes for the modules.
To test this, we define the following piecewise-linean path, designed such that 0, + 0g appears as an intermediate point:
+01 + 21 2
if S 0.5
70, (1)
(4)
2 - 0 + 02 if t > 0.5
This path satisfies: 7(0) 01, (1) 82, and (0.5) 01 + 02. We evaluate the loss along 7t)) and interpret low-barrier profiles evidence that the modules are composable by construction.
3.3.2 VIA LOGIT AVERAGING
In addition to weight summation, we consider a second compositional strategy, i.c., merging mod- ules directly logit-space. Let fe, a submodule trained with ME on rewarded classes R, S C. For an input z, define its logits as 2(0) = fo, (z) E CI, and thus g(t) softmax(2()). By con- struction. ME encourages each fo, to output peaked distribution on samples in R2 and uniform distribution on C,R,, Hence, each submodule behaves as specialist expert, informative only on its rewarded set. We merge N such experts through a convex combination of their logits (specifically, their average),
N
2 =
Wi
2 (i)
w; 2 0, Σ w; =
(5)
1=1
4

---

Published as a conference paper at ICLR 2026
529 938
4.20
039
3.40
40140 15.00 22.0 23 11.10 1640 20420
1:00
1030
8:0
15:00
21.00
600
6.00
340
11.00
540
20132 tass 18A9 20 10.40 201 p0vt
13:00 608
17.00
9:00
21ea0 1340 11:00 an 1930 1ovo souid 3000 13140 SELANIN 1840 11:00
1929
22.00
100
1540
2800
500
377 1433 MLA poear pance 19.0 400
5.06 2:00 020 -- 710 440 6.00 5.0 15.00 11.0
1.0 1120
(a) Classes [0]+[6]
(b) Classes [0.1]+[8.9]
(c) Classes |0 4]+[5-9]]
Confusion matrices (total predicted counts per class) for three representative pairwise merges in weight-space of shallow MLP submodules on MNIST. Even after merging, each rewarded class remains highly separable, and predictions on non-rewarded classes remain nearly uniform.
Figure 2:
and predict y softmax(()) This operation has direct probabilistic interpretation: since softmax is an exponential normalization, Eq. ilss equivalent to a logarithmic opinion pool (Genest and Zidek 1986), i.e.. a product-of-expert aggregation (Hinton 2002).
N II exp(= ( 1
N x II 1=1
yh softmax(3)s x exp
W7
(8) kk
mo
(6)
Under ME, any expert that is non-rewarded for outputs nearly uniform g()) which contributes only a innseenndeeendntt factor in Eg and thus cancels after normalization. Therefore, logit averaging automatically ignores irrelevant experts and preserves only the specialists that match x, explaining the strong task performance observed in our experimental evaluation.
4 EXPERIMENTS
We empirically evaluate our method on multiple datasets, aiming to answer the following questions: (i Can ME training enforce meaningful class-specific specialization? (ii) How well do merged submodules recover generalist performance? (iii) Does iterative pruning improve the quality of submodules and composability (iv) Does our training procedure induce mode connectivity between independently trained modules and does it scale with the number of modules?
4.1 TRAINING AND EVALUATION PROTOCOL
We apply our ME and IMP independently to each class or subset R C C. Each submodule trained as in Algorithm I and evaluated on: (i) rewarded accuracy, i.e., classification accuracy on samples from R; (ii) non-rewarded entropy, i.c., the average predictive entropy on inputs from C \ R; and (iii) the confusion matrix, used qualitatively assess specialization and leakage. In partieular, it is worth mentioning that in our training procedure we simulate a scenario in which only the classes in R are labeled, but we still have access to the other samples. Formal definitions and implementation details are provided in Appendix B
To evaluate composability, we used [R] (1,2,5) to test pairwise merging. For a specific car- dinality, the model is trained to recognize solely a selected set of classes, of the given cardinality and sampled among the ones available for that specific dataset. We sample 10 pairs of class sets (RR... R2 with Rex n R2x+1 = 0 and R2R. R2*+1 spanning in the set of combinations for the dataset's class set. For each pair, we perform 5 independent random seeds and merge a pair into a single model. For example, with |R] = 1, the procedure consists of: training NN using our ME rewarding one specific class Ro. then a different NN is trained on a different single class R,, and then the two models are merged and tested. The resulting model should perform well on the 2 classes. This procedure is repeated for 5 seeds and 10 paired sets to ensure statistical evidence. When |R|
> 1, two NNs trained to specialize time, and there is no overlap between the sets. Figure 2 depicts an example of resulting confusion matrices for these pairwise-mergin experiments.
two different sets of classes at the same
5

---

Published as a conference paper at ICLR 2026
Table 1: Rewarded accuracy and non-rewarded entropy of sub-modules obtained using ME with and without IMP. Results are averaged over 5 runs and across classes. Number of classes for dataset: 10 for MNIST and FMNIST, for HAR and a balanced subset of 4 for Yeast. Accuracy is high on rewarded classes and predictions are nearly uniform on excluded ones.
MNIST
FMNIST
HAR
Yeast
Model
IMP
Entropy Rewarded Ace
Entropy
Rewarded Ace
Entropy
Rewarded Ace
Entropy
Rewarded Acc
Shallow
No
2.296
(000))
0.998
2.296 (0.0m))
0.998 (0000)
1.762
(D017)
0.997 mam
1.298 (105))
0.995
MLP
Yes
2.293 (0004)
0.999
0100)
2.293 (109)
0.999 (0001)
1.757
(0023)
0.996 (100))
1.297 (119))
0.998 (1000)
Deep
No
2.298
(0003)
0.997
(0.00)
2.285
(0.03))
0.995
(0004)
1.772
(0014)
0.992 m00.)
1.302 (005
0.996 (0009)
MLP
Yes
2,300
(0001)
0.998
(1007)
2.291
(1:03)
0.991
(600r)
1.762
0.999 muns)
1.302 (199)
1.000 (n/000
CNN
No
2.302
100000)
0.998
(0.00)
2.302
(0.000
0.996
(6004)
Yes
2.302 100000
0,994
(1000)
2.302
(1.000))
0.992
(ouX5)
4.2 EXPERIMENTAL SETUP
We evaluate our approach on four model classes: MLPs (denoted as shallow for 1 hidden layer and as deep for hidden layers). CNNs (LeNet-style), ResNet18, and VGG1I (with or without Batch- Norm). For image data, we use MNIST (train/test 60K/10K) (LeCun et al. 1998), Fashion MNIST (FMNIST, 60K/10K) (Xiao et al. 2017). and CIFAR-10 (60K/I0K) (Krizhevsky et al., 2009). with 10 classes for each dataset. These datasets are in line with similar works (Liu et al. 2023] Malakar- jun Patil et all 2023), but to get a broader impact we include two tabular classification datasets from the UCI repository: Human Activity Recognition (HAR, 7K/3K) (Reyes-Ortiz et al., 2012) (6 classes), and Yeast (///0 (Dua and Graff] 2017) (a balanced subset of classes). For each model- dataset pair we train the modules both with and without IMP and merge them both in weight-space and logit-space.
We compare our ME mainly with two baselines, showing that entropy maximization plays a crucial role in isolation. In particular, we compare ME with: Quasi-MaxEin loss (QME), a loss function that on non-rewarded classes ignores the rewarded classes and produces uniform distribution solely on non-rewarded classes; and CrossEntropy loss (XE), the standard classification loss, where labels are not modified but the model is trained only on the classes in R. We additionally consider XE with label- aabel-smoothing (A 0.1), which replaces the one-hot target distribution with a smoothed version; and XE with confidence-penalee which adds an entropy rregoarraruulaiization term. For the formal defini- tions, we refer to Appendix B Due to lack of space and underperforming behavior, results related to label-smoothing and confidence penalty are shown Table 9 Appendix [ Full dataset, prepro- cessing and experimental details are reported in Appendix B Next we describe the main outcomes.
Extensive and detailed experimental
results are provided in Appendix C
4.3 SUBMODULE BEHAVIOR WITH MAXENT LOSS
We begin by evaluating submodules trained solely with ME, without pruning. Each subnetwork is tasked with recognizing one specific class from the original dataset, and to output uniform predic- tions on all others. For each of 5 random seeds, we train separate submodule per class. The results in Table I are aggregated across both classes and seeds, providing average values and standard deviations for overall trends.
Table reports the accuracy on rewarded classes and the average entropy on non-rewarded sam- ples. Accuracy is consistently high (close to 100%) across all classes, confirming that submodules correctly specialize on their class. The entropy, meanwhile, approaches the theoretical value of log(NNlsses)) for uniform distribution, e.g. log(10) 2.30 for MNIST and FMNIST, suggesting that predictions on excluded classes are nearly uniform. These results suggest that our proposed submodules can effectively recognize individual classes and exhibit the theoretical behavior they were designed to emulate, across different architectures.
4.4 PAIRWISE MERGING
We further evaluate the effectiveness of our method through pairwise merging experiments across different configurations. In Figure 3 we show the effectiveness of IMP on MNIST and FMNIST on three architectures (Shallow MLP, Deep MLP, and CNN) and all cardinalities, while merging in weight-space. We can in particular that IMP improves the performances consistently for a Deep MLP, while benefiting more modestly the Shallow MLP and the CNN.
6

---

Published as a conference paper at ICLR 2026
Table 2: Average rewarded accuracy of pairwise-merged sub-modules, merging either via logit av- eraging or weight summation, across architectures, datasets, cardinalities (R], and loss functions: CrossEntropy (XE), Quasi-MaxEnt (QME), and MaxEnt (ME), with IMP applied. Results are av- eraged over 5 seeds and 10 pairs. For each group model-cardinaliyy-aaaae the best approach is highlighted in bold. ME consistently achieves the highest rewarded accuracy.
FMNIST
MNIST
HAR
Yeast
Model
R
Loss
Logit
Weight
Logir
Weight
Logit
Weight
Logit
Weight
XE
0.859 0L880
0.813
0.209
0.798 (0162) 0.679 (1189)
0.882 (0.164)
0.866
00110
0.667 (0.00) 0.660 (0.04)
QME
0,962 (ues)
0,868
(1.47)
0.984 (0000)
0.973
mor
0.955 (0013)
(0241)
0.734 (0.2))
0.737 (0.122)
ME
0.983 (./2)) 0,980
0.992
(1.00) 0.991 (0.05)
0.982
(8026)
0,983
(0:06
0.844 (0.115
0,843
(0.110
0.777 0.:11:
0.852
0.831 (1007)
0.870 (uoe 0.864 (0.29))
0.439 (omss))
0.419 (0019)
Shallow MLP
2
QME
0,953 (001))
0.918 (000))
0.982 (0.000) 0.963 m005))
0.937 (0/10))
0.919
m..0m
0.474 (001
0.475 (0/01)
ME
0.960 (1023) 0.952 (0.25)
0,983 (0.005) 0.980 (0.000)
0.949 (00044)
0.945
m0.16)
0.565 (00665)
0.549
(0.04))
XE
0.655
msen
0.434
0.855 (ucon) 0.831
5
QME
0.865
(u02)
0.752 (0006)
0,950 (0001)) 0.909 (0.02))
ME
0.867
m0m
0.827
0.952 (1000)) 0.945 (0:00)
XE 0.889
1.050
0.813 01212)
0.834
(0.125) 0.668 (1.77))
0.887
(0.000)
0.840 (.314)
0.689 (0005))
0.674 (0/95)
QMF 0,956
0.858 00.177
0.971
0,960 (1109)
0.943
0.923 (0.90))
0.800 (0.11))
0.807 (0:12))
ME
0.978
0000))
0.973 00.400
0.992 (0.000)
0.991 (0.00))
0.982
(0029))
0,982 (0:00m
0.844 (0.117) 0.839 (0.119)
X1: 0.790
0.748 (0.121)
0.841 (0.0m))
0.724 (1085)
0.857 (006s)
0.821 (0.065)
0,400 (0033)) 0.372 (0027)
Deep. MLP
2
QME
0,947
0.023)
0.846 00.133)
0.979
(0.007) 0,910 (0.05))
0.926
(00200)
0.868
(0:065)
0.485 (0424)) 0.486 (0:020)
ME
0.955
(.004) 0,930 (0.000)
0.983 (0,005) 0.975 (0.00))
0.942
(0014))
0.870
(0.208)
0.608 (0.01)) 0.598 (0005)
XF 0.624
m.0a)) 0.500 m0:4
0.799
(1o07)
0.687 (1:00)
QME
0.649
(0.09;
0.941
(0:00))
0.807 (0,00)
ME
0.852
m000)) 0,800 (0.01:
0.944
(0.00)) 0.919 (00)
XB 0.633
(.182)
0,576 00.155)
0.539
(0.00)) 0,520 (0.00))
QME
0,955
mnon 0.896
0.986 (1cor)) 0.952
ME
0.983
(100)) 0.966 (1307)
0.997
(0.02) 0.984 (0.000
XE 0,568
0m 0.630 00.10:
0.675
0.638
CNN
2
QME
0,959
01077) 0.898 o100)
0.992
(0.00)) 0.945 (1:54))
ME
0,972
(:07)) 0.926
0.994
0.964
XE
0.625
0.002) 0.493
01.14)
0.901
(0,055)
0,776 (1.66))
QME
0.912 0.00
0.790
(0.045)
0.986 (0.000)
0.909 (0.044)
ME
0.915 0L002) 0.791 (1.03))
0.988 (0.c01) 0.943 (0.054)
For what it concerns IMP itself, for each model we have tried different level of pruning. Interestingly, Shallow MLP and Deep MLP did not show specific harm with pruning. enabling us to use a percentage of 99%. For the Convolutional Networks (CNN, ResNet18, VGGII), the optimal pruning ratio was 60%. More details about this are provided in Appendix B.7 Notice that each model pruned over two iterations, followed by a final training phase.
Table reports the aggregated metrics over seeds and merged pairs when using IMP. The full results are in Ta- ble ble and Table of of Appendix C Rewarded accuracy re- mains high (often 2 0.90), indicating that each merged module retains its classification ability. The mean en- tropy on non-rewarded classes stays close to the theoret- ical maximum of log(NNeasses), e.g. it is around 2.28 for MNIST and FMNIST, confirming minimal interference (see Table D
dit
and Ta the Appendix
Moreover, we can see that ME outperforms QME in all cases. Besides being such a small change in the learning phase (ME enforces uniformity over all neurons for non- rewarded inputs, while QME enforces uniformity over all but the rewarded ones), this loss function loses in perfor- mance with respect to ours. This surprising result is prob- ably due to the leakage of information on newareed classes. In particular, the QME incentivizes changes in the flow of information even for new-rewarded classes, which violates the MaxEnt principle. In contrast, ME en-
Figure 3: Impact of IMP on pairwise merging in weight-space for different architectures and cardinalities, consid- ering the MNIST (top) and FMNIST (bottom) datasets, and using the ME loss. IMP consistently benefits the Deep MLP, while its impact is more modest for the Shallow MLP and the CNN.
7

---

Published as a conference paper at ICLR 2026
Cardinality
FHNIST
1.0
0.0
+0.
-0.2
0.0
Coperrcopy MOKERE OuosRVarEnt
CresentrUy MaxEnt QuesiMakEm
Cross (000
MoxEnt Quos Moabnt
0.4
Dressfmmpy
Mwifnt
QuasBladfie
FMNIST
MNIST
CIARR0
Cardinality =
1.0
0.10
0.8
0.05
0.4
-2.05
COSSERSUP 9py MOKErE OuacMarKnn
CressEntrgyy MakEn: QuesiMueE
CrossE HUODY
MoaEnt QuSS MoaEnt
-0.10
MovEm: QuisINaxErc
FMNIST
MNIST
CHAR10
Cardinality
0.1
0.0
0.2
Figure Left (3x3 grid): rewarded accuracy ResNet18 on FMNIST, MNIST and CIFAR- 10 across cardinalities, loss functions, and merge strategies (logit vs. weight), with and without IMP, confirming that our framework extends to larger architectures. On simpler datasets (FM- NIST, MNIST), logit- and weight-space merging yield comparable results, whereas on CIFAR- 10 logit-space merging is substantially more effective. Right (vertical stack of 3 plots): differ- ence in rewarded accuracy between VGG1I with and without BatchNorm FMNIST (A VGG11-BY - VGG11) 1), showing that BatchNorm re-estimation consistently improves weight-space merging with IMP. Results are averaged over 5 seeds and 10 pairs.
courages only information from the rewarded classes, improving the specialization and better avoid- ing spurious activations. Our method also outperforms standard XE. This was expected, since XE intended train with all the labels once, while in this case its effectiveness is hindered by the reduced number of classes on which it is optimized.
Additionally, Table 2 we report both the results of merging in weight-space and in logit-space. We can clearly see that logit merge outperforms the weight merge in almost every case (the only exception is CNN with |R| = with XE). Moreover, logit merge mostly unaffected by the IMP methodology, there is no need of pruning when the submodules are not going to be merged in weight-space, where otherwise there could be interferences.
Focusing larger models, Figure 4 (left) shows the results ResNet18 trained within our frame- work on FMNIST, MNIST and CIFAR-10 (full numerical results are in Table/6] Appendix (C.. These experiments demonstrate that technique is extendible larger architectures and suggest differ- ent analysis for simple and complex datasets. On one hand, for FMNIST and MNIST, results are similar for both logit- and weight-space merging: our ME loss consistently outperforms XE and QME, especially at cardinality 1, while IMP provides limited benefits in this setting, yielding per- formances roughly on par with training without IMP. These results suggest that pruning removes irrelevant weights, thus reinforcing the functional isolation induced by the loss. On the other hand, for more complex datasets like CIFAR-IC 10, logit-space merging is much more effective, suggesting that in this setting it is not trivial to avoid potential interference between weights.
Investigating the applicability of our technique to larger networks, we have also considered the role of BatchNorm (BN). When merging in weight-space models with BN, it suffices to re-estimate the running statistics on data from the union of the rewarded classes, keeping all learned affine parameters frozen. This simple step is enough to enable weight-space merges for ResNet18 8 and VGG11 with BN, with a clear impact on the performance. In particular, in Figure 4 (right) we show the difference in accuracy between VGG11 with and without BN on FMNIST (full numerical results are in Table iin the Appendix(C). Clearly, no issues arise when performing logits-space merging of models with BN. In Appendices C.8 and C.10 further validate pairwise merging
8

---

Published as a conference paper at ICLR 2026
large-scale image dataset (Imagenette) and text classification (IMDB and 20 Newsgroups), confirming that the same conclusions hold in these settings.
4.5 MERGING ANALYSIS
(nd inogse/ndded Weight
mogginnnee Logis
In this section, we further evaluate our method in the more challenging scenario of a complete merge and provide additional analysis based on mode connec- tivity.
1.00
6.75
MAST
6.50
hAR
Yead
5.00
4
10
Complete Merge. We perform a complete merge experiment. For each dataset, we randomly generate 10 permutations of the class labels. For each of 5 random seeds, we train one submodule per class us- ing either ME alone or in combination with IMP. We then merge the submodules incrementally, following the order dictated by the permutation: at step k, the merged model contains submodules for the first k classes in the permuted list.
Inceemataal Stop
Intremariaa Step
(a) Shallow MLP
3.00
Daboct
MAST
Renurded
0.50
0.25
0.00
+ ho
Inmemerrall Step
(b) Deep MLP
Figure 5 reports the average rewarded accuracy each merge step, aggregated across permutations and seeds. We can see that submodules trained with IMP (solid lines) consistently outperform or match those trained without pruning (dashed lines) when merg- ing in weight-space. On Yeast, performance de- grades more rapidly as modules accumulate. Shal- low MLPs show good composability even in weight-space, where accuracy stays above 90% up to the full merge. Deep MLPs do not out- perform shallower ones overall. CNNs also face a performance degradation, but IMP importantly ben- efits their performance for weight merge.
me pa. middee Weghts
ma pa. mndred Log
3.00
6.75
Cataret HAST
6,50
PHIIST
0.00
4
6
35
8
10
Intemnrrall Stop
Inowneria Stea
(c) CNN
MNIST
3.00
Delenet
0.30
0.25
195
0.00
ho
The specific comparison between Shallow and Deep MLP suggests that additional depth and narrower layers do not benefit composability in weight-space, augmenting the possibility of intereference. This suggests that width has a crucial role in enabling composability in weight-space. Moreover, CNNs face a significant degradation while ResNet18 (a much deeper model) degrades only CIFAR-10. This further indicates the key role of BN in enabling smooth merge even in spacc. Additionally, we can see that logit merge significantly limits the degradation, recovering almost always the best per- formances on each dataset. The difference is particu- larly evident for CIFAR-IO, 10, for which weight merge fails dramatically, while logit merge achieves good performances. As a stress test, in Appendix C.9 we extend the complete merge to 100 modules on CIFAR- 100, finding that logit merging degrades gracefully across all 100 steps without sudden compositionality bottleneck.
Intemantall
(d) ResNet
Figure 5: Average rewarded accuracy across incremental submodule merging across ar- chitectures, aggregated over seeds and 10 class permutations. Solid lines denote mod- els trained with IMP, dashed lines without. Logit-space merging (right column) con- sistently limits accuracy degradation across all architectures and datasets, while weight- space merging (left column) benefits sub- stantially from IMP.
Overall, these results confirm that ME trains composable modules across a variety of architectures and datasets. For weight-space merging. while we have shown that it is possible to train in our proposed way, the suboptimal results indicate that further study should be conducted on pruning and merging techniques the role of the width. Nevertheless, the strong performance of logit- space merging demonstrates that ME is always able to train composable modules, regardless of the complexity of the architectures the dataset.
Mode Connectivity. To better understand why merging submodules via weight summation of- ten effective, we study the connectivity of models in weight-space. Specifically, we follow the formulation of Lubana et al.] (2023), and evaluate whether the merged model lies on a low-loss path between its components, using the ME loss as criterion. We analyze complete merges the

---

Published as a conference paper at ICLR 2026
88 25-
Loss -20
-9 20
88 -110
Losd 56
Loss
-50
03
03
1.0
0, = [0] 02 [1]
0, = [0]+[[]+[2]
0, = [0]+ +[4]
01 [0] [0]+.... [[[]
01 [0] +[8]
02 = [3]
02 [5]
02 [7]
02 [9]
Figure 6: Loss barrier along the linear interpolation path T0; - (1) considering multiple steps of the complete merge starting from 0 to 9 for the FMNIST dataset (train and test errors, in blue and orange respectively, mean and std across 5 runs). Loss values at each f are relativized with respect to the corresponding interpolated emrors of 01 and 02 and rescaled percentages. Values close to or lower than 0 indicate mode connectivity. Barriers remain near zero throughout most of the merge sequence, supporting the hypothesis that ME training induces weight-space composability, with only mild interference emerging in the later steps.
FMNIST dataset using shallow MIPs. For each step in the merge sequence- combining submodule obtained from the merge [0] +.... + [k: 1] and one trained on class k-we evaluate ME along piecewise linear path between the previous model and the new submodule, with the midpoint corresponding to their weight summation. Figure shows the relative loss barrier for each step, computed as the gap between the loss the midpoint (merged model) and the linear interpo- lation of the endpoint losses. Values near or below zero indicate mode connectivity. In the majority of cases, merged models correspond to local minima or lie along smooth, low-loss trajectories, with no significant barriers emerging along the merge path. This supports the hypothesis that the training procedure induces functional composability in weight-space. However, small barriers (around 2%) begin to emerge in the later steps of the complete merge-when many submodules have already been added-suggen suggesting mild interference.
5 DISCUSSION AND LIMITATIONS
Our proposed modular design might open avenues, but also presents limitations and open chal- lenges. IMP seems to help submodules isolation and thus weight-space merging, but the overlap between submodules should be carefully demistified, in order to optimize the best IMP strategy to isolte and merge subnetworks in this setting. While we rely on simple summation for the weight- space merge,
sophisticated merging techniques-e.g. Git-Rebasin Pleas (Ainsworth al.) 2023 Nasery et al., (2025) may further improve composability, particularly in challenging archi- tectures. However, merge logit-space already allows to compose our modules regardless of the architecture.
Furthermore, we believe effective modular training can find application in machine unlearning (e.g., removing or re-training just the single module), NN verification (where formally verifying an entire network is often intractable), federated learning (where merging different networks a key step), mechanistic interpretability by design, and many other settings. Showing concrete downstream uses is indeed valuable, and we plan to explore this in multiple future works.
6 CONCLUSIONS
We introduced principled framework for training sparse neural submodules that specialize in recog- nizing a designated subset of classes while deliberately remaining ignorant of others. This behavior encouraged via novel KL-based MaxEnt loss. In addition, we propose to IMP procedure to select only the relevant weights, i.c. the circuit. Together, they yield compact, high-performing subnetworks that with simple datasets and architectures can be composed via simple weight summa- tion-a procedure supported by empirical evidence showing that such modules are mode connected. For complex datasets and architectures, show that the modules composable in logit-space, producing product of experts, each one specialized on a specific class, and thus highly interpretable.
Our approach strengthens the agenda of modular NNs by providing submodules with well-defined functional roles. At the same time, it creates a controlled testbed for mechanistic interpretability. where circuits are available by design rather than discovered post-hoc. the long run, we envision functional submodules as key abstraction for building interpretable neural systems.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our results, all code necessary to reproduce the experiments presented in this paper is provided at the following link: https://gtthu..com/HHii http/raaneesooasooooooppaaaiaaa Subnetworks-Maxhh Additional details are provided in Appendix [3] which thoroughly documents the experimental setting: training configurations and pipeline, evaluation metrics, mode connectivity tests, datasets, pruning schedules, and hyperparameter choices.
AUTHOR CONTRIBUTIONS
This paper is the result of a real team effort, in which each author has been fundamental to the final outcome. Specifically, all authors contributed refining the methodology, programming and extending the codebase, conducting experiments, analyzing the results, and revising the manuscript. Among the authors, Francesco Caso stands out since originated the main research idea, defined the theoretical motivation based on the maximum entropy principle and coordinated the project. Fabrizio Silvestri supervised the work, provided guidance, and contributed shaping the research direction.
ACKNOWLEDGMENTS
Francesco Caso, Samuele Fonio, Simone Monaco, and Nicola Saccomanno sincerely thank Prof. Pietro Liô for hosting them during research visit in 2024, which laid the foundation for this work. Moreover, they also thank Lorenzo Petrosino for valuable discussions during the initial stages of the project. Francesco Caso acknowledges support from Renaissance Philanthropy through the AI for Math Fund for research conducted at the Department of Computer Science and Technol- ogy, University of Cambridge. Nicola Saccomanno acknowledges travel support from the Euro- pean Union's Horizon 2020 research and innovation programme under grant agreement No 951847, and the support from the Interconnected Nord-Est Innovation Ecosystem (iNEST), which received funding from the European Union Next Rext-GenerationEUU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) MISSIONE 4 COMPONENTE 2.NVESTIMENTO 1.5 D.D. 1058 23/06/2022, ECS00000043). Simone Monaco acknowledges travel support from ELIAS (GA no 101120237).
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models mod- ulo permutation symmetries. In The Eleventh International Conference on Learning Representa- tions, 2023. URL httpttt://peeeeeiiiiiee
net forum? webormmmi--CCe C000MMmMmmm
Anisa T Bajkova. The generalization of maximum entropy method for reconstruction of complex functions. Astronomical and Astrophysical Transactions, 1(4):313-320 1992.
Rebekka Burkholz. Convolutional and residual networks provably contain lottery tickets. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages https::/proceediine
PMLR, 17-23 Jul 2022. URL www..press/ee.mmm.m.....oeeeee.e Setess/vvv22// http:////.1///urroo1a2.......... 22tm2mm2ammmlmmmmmm
Rebekka Burkholz, Nilanjana Laha, Rajarshi Mukherjee, and Alkis Gotovos. On the existence of universal lottery tickets. In International Conference Learning Representations, 2022. URL
net www.:::doommdmmmmmmma
Andrea De Martino and Daniele De Martino. An introduction to the maximum entropy approach and its application inference problems in biology. Heliyon, 4(4), 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255 2009. doi: 10. 10..
Dheeru Dua and Casey Graff. Uci machine learning repository: Yeast data set. https:
7/archive ve. ics uci htt.eeea//mmaaaeeeeeeeeeee
2017. Accessed: 2024-05-01.

---

Published as a conference paper at ICLR 2026
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, arXiv preprint 1rXiv:1803.03633 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13- July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3259-3269, PMLR, 2020.
mlr. htlp.prres//11//rrnntntt
html
Christian Genest and James V Zidek. Combining probability distributions: A critique and an anno- tated bibliography. Statistical Science, I(I):114-135, 1986.
Sharath Girish, Shishira R Maiya, Kamal Gupta, Hao Chen, Larry Davis, and Abhinav Shrivastava. The lottery ticket hypothesis for object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 762-771, 2021.
Wei Han, Changgang Zheng. Rui Zhang, Jinxia Guo, Qinli Yang. and Junming Shao. Modular neural network via exploring category hierarchy. Information Sciences, $69:496-507, 2021.
Adam Hazimeh, Alessandro Favero, and Pascal Frossard. Task addition and weight disentangle- models. In Workshop Efficient Systems for Foundation Models II@
ment in ICML2024, 2024.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
Jeremy Howard. Imagenette, 2019.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
Sohei Itahara, Takayuki Nishio, Masahiro Morikura, and Koji Yamamoto. Lottery hypothesis based unsupervised pre-training for model compression in federated learning. In 2020 IEEE 92nd Ve- hicular Technology Conference (VTC2020- pages 1-5. IEEE, 2020.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. Advances in neural information processing systems, 31, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Domagoj Kuić. Maximum information entropy principle and the interpretation of probabilities in statistical mechanics- a short review. The European Physical Journal B, 89:1-7,2016
Ken Lang. Newsweeder: Learning to filter netnews.
http:// http://wwone.eo com/ jason./
20Newsgroups/| 1995.
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 2324, 1998.
Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang, Yang Xiao, Ruimeng Ye, Yang Zhou, Wei- Shinn Ku, and Bo Hui. A survey of lottery ticket hypothesis. arXiv preprint 14X4v.2400.04861, 2024.
Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, 26 (6(1))41, 2023.
Ekdeep Singh Lubana, Eric J. Bigelow, Robert Dick, David Scott Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Confer- ence Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawati, USA, volume 202 of Proceedings of Machine Learning Research, pages 22965 23004. PMLR, 2023. URL wwt.:sss.w......./me
https: fttps://procees/: ngs. ml www.ss/ssvvi. /ttess//v22/// wwtp2/////uuaa2cc. wuw.aaaaaaa.. h3m
12

---

Published as a conference paper at ICLR 2026
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguisties. URL http: 7/ 7/www. ac web. org/ antho http///noool/y/111111111
Shreyas Malakarjun Patil, Loizos Michael, and Constantine Dovrolis. Neural sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis. Advances in Neural Information Processing Systems, 36: 111118845-1885 18531, 2023.
Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of learning: Universal fourier features emerge in invariant networks. In The Thirty Seventh Anmual Conference on Learning Theory, pages 3775-3797. PMLR, 2024.
Daniel Marezak, Barttomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert. Magmax: Lever- aging model merging for seamless continual learning. In European Conference on Computer Vision, pages 379-395. Springer, 2024.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons, Advances in Neural Informa- tion Processing Systems, 33:17153-17163, 2020.
Anshul Nasery, Jonathan Hayase, Pang Wei Koh, and Sewoong Oh. Pleas-merging models with permutations and least squares. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 1429111/iiiilll.0004.00000
Charles O'Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identifica- tion in language models. arXiv preprint arXiv:2405.12522, 2024.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Sys- tems, 36:66777-66754, 2023.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 065488, 2017.
Jorge Reyes-Ortis Davide Anguita, Alessandro Ghio, Luca Oneto, and Xavier Parra. Human activ- ity recognition using smartphones. UCI Machine Learning Repository, 4:2, 2012.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
Nosseiba Ben Salem, Younès Bennani, Joseph Karkazan, Abir Barbara, Charles Dacheux, and Thomas Gregory. Modular neural network approaches for surgical image recognition. arXiv preprint 2301 000880800000..8 2023.
Naomi Saphra and Sarah Wiegreffe. Mechanistic? arXiv preprint arXiv:2410.0908) 2024.
Ryan Van Soelen and John W Sheppard. Using winning lottery tickets in transfer learning for con- volutional neural networks. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2019.
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In Proceedings of the 41st International Conference on Machine Learning, pages 80268-50287 2024.
13

---

Published as a conference paper at ICLR 2026
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 708.07178..77777, 2017.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng
Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024.
Siqi Zeng. Yifei He, Weiqiu You, Yifan Hao, Yao-Hung Hubert Tsai, Makoto Yamada, and Han Zhao. Efficient model editing with task vector bases: theoretical framework and scalable ap- proach. arXiv preprint arXiv:2502.01015, 2025.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL /ttppppss//pprrroeeds.ss..errrrr new ccppppprrcrpppppeepcp filesi
6par://000/aa///////ffffroooooo0 Paperr..
14

---

Published as a conference paper at ICLR 2026
COMPOSABLE SPARSE SUBNETWORKS VIA MAXIMUM-ENTROPY PRINCIPLE APPENDIX
APPENDIX STRUCTURE
The appendix is structured as follows:
Appendix describes the extended related work;
Appendix focuses on detailing the experimental setting our work. Specifically, we further describe the training configurations and pipeline (Appendix [B.1. the metries employed to evaluate the proposed methods (Appendix B.2). the formal definitions of the baseline loss functions (Appendix B.3), the procedure developed to test for presence or absence of mode connectivity (Appendix [B.4), the datasets considered for our experi- ments (Appendix B.5). the pruning ratio per iteration (Appendix [.6). and the hyperpa- rameters our proposed procedure (Appendix (B.7))
Appendix C focuses on providing additional experimental results to support the find- ings in the main text. We report extended confusion matrices for individual submod- ules without (Appendix C.II and with pruning (Appendix C.2). as well as for pairwise (Appendix C.3) and incremental merges (Appendix C.4 We also include loss land-
scape plots from the mode connectivity analysis (Appendix C.5 and a comprehensive table comparing merging results with and without pruning across all merge strategies (Appendix C.6). as well as full baseline comparison including Confidence Penalty and Label Smoothing (Appendix C.7). Finally, further analyze possible extension of our framework, we present experimental results on Imagenette, demonstrating generalisation to large-scale image datasets (Appendix C.8). experimental results on CIFAR- 100, prob- ing the limits of compositionality with up to 100 merged modules (Appendix C.9). and experimental results on text-clasiification benchmarks (IMDB and 20 Newsgroups), show- ing that the method transfers beyond image classification (Appendix C.10
15

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Modular Neural Networks and Interpretability. Several works have explored modular neural architectures that learn decompose computation across tasks (Kirsch et al. 20188] Han al.. [021] Salem et al. 2023). In particular, KKircch et al. (2018) introduced a method to jointly learn both the modules and their composition end-to-end, showing interpretable specialization through controller- based selection. However, their approach does not isolate class-specific knowledge: modules are reused across multiple tasks and selected based on input context, making it difficult to extract inde- pendently meaningful functional units.
In contrast, our work explicitly aims to construct class-specific functional modules, where each module is trained to specialize in a single class and produce maximum-entroe outputs for all oth- ers. This sharp functional separation not only promotes compositionality via weight merging. but also improves interpretability; submodules can be understood in terms of their response to a single concept. Unlike post-hoc analyses, we impose structure during training, yielding functional units with clear class semanties and compositional behavior. This aligns with the goals of mechanistic interpretability (Saphra and Wiegreffe| 2024), where subnetworks (or "circuits" " (Olah et al. 2020)) analyzed as self-contained structures responsible for identifiable sub-tasks.
Our method can be seen as a complement to neuron-level analysis techniques such as those in Mu and Andreas (2020), which decompose single neuron activations into logical or perceptual compo- nents. However, such analyses often suffer from polysemanticity- same neuron responding to unrelated stimuli-limiting their explanatory power. Instead, we enforce functional modularity at the network level, allowing the identification of interpretable, composable subgraphs. This modular decomposition also conceptually connected to recent work on harmonics of learning (Marchetti iet al. 2024), where symmetry structures constrain the function space; our class-specific modules may correspond to symmetry-aligned components in this broader framework.
Maximum Entropy Principle. The principle of maximum entropy (MaxEnt), introduced by Jaynes (1957), asserts that when making inferences under partial knowledge, one should prefer the probability distribution that satisfies known constraints while remaining otherwise maximally uninformative. Originally formulated to connect statistical mechanics and information theory, Max- Ent has since become foundational tool in probabilistic modeling and statistical inference (Kuić, 2016 De Martino and De Martino, 2018). Its generalization to complex-valued functions has also been explored, for example image reconstruction tasks (Bajkova 1992).
In our setting, MaxEnt serves as a guiding principle for functional specialization: we train submod- ules to be highly confident only on their assigned class and maximally uncertain (i.e., high-entropy) elsewhere. This interpretation differs from standard entropy regularization, which is typically used to smooth predictions or calibrate confidence scores (Marezak et al. (202)). Rather than encour- aging mild uncertainty. our maximum-entrop loss enforces uniformity outside the rewarded class set actualiziin the MaxEnt principle to promote class exclusion and module isolation. To our knowledge, this use of entropy as a strict compositional prior is novel within the context of modular deep learning.
Lottery Ticket Hypothesis and Sparse Training. Our approach builds on the Lottery Ticket Hy- pothesis (LTH), which states that sparse subnetworks in randomly initialized networks can be trained to match the performance of the full dense model (Frankle and Carbin 2018 Zhou et al.) 2019 Liu eet al. 2024). This idea has been extended theoretically to show that such subnetworks exist with high probability even in convolutional architectures with ReLU activations (Burkholz (022). Em- pirical studies also demonstrate that global unstructured pruning outperforms structured pruning in high-sparsity regimes (Girish et al. 202T). We adopt global pruning in our setting to retain the most functionally relevant connections across the entire network. LTH has been further explored in transfer learning (Van Soelen and Sheppard, 2019; Burkholz et al. (022) and federated learning (Itahara et al.) 2020). In contrast to prior work that uses pruning primarily for compression or trans- fer, we leverage it to enforce functional isolation: pruning removes spurious capacity, encouraging submodules to specialize only on their rewarded classes.
Mode Connectivity and Model Merging. Recent work on model merging has explored how in- dependently trained networks can be combined through linear or non-linear paths in weight-space,
16

---

Published as a conference paper at ICLR 2026
property known mode connectivity (Ilharco al. 2023) This has inspired techniques for task arithmetic (Ortiz-immnnez et al., 2023; Yang et al., 2024 Wang et al. 2024), which define task vec- tors differences between fine-tuned and base models, enabling operations like unlearning and multi-task interpolation. These methods typically assume large pre-trained models and alignment in parameter space, and focus editing rather than modular design.
Other approaches, such as Git-Rebasin or PLeas (Hazimeh et al. 2024, Zeng et al. 2023 ) address the challenge of merging models with different initializations techniques focus on how to combine models after training, our method focuses on how to con- struct modules that are composable by design. Specifically, we show that submodules trained with -entimmm-eetropyy entropy loss and pruning often exhibit linear mode waweecttvityyalllwweee for effective merging via weight summation-whi also identifying cases where destructive interference occurs.
weight alignment. While these
maximum
Our results suggest that mode connectivity may emerge naturally when modules are trained under entropy-based -based functional isolation, without requiring explicit alignment. This positions our work as complementary to task arithmetic and merging strategies: we do not assume pre-trained baselines, instead provide a procedure to build sparse, specialized components that are compatible by construction.
17

---

Published as a conference paper at ICLR 2026
B EXTENDED EXPERIMENTAL SETTING
B. TRAINING CONFIGURATION AND PIPELINE
We report here the key training configurations used in our experiments. All models were trained using the maximum-entropy loss (ME) and iterative magnitude pruning (IMP), with two pruning iterations and final sparsity of 99% for MLPs and 60% for CNNs. unless stated otherwise. The optimizer used was Adam. We use the highest accuracy achieved on the validation set our check- pointing strategy. All experiments are conducted on workstation equipped with an Intel Core 19-10940X (14-core CPU running at 3.3GHz), 256GB of RAM, and a single Nvidia RTX A6000 GPU with 48GB of VRAM.
Temperature scaling for the softmax was fixed to t. Inputs were normalized when appropriate, and all datasets were used with standard chining/validation splits. Each experiment was run with random seeds.
All YAML configuration files are released alongside the code repository for full reproducibility.
Model Architectures and Hyperparameters
Common Settings. All models use ReLU activations, are trained with a batch size of 64, and are evaluated using the same training and pruning pipeline.
Table 3: Architecture configurations for the MLP and CNN models used in our experiments, includ- ing hidden layer sizes, dropout rate, and the temperature scaling for the softmax f.
Model
Hidden Layers
Dropout
Shallow MLP
[512]
0.5
5.0
Deep MLP
[512, 256]
0.5
5.0
CNN (LeNet-style)
Conv [120, 64]
1.0
The learning rate was set 10-3 for MNIST and FMNIST across all architectures, and to 10-4 for Yeast and Human Activity Recognition, where only Shallow and Deep MLPs were evaluated.
Dataset-specific Input Dimensions and Classes
MNIST / FMNIST: 784 input features (flattened 28 28 images), 10 classes.
Human Activity Recognition (HAR): 561 input features, 5 classes
Yeast: input features, balanced subset of classes.
Training Pipeline. Here we present an overview of generic experiment. For each experiment:
Given the set of classes C in the dataset, choose subset classes C.
Train one subnetwork module per class or per subset R. C C', .usinn ME and IMP.
Evaluate each subnetwork using:
Accuracy rewarded classes;
Mean entropy on non-rewarded classes;
Confusion matrix on the validation set.
B.2 FORMAL DEFINITION METRICS
Accuracy Rewarded Classes. Let R C C be the set of rewarded classes, and let DR denote the
subset of validation data whose labels belong to R. The accuracy over rewarded classes is defined
18

---

Published as a conference paper at ICLR 2026
Accuracy rewarded
Σ
1[((x)
IDR val (x.y))
where 1|:] is the indicator function. This measures how well the subnetwork performs on the class subset it is trained to recognize.
Mean Entropy on Non-Rewarded Classes. Let D be the validation subset containing sam- ples from non-rewarded classes. We define the average entropy of the model's predictions over this set as:
Entropy non-rewarded
Σ H (ū(x))
(8)
D
DEDOOR
where ((x) is the predicted probability distribution for input x, and (p)) denotes the entropy of a distribution p, defined as:
C
H()) = Pt log Ps
This metric quantifies the uncertainty the model cxhibits inputs it is not supposed classify, with higher values indicating closer adherence to the maximum-entropy principle.
Confusion Matrix. Given a validation dataset Daal and a trained model for we define the confusion matrix M Nicxia such that each entry Mij counts the number of samples with true label that are predicted as class j:
Myj
Σ 1 -1= (
(10)
where ((x) = arg max; (x) is the predicted class. This matrix allows us to qualitatively inspect specialization and interference across class predictions.
B.3 FORMAL DEFINITION OF BASELINES
Quasi-MaxEnt loss (QME): loss function that non-rewarded classes ignores the warded classes and produces uniform distribution solely on non-rewarded classes. For- mally referring equation]
0: = d 6 [(
y E R, otherwise.
(11)
CrossEntropy loss (XE): the standard loss used in standard classification tasks. this case, we do not modify the labels, which remain standard, but the model is exposed only to the classes in R, rather than all available classes.
XE with Label-Smoothing (A = 0.1), which replaces the one-hot target distribution with smoothed version
1 X, if y,
.S yk
otherwise.
K-1
XE with Confidence-Penally, which adds rntropyyregallarizttoo term
L XF(p.v) (p))
19

---

Published as a conference paper at ICLR 2026
B.4 ADDITIONAL EXPLANATIONS ON MODE CONNECTIVITY
To further shed light on why merging via summation works (as demonstrated in the main paper) for simple datasets, we study also the loss landscape through the lens of mode connectivity.
Mode connectivity refers phenomenon observed in the loss landscape of neural networks. When a neural network is trained, the optimization process (e.g. SGD) typically finds a set of weights that minimizes the loss function. This set of weights is called mode high-dimensional loss landscape. Several contributions (Frankle et al. 2020) Lubana et al. (002)) pointed out that these different modes (solutions found by, e.g., training the same model architecture multiple times with different initializations or training procedures) are often not isolated: rather, they can frequently be connected by paths along which the loss value remains consistently low.
local minimum in the
Many paths of different shapes can connect two models, 01 and 02, in weight-space. We are specifically interested in testing for a path without significant loss barriers that includes the model (1.22] = +2. which results from merging the original models via summation. A key aspect of our setup, diverging from typical mode connectivity studies, is that 0, and 02 are initially specialized for different sets of rewarded classes. The merged model [1222 is intended to operate on the union of these class sets. Consequently, we evaluate for barriers using our maximum--ntrup loss function. This loss function is consistently applied across the path and for the endpoint models (01, f2), and it considers the full union of rewarded classes relevant to 11.21-
Formally, in line with Lubana et al. 2202),, let 01 and 0g be two sets of weights, D dataset, fo a neural network parametrized by weights 6, and L our specific maximum maximum-entrropy entropy loss function (evaluated D considering the union of rewarded classes as described above). We say that Q1 and 02 are mode connected along the path 701-12 (1) if AI E [0, 1] [(J
((Ta,,_,, -a, (1) (D)) < (1 -1))- 1)-
C(fo, +t- (D)) + where E is small margin, set to 2% of the first term on the r.h.s. following Frankle al. (2020).
Each point along the path -0,-0, (t) represents a valid set of weights for the network f(-). Stan- IMC
dard linear mode connectivity considers the path 70,-88 -8, (t) (1 t) 01 + 62 For our case, we require a path -0,-+0;()) (t) that satisfies: (i) for 70, -0, (0) 0,. (ii) for 70, -0, (1) = 02 and (iii) 3t E j0, s... s.t. 70,-0, (t') = 0 + 02 As a consequence, we define the mode connectivity with respect the following piecewise linear path:
0 + 0g 2((1 f)
if S 0.5
Yo, -0, (t)
(12)
if > 0.5
It is trivial to see that definition fulfill our properties, especially (iii) for 0.5.
B.5 DATASET DETAILS
MNIST and Mashion-MNIST. Both are standard 10-class image classification benchmarks with grayscale 28 x 28 images. We normalize pixel values to [0, 1] and use the standard train/validation splits 6,,000 train, 10,000 test).
Human Activity Recognition (HAR). This dataset contains sensor data from smartphones, la- beled with six different physical activitics. Each example has 561 standardized features. We adopt the original train/test split (7,352 train, 2,947 test) and reserve validation portion from the training set.
Yeast. A protein localization dataset with 10 classes and 8 features. We normalize each feature to zero mean and unit variance, and split the dataset using a 65/15/20 ratio (on 1,299 samples). Since the class distribution is highly imbalanced, we focused on the most represented 4 classes (i.e., CYT, NUC, MIT, ME3).
Preprocessing. For all tabular datasets, we apply z-score normalization and encode labels as inte- gers. We use early stopping the validation set and batch sizes of 64 during training.
20

---

Published as a conference paper at ICLR 2026
B.6 ITERATIVE PRUNING SCHEDULE DERIVATION
We follow the Iterative Magnitude Pruning (IMP) framework, in which neural network is pruned multiple iterations. At each pruning step, fraction K of the remaining weights (those with the smallest absolute value) is removed. This process is repeated for N iterations until target sparsity level P (0, 1) reached, corresponding retaining only a fraction of the original parameters.
Pruning Rate Schedule. Let St denote the fraction of weights remaining after the i-th pruning iteration. The pruning process is multiplicative, meaning that:
S (1 K)';
assuming So = 1 (i.e., all weights are initially present). After N pruning iterations, desire final sparsity P. which implies a remaining fraction SN = P. Therefore, we solve:
(1 - K) = P.
Solving for K, we obtain the pruning rate per iteration:
= -
This schedule ensures that pruning a fixed fraction K of the remaining weights each of the N steps results in an overall sparsity of P at the end of the iterative process.
Pruning Algorithm. Algorithm
2
summarizes the full procedure used in our experiments.
Algorithm 2 Iterative Magnitude Pruning with Maximum Entuuop Loss
Require: Initial weights lo dataset D, rewarded classes R, number of pruning iterations N, target
sparsity P, epochs per iteration E
1: <- 0g
2: K + (1 P)1/N
3: for iteration i = to N do
4: Train model fe on using maximum -nntuppppee loss with reward set R for E epochs
5: Prune the fraction K of weights in 0 with the smallest absolute value
6: Reset remaining weights
their initial values from 0g
7: end for
8: Train the final pruned subnetwork on D using the maximum-entroop loss with R. for E epochs
B.7 HYPERPARAMETER TUNING
We perform a small-scale hyperparameter study with respect to two key components the training pipeline: the softmax temperature used in the maximum-entropy loss, and the pruning ratio applied iterative magnitude pruning. The temperature analysis is conducted on MNIST using a shallow MLP in the complete merge setting. For the pruning ratio, we provide one on the shallow MLP in the same complete merge scenario, and one on CNNs (LeNet-style) in the pairwise merge setting, using both MNIST and FashionMNIST. All results are averaged over random seeds.
complementary studies:
Temperature. We study the effect of the temperature hyperparameter t E {1,3, 5, 7}. which scales the logits before the softmax activation and thus controls the sharpness of the output dis- tribution. Lower temperatures produce more peaked distributions, while higher values encourage uniformity.
Figure 7a and 76 show the average entropy on non-rewarded classes and the rewarded accuracy across merge steps for different temperatures.
21

---

Published as a conference paper at ICLR 2026
100
120
(a) Entropy non-rewarded samples at each incre- mental merge step for different temperatures.
(b) Rewarded accuracy at each incremental merge step for different temperatures.
Figure 7: Effect of temperature scaling on specialization and composability. Temperature controls the confidence of the softmax output; t 5 achieves good trade-off between high entropy on non-rewarded classes and accuracy on rewarded ones.
Pruning Ratio. We evaluate the impact pruning ratio on submodule performance and merge behavior for two architectures: shallow MLP and CNN (LeNet-style). For the shallow MLP, focus on the complete merge setting, testing three final sparsity levels-90%. 99%, and 99.9% justed across IMP iterations. As shown in Figure 88, pruning to 99% yields the best trade-off between specialization and compositionality: preserves rewarded accuracy throughout the merge process. Lower sparsity (90%) retains excessive capacity, causing interference when modules are merged. Higher sparsity (99.9%) leads to underfit- ting and degraded performance.
maintains high entropy on non-rewarded inputs and
For CNNs, we instead study pruning in the pairwise merge setting, focusing on class subset sizes [R] 2 and across MNIST and FaShionMNIST. The results, summarized Figure show that pruning ratio of 0.6 consistently yields the highest average rewarded accuracy across both datasets and cardinalities. This indicates that, unlike MLPs, CNNs benefit from less aggressive sparsification.
(a) Entropy non-rewarded samples across merge steps for different pruning ratios.
(b) Rewarded accuracy across merge steps for differ- ent pruning ratios.
Figure 8: Effect of pruning ratio on submodule specialization and merge stability. A pruning ratio of 0.99 yields good balance between maintaining accuracy and increasing entropy on non-rewarded classes.
22

---

Published as a conference paper at ICLR 2026
0.9
Accuracy
0.8
0.7
0.6
Dataset
Cardinal
MNIST
FMNIST
0.5
0.5
0.7
0.8
1.0
Prune Ratio
Figure 9: Effect of pruming ratio on CNN accuracy after pairwise merging on MNIST and Fashion- MNIST, with cardinality [R] and The optimal pruning ratio is 0.6.
23

---

Published as a conference paper at ICLR 2026
C EXTENDED EXPERIMENTS
In this section, we present additional experimental results and qualitative insights.
First, report representative examples for three settings:
Shallow MLP on MNIST
Deep MLP on Human Activity Recognition
CNN (LeNet-style) on FashionMNIST
Then, we provide the following extended results, each complementing the findings reported in the main paper:
a comprehensive set of tables and figures reporting the full pairwise submodule merging model architectures, pruning settings (with and without IMP), merge strategies (logit vs. weight), cardinalities (R], and loss functions (XE, ME, QME). These extend the main paper, where only pruned results
results across all configurations-varyyee
shown (Ap-
pendix C
a complete baseline comparison including Confidence Penalty (CP) and Label Smoothing (I.S), showing that ME is not re- parpaettrristioon of standard regularisers (Appendix C.7
pairwise and complete merge results on Imagenette, demonstrating generalisation to large- scale image data (Appendix C
pairwise and complete merge results on CIFAR-100 with up 100 modules, stress-testing the limits of compositionality (Appendix C.9
pairwise merge results on IMDB and 20 Newsgroups, showing that the framework trans- fers beyond vision tasks C.pendix(C.10))
C.1 CONFUSION MATRICES (No PRUNING)
In this subsection, we show submodules trained using the maximum-entrropy loss without applying iterative magnitude pruning (IMP). For each model-dataset pair, we show the confusion matrix of submodule trained on a single rewarded class. These visualizations allow to assess the degree of specialization and the presence residual structure in the predictions for non-rewarded classes.
In all cases, the models exhibit strong activation the rewarded class, while showing varying degrees of non-uniformity on the excluded classes especially evident in the confusion matrix rows corresponding to non-rewarded labels.
24

---

Published as a conference paper at ICLR 2026
Figure 10: Confusion matrices (total predicted counts per class) for submodules trained on classes 0-9 without pruning, using the maximum-entree loss. Each model is trained to specialize on one class and suppress predictions on others. Results refer to the Shallow MLP architecture applied to MNIST dataset.
Confusion matrices (total predicted counts per class) for submodules trained on classes 0-4 without pruning, using the maximum-entropy loss. Each model is trained to specialize on one class and suppress predictions on others. Results refer to the Deep MLP architecture applied to the HAR dataset.
Figure 11:
25

---

Published as a conference paper at ICLR 2026
Figure 12: Confusion matrices (total predicted counts per class) for submodules trained on classes 0-9 without pruning, using the maximum-entropy loss. Each model is trained specialize on one class and suppress predictions on others. Results refer to the CNN architecture applied to the FMNIST dataset. The CNN was pruned with 99% pruming ratio.
26

---

Published as a conference paper at ICLR 2026
C.2 CONFUSION MATRICES AFTER PRUNING
We report confusion matrices for submodules trained with both the maximum-entropp loss and iter- ative magnitude pruning (IMP). These models are expected exhibit stronger functional isolation, with reduced leakage across non-rewarded classes. Comparisons with Section C. highlight the impact of pruning on output uniformity and specialization.
Figure 13: Confusion matrices (total predicted counts per class) for submodules trained on classes 0-9, using the maximum-entree entropy loss and pruning. Each model is trained to specialize on one class and suppress predictions on others. Results refer to the Shallow MLP architecture applied to the MNIST dataset.
Figure 14: Confusion matrices (total predicted counts per class) for submodules trained on classes 0-4, using the mexximum-entropp loss and pruning. Each model is trained to specialize on one class and suppress predictions on others. Results refer to the Deep ML. architecture applied to the HAR dataset.
27

---

Published as a conference paper at ICLR 2026
Figure 15: Confusion matrices (total predicted counts per class) for submodules trained on classes 0-9, using the mexximum-entrope loss and pruning. Each model is trained to specialize on one class and suppress predictions on others. Results refer to the CNN architecture applied to the FMNIST dataset. The CNN was pruned with 99% pruning ratio.
28

---

Published as a conference paper at ICLR 2026
C.3 CONFUSION MATRICES FOR PAIRWISE MERGES
We show representative confusion matrices for pairwise merges of submodules, each specialized on a distinct class. These examples illustrate that, even after weight summation, the merged model re- tains high confidence on rewarded classes and preserves uniform predictions on others confirming compatibility and lack of interference.
Figure 16: Confusion matrices (total predicted counts per class) for 16 pairwise merges of Shallow MLP submodules on MNIST. Each cell shows the average across seeds.
29

---

Published as a conference paper at ICLR 2026
Figure 17: Confusion matrices (total predicted counts per class) for 10 pairwise merges of Deep MLP submodules on HAR. Each cell shows the average across 5 seeds.
124
Figure 18: Confusion matrices (total predicted counts per class) for 16 pairwise merges of CNN submodules on FMNIST. Each cell shows the average across 5 seeds. The CNN was pruned with 99% pruning ratio.
30

---

Published as a conference paper at ICLR 2026
(a) Card.
(b) Card. 2
(c) Card. 5
Figure 19: Confusion matrices (total predicted counts per class) for pairwise merges of CNN sub- modules on FMNIST, for exemplary modules of cardinality 1, 2, and 5 each. Each cell shows the average across 5 seeds. The CNN was pruned with 99% pruning ratio.
31

---

Published as a conference paper at ICLR 2026
C.4 CONFUSION MATRICES FOR INCREMENTAL MERGES
This section presents confusion matrices from various stages of incremental merging, where multiple submodules are combined by one. The results demonstrate how compositional behavior is maintained across merge steps, and how the model continues to correctly isolate rewarded class behavior while suppressing predictions on excluded ones
(a) Step
(b) Step 2
(c) Step 3
(d) Step
(c) Step
(f) Step 6
(g) Step?
(h) Step
Step 9
ii Step
Figure 20: Confusion matrices (total predicted counts per class) for incremental merge steps of a Shallow MLP on MNIST. Each submodule trained independently and merged following in- creasing order. Results averaged over seeds. The structure of predictions remains clean and interpretable throughout the merge.
(a) Step 1
(b) Step 2
(c) Step 3
(d) Step
(c) Step 5
Figure 21: Confusion matrices (total predicted counts per class) for incremental merge steps of Deep MLP on HAR. Each submodule is trained independently and merged following increasing order. Results are averaged over seeds.
32

---

Published as a conference paper at ICLR 2026
(a) Step
(b) Step 2
(c) Step
(d) Step
(c) Step
(I) Step 6
(g) Step 7
(h) Step
(i) Step 9
(i) Step 10
Figure 22: Confusion matrices (total predicted counts per class) for incremental merge steps of CNN FMNIST. Each submodule is trained independently and merged following increasing order. Results are averaged over seeds. The CNN was pruned with a 99% pruning ratio.
33

---

Published as a conference paper at ICLR 2026
C.5 MODE CONNECTIVITY
In this section, we present additional plots analyzing the loss landscape between merged submod- ules. Following the framework described in Appendix [.4 evaluate the relative difference in loss along the piecewise linear path connecting two modules and their sum.
We show incremental merging results for Deep MLP on the HAR dataset and for a CNN on FMNIST, for both incremental and pairwise merging settings, In the majority of the cases, the pictures describe the desired behaviour (no or limited increase above 0). As already mentioned in the main paper, it can also be observed that for the CNN, after many incremental merges (e.g., steps and 10), the difficulty in achieving an optimal merge is reflected in the presence of barriers in the loss function landscape.
Y
Loos
Card. 1
(b) Card. 2
(c) Card. 5
Figure 23: Values for "0, -0 (t) for pairwise merges of CNN submodules on FMNIST, for exem-
plary modules of cardinality 1, and 5 each (same Figure T9p. Results are averaged over 5 seeds. The CNN was pruned with 99% pruning ratio.
Loss 56
-25
25
1,00
(a) Step
(b) Step 3
(c) Step 4
(d) Step 5
Figure 24: Values for - 0, (t) for incremental merge steps of Deep MLP on HAR (same as Fig- 21 Each submodule trained independently and merged following increasing order. Results are averaged over 5 seeds.
V
Loss
La3s
Loss
-10
40
20
20
(a) Step
Stop
Step
Step
Step
Loss
losk
-20
-00
(f) Step
(g) Step 8
(h) Step 9
i Step 10
Figure 25: Values for 70 (t) for incremental merge steps of a CNN on FMNIST (same as Fig- ure 22 Each submodule trained independently and merged following increasing order. Results are averaged over 5 seeds. The CNN was pruned with a 99% pruning ratio.
34

---

Published as a conference paper at ICLR 2026
C.6 COMPLETE RESULTS FOR PAIRWISE SUBMODULE MERGING
In the main paper (Section 4) we report the results of pairwise submodule merging under the Iter- ative Magnitude Pruning (IMP) regime, this generally led to the best overall performance. For completeness, here we provide the full set of results, including both pruned and non-pruned con- figurations. Table 2 reports the average rewarded accuracy and average entropy on non-rewarded classes for all model families (Shallow MLP, Deep MLP, CNN), across multiple datasets (MNIST, FMNIST, HAR, Yeast), cardinalities |R] E 1, 2, 5, and loss functions (CrossEntropy, Quasi-MaxEin and MaxEnt).
We observe that:
Models trained with IMP consistently outperform their non non-pruned counterparts in most settings, particularly when using the MaxEnt loss.
The MaxEnt loss yields the most reliable and composable modules across all model types and datasets, both with and without pruning.
Non-pruned models trained with standard CrossEntropy struggle to maintain entropy on non-rewarded classes, often resulting in degraded compositional performance.
The benefit of pruning especially marked in deep MLPs, where without IMP the merging process suffers from significant degradation in rewarded accuracy.
These extended results support the conclusions drawn in the main paper: pruning strengthens func- tional specialization and enhances composability, especially when combined with entropy-based objectives.
35

---

Published as a conference paper at ICLR 2026
Table 4: Performance of meirwise-mergen sub-modules with Shallow MLP and Deep MLP on FM- NIST, MNIST, HAR and Yeast, merged either via logit averaging or weight summation, across car- dinalities (R], and loss functions: CrossEntropy (XE), MaxEnt (ME), and Quasi-MaxEat (QME), Results are averaged over 5 seeds and 10 pairs.
with and without Iterative Magnitude Pruning (IMP))Results
FMNIST
MNIST
HAR
Yeas!
Model
[
IMP
Ioss
Merge
Eatropy
Rewarded Aue
Esropy
Rewarded Auc
Encrugy
Rewarded Aue
Entropy
Rewarded Ave
XF
Logt
Weight
0.045 jurss
0.308 au:99
0.852 (umm
0.001 10:00
0.580.2.mm
0,806 autt
0,992 surn
0.518 0:39)
0,582 mete
0.892 aum
0.811 wms
0.176 y1dle)
1099 sum
0.559 va 1400
0.692 (e:44))
ME
Logit Weight
2.295
0:988 (2.07)
2:299
2,000
suod)
1,512 (30t)
(.81))
wun
0.862 (8:15)
0,829
QME
Lagit Weight
2.035
2,107
2,040
2.105
0.974
0.955
0 p1e0:
1.20
0.792
0.792
XB
Laya Weight
(1266
0.561
0.813 3m:
0.859
0.452
0.667
0.679 (1:0)
0.758 1m
0.8%6 0:01)
0.571
0.883
0.566
1 1.364 pHo1:
1.37
0.667
0,660 (5:96
Yos
MB
Logit Weight
2,294 Juxta
2.279
0,980 (3850)
0.983 (2429)
2,288
2.299 (300))
0.992 mum
0.991 auoe)
1,590 od11
1,555 (.22)
0.982 wud
0.983 wuow
1.366 sunts
1.257 suoe:
0,545 (8:11))
0.546 (8.11))
QME
Logit Weight
2,109 auss
2.047 suz2
0.568 (1.4))
0.962 (uas))
2 (0.04)
2,102 (1.09)
0.954 suoes
0.973 sut
1,183 (123))
1,076 (0256)
0.554 mum)
0.555 muzs
1,300 suoe
1,366 suot:
0.734 (8:22)
0,737 (8152)
XF
Lage Weight
0.064 jomv
0.685 9119
0.555
(116)
0.133
771 aoohh
0.356 0:30)
0.152 mm
0.918 sum
0.773
0,437 1e/00
Shallow MLP
ME
Logt
Weight
2.785
0,549
QMB
Lopt Weight
2.108
1.999
0,955
0
2,699
1,66
0.967
0.954
0,286
1,129
0.958
0.929
0,583 (e
0,556
XB
Loga Weight
0.309
D.610 pures
0,813
0.777
0.334 (1:00)
0.852 wucc
0.831 (000
0.691 (140))
0.411 (0310)
11:70 ausic
0.864 auss
0.449 (a:
0,419 (8:31))
Yes
MR
Lagat Weight
2,283 puan
2.252 juic
0,952 (uass
0.060 att
2.364 (500))
2,200
0.983 sum
0.981 sumss
1,509 pzin
1,568 9219
(1944 susee
0.945 muse
0.54p ca.
0.565 velach
QMB
Logt Weight
2.114 sunt
1,773 (434)
0:955 dazz
0:918 (a00)
2.101 (a01))
1,688 (1103)
0.982 summ
0.963 suis))
0.577 (034)
1,14 oaea
0.937 auooe
0.919 suoe
0,424 (0:10)
0,475 (1/21)
XE
Logit Weight
0:451 (1.00)
0.516
susse
ME
Weight
0.683
0.974
QME
Woight
Laga
0.877
0,713
0.891
0.972 avom
XB
Logt
Wright
0,665 niatts
0.434 suo
0.831 ave3)
0.855 auoe
Yes
MB
Logit
Weight
0.567 duao)
0,827 daas
0.945 summ
0.952 summ
QMF
Lugit
Weight
0.965 u.ao))
sum
XF
Logit
Weight
0.151 1475
0.016
0.566 aa100
0,420
eveth
0,218
oatb
0.797 1124
wum
0.19 puen
0.724 (8:00)
0,509
MB
Logit
2
2.294
0,982
0,586
2,300
2,300
0.833
0.994
1,500 0013
0.977
0.979
1.399
0,859
0,850
QMB
Weight
Lagia
2.108
1,836 (4153
0:060 ala
0.91
2 ma0rs
1,995 (1.14)
0.953 gimw
0.975 auz2
0.792 (0179)
1.156 o190
0.965 muo
(1949 aim))
0.594 2113)
1.356 yomr:
0,787 (8.00)
0.777 (e:0
XF
Laght Weight
0.460 yuaru
0.216 suren
0. 8,88 1na
0.813
0.656 (116
0.486 1 1466
0,834 (3
0.688 a0ci
0.711 pmm
0.543 pmmm
0.840 mm0
0.887 1w
1 1.88 yomr
1.376 pino
0.674 (o/w
0.689 1 Wi
Yes
MB
Laga Weight
2.235 yiir
0:973
0:978
2.901 so
2.204 (0:00
0.991 siom
0.992 m000
1,081 (140))
0.981 miws
0.981 mres
1.35 pump
0.819 18:1
0.844 18:17)
QMB
Logit
Weight
2,108
1.844
0:956 (3.09)
0.558 (1.52)
2,095 (8:00))
2,002 (8:23)
(uom)
0,779 (22)))
(224))
0.923 wei
suan
sum
0,900 (8:13)
0,907 (8:12))
XF
Logit
00066
0.271
0.655
0.133
0.564
0.799
0.580
0.360
0.794
0.665
0.527
0,438
Deep MLP
MF
Logt Weight
2.291
2.264
0,966
0,796
2,305
2
0.685
0.989
1,219
1,569
0.935
0.948 wunt)
0.619 ventts
0,619
QMB
Logt
Weight
2.113 su
1,960 94284
0,954 mal0
0,962 (M
2,107 (0009
1.825 (8155)
0.956 aum
0.900 auth
0,238 9m
1,134 0:25)
0.949 sume
0.919 veria
0,001 (0:221>
0,582 (e:11
XB
Logt
Weight
01435 a119
01226 9x114)
0,748 m.z2
0,790 dame
0.498 (1.00)
0.194 (5:99)
0.841 Jume
0.724 susen
0,400 0229)
0,601 0336)
0.857 auss
0.821 wures
0,372 (9:337)
0,400 (eu211
Yes
ME
Lagia
Weight
2.225
2.290 poics
0:055 aa
0:930 asa
2.372 1800
2.208 saa00
0.953 m00m
0.672 (55
0.870 aaree
0.943 aarocc
0.508 (8-2150
0.608 (9:1110
QMB
Weight
Lagit
2.118
1.347
0.546
0.910
00:455
0.
XBI
Weight
Logit
0,586
0,556
No
ME
Logit
Weight
0,582 (lad))
0,485 (lac)
0.975 auoi)
0.639 auo-1)
5
QME
Logtt
Wright
0.681 amam
0:001 maa
0.784 avees
0.975 auom
XB
Logit
Weight
0,624 casso
0.500 aso
0.687 ause
0.290 aumm
Yes
ME
Logit
Weight
suson
QME
Weight
Lugit
36

---

Published as a conference paper at ICLR 2026
Performance of pairwise-merged sub-modules with CNN on FMNIST and MNIST, merged either via logit averaging weight summation, across cardinalities [R], and loss func- tions:
Table 5:
CrossEntropy (XE), MaxEnt (ME), and Quasi-MaxEnt (QME), with and without Iterative Magnitude Pruning (IMP). Results are averaged over seeds and 10 pairs.
FMNIST
MNIST
Model
R
IMP
loss
Merge
Entropy
Rewarded Acc
Entropy
Rewarded Ace
Logit Weight
0.240 (0.204)
0.588 (0.158)
0.338 (0.262)
0.517 (0.83)
0.045 (0.097)
0.592 (0.175)
0.077 (0.172)
0.517 (0.064)
No
Logit Weight
2.296 (0.003)
0.987 (0.021)
2.301 (0.000)
0.997 (0.02))
ME
2.226 (0.107)
0.950 (0.056)
2.102 (0.273)
0.977
QME
Logit Weight
2.106
(0.010)
0.969 (0.028)
2.097 (0.006)
0.992 (0.003)
1.272 (0.324)
0.906 (0.098)
1.210 (0.301)
0.970 (0.047)
Logit Weight
0.151 (0.151)
0.633 (0.182)
0,296 (0.254)
0.539 (0.103)
XE
0.022 (0.043)
0.576 (0.158)
0.023 (0.104)
0.520 (0.070)
Yes
Logit Weight
2.294 (0.004)
0.983 (0.024)
2.301 (0.001)
0.997 (0.002)
ME
2.189 (0.112)
0.966 (0.037)
2.021 (0.357)
0.984 (0.020)
Logit Weight
2.091 (0.009)
0.955 (0.031)
2.085 (0.003)
0.986 (0.007)
QME
1.096 (0.287)
0.896 (0.131)
0.969 (0.371)
0.952 (0.081)
Logit Weight
0.354 (0.164)
0.603 (0.122)
0.630 (0.165)
0.753
XE
0.081
(0.050)
0.664 (0.164)
0.095 (0.050)
0.711 (0.130)
No
Logit Weight
2.287 (0.004)
0.973 (0.017)
2.299 (0.001)
0.995
CNN
ME
2.147
(0.093)
0.894 (0.065)
1.981 (0.204)
0,948 (0.040)
Logit Weight
2.110 (0.013)
0.964 (0.017)
2.094 (0.006)
0.993 (0.002)
QME
1.188 (0.318)
0.909 (0.055)
0.743 (0.376)
0,936 (0.061)
Logit Weight
0.164 (0.126)
0.568 (0.110)
0.438 (0.165)
0.675 (0.139)
XE
0.047 (0.037)
0.630 (0.162)
0.062 (0.052)
0.638 (0.135)
Yes
Logit Weight
2.281 (0.007)
0.972 (0.017)
2.297 (0.002)
0.994 (0.003)
ME
2.055
(0.190)
0,926 (0.017)
1.649 (0.466)
0,964 (0.040)
Logit Weight
2.089 (0.010)
0,959 (0.017)
2.083 (0.003)
0,992 (0.003)
QME
0.694 (0.372)
0.898 (0.065)
0.366 (0.260)
0,945 (0.054)
Logit
0.632 (0.045)
0.830 (0.041)
XE
Weight
0.462 (0.034)
0.708 (0.087)
No
Logit
0,914 (0.002)
0.988 (0.001)
ME
Weight
0.803 (0.055)
0.915 (0.042)
Logit
0.908 (0.002)
0.987 (0.001)
QME
Weight
0.767 (0.043)
0.884 (0.043)
Logit
0.625 (0.052)
0,901 (0.025)
XE
Weight
0,493 (0.144)
0.776 (0.061)
Yes
Logit
0,915 (0.002)
0.988 (0.001)
ME
Weight
0.791 (0.083)
0.943 (0.034)
Logit
0.912 (0.004)
0.986 (0.000)
QME
Weight
0.790 (0.04))
0.909 (0.034)
37

---

Published as a conference paper at ICLR 2026
Table 6: Performance of serwise--erged sub-modules with ResNet18 FMNIST, MNIST and CIFAR-10, merged either via logit averaging or weight summation, across cardinalities [R] and loss functions: CrossEntropy (XE), MaxEnt (ME), and Quasi-MaxEnt (QME), with and without Iterative Magnitude Pruning (IMP). Results are averaged over 5 seeds and 10 pairs.
FMNIST
MNIST
CIFAR 10
Merge
Entropy
Rewarded
Entropy
Rewarded
Entropy
Rewarded
Lopit Weight
0.804
0.519
0.804
0.529
0.705
0.549
0.388 (000))
0.721 (0.20)
0.423 (004))
0.748
(0oss)
0.650 (0014)
0.510 (0022))
Logit Weight
2.296
0.987 (0034)
2.301 (0000)
2.287
0.947
ME
2,208
(8050))
0.982 (M025)
2.282 (0013))
0.987
(0054)
18003 (8219)
0.881 (0,090)
QMR
Logit Weight
2.132 m0000)
0.964 (0031))
2.124 (0000)
0.993
(0,00))
2.184 (0009)
0.866
(0/05))
2,030
0.944 (0050)
2.072 (mot))
0.983
(0011)
1,979
0,587 (0/09))
XE
Logit Weight
0.809
(0000)
0.536 () 10:000
0.808 (0000)
0.482
0.686 (0000)
0.547 (0/049)
0.487
0.691
0.500
0.656
0.637
0.508
Yes
ME
Logit Weight
2.292 (0.000)
0.987 (0:03))
2.300 (0000)
0.997
s0/00h
2.286 (0000)
0.953 (0/044)
2.249
0.878
2.293 (0000)
0.987
(8056)
1,504
0.812 (0.145)
QME
Logit
2.113 (0o0e)
0.948 (004m)
2.112 (0004)
0.991
(0800)
2.155 (0000))
0.862 (0u045)
Weight
(0000)
(0005)
(o000)
(0.00)
(0.000)
(9:101)
Logit
0.902
0.644 (0.000)
1.058 (0.14))
0.816
(0696)
1.005
0.724 (0,052)
Weight
0.136
mo00))
0.808 (0009))
0.136 (0028))
0.868 (0084)
0.306 (0.00)
0.528 (ouo8))
ResNet
Logit Weight
2.285
0.972
2,300
0.996
2.273
0.927
2.110
(ooem)
0.964 (0.02m)
2.276 (0010)
0.991 (0/08))
1.484 (0,35)
0.681 (0,055)
2
QME
Logit Weight
2.126 mort
1.830
0.962 (0008)
0.936 (Q.00)
1.978 (woas)
(000e)
0.983
(0016)
2.128 (801s)
1,245 (0.34))
0.561 20/0400
0.905 (0v025)
XE
Logit Weight
0.842 mos2)
0.729 (0.266)
0.908
(0072)
0.881
(0,069)
0.963 (0053)
0.701 (0u053)
0.101
0.768
0.113
0.832
0.565
0.527
Yes
ME
Logit Weight
2.276 (0:00))
0.971 (0.08))
2.299 pocon
0.994
2.269 (001))
0.894 (o.01T)
2.164
0.955 (0:25))
2.286 (001))
0.989
(0.00)
1.619
0.647
QMB
Logil
2.101 moin
0.955 (0008)
2. 108 (0008)
0.994
(0002)
2.146 (oo00)
0.870 (0u035)
Weight
(p00r)
(1122))
(o.co))
1.406 jo.re0
(0.045)
Logit
0.701 (0.062)
0.959 (0,006)
0.740 (0.00.)
Weight
(0:11))
(0.83)
0.392 (0.023)
Logit
0.906
0.990
0.852
Weight
0.874 (0009)
0.977 (0011))
0.486 (0u020)
5
QMB
Logit
Weight
0.866 (0/22))
0.904 (0004))
0.887 (0.122)
(0000))
0.521 (0.00.9
0.846 (00004)
XF
Logit
Weight
0.788 (0/000)
0.747 (0004))
0.958 60,000
0.897 (0071)
0.362
0.693 (0v023)
Yes
ME
Logit
0.903 (00065)
0.991 (0000))
0.775 (0002))
Weight
0.861
0.985
0.328
Logit
0.905 (0/004)
0.990 (0:00)
0.781 (0.00.))
QME
Weight
0.861
(0/02))
0.350
38

---

Published as a conference paper at ICLR 2026
Table 7: Performance of pairwise-merged sub-modules with VGG FMNIST, merged either via logit averaging or weight summation, across cardinalities RR), and loss functions: CrossEntropy (XE), MaxEnt (ME), and Quasi-MaxEnt (QME), with and without Iterative Magnitude Pruning (IMP). Results are averaged over seeds and 10 pairs.
FMNIST
Model
R
IMP
loss
Merge
Entropy
Rewarded Acc
Logit Weight
0.000 (0.000)
0.501 (0.006)
XE
0.000 (0.000)
0.500 (0.000)
No
Logit Weight
2.296 (0.003)
0.986 (0.023)
ME
1.732 (0.712)
0.938 (0.101)
Logit Weight
2.108 (0.017)
0.971 (0.027)
QME
2.148 (0.059)
0.713 (0.201)
Logit Weight
0.275 (0.253)
0.558 (0.138)
XE
0.000
(0.000)
0.500 (0.000)
Yes
Logit Weight
2.293 (0.005)
0,989 (0.023)
ME
2.174 (0.174)
0.914 (0.116)
Logit Weight
2.095 (0.023)
0.957 (0.035)
QME
2.004 (0.249)
0.663 (0.214)
Logit Weight
0.143 (0.092)
0.707 (0.133)
XE
0.271 (0.214)
0.575 (0.138)
No
Logit Weight
2.287 (0.005)
0.969 (0.018)
VGG
ME
2.047
(0.346)
0.762 (0.089)
Logit Weight
2.112 (0.017)
0.964 (0.017)
QME
2.101
(0.215)
0.704 (0.121)
Logit Weight
0.090 (0.058)
0.690 (0.143)
XE
0.240
(0.201)
0.524 (0.140)
Yes
Logit Weight
2.280 (0.008)
0.971 (0.018)
ME
2.005
(0.274)
0.689 (0.100)
Logit Weight
2.088 (0.011)
0.959 (0.018)
QME
2.003 (0.235)
0.611 (0.104)
Logit
0.680 (0.056)
XE
Weight
0.452 (0.046)
No
Logit
0.913 (0.003)
ME
Weight
0.432 (0.079)
Logit
0.911 (0.001)
QME
Weight
0.411 (0.060)
Logit
0.628 (0.052)
XE
Weight
0.330 (0.052)
Yes
Logit
0,912 (0.004)
ME
Weight
0.388 (0.121)
Logit
0.911 (0.004)
QME
Weight
0.356 (0.083)
39

---

Published as a conference paper at ICLR 2026
Table 8: Performance of pairwise-merged sub-modules with VGG equipped with BatchNorm FMNIST, merged either via logit averaging or weight summation, across cardinalities [R], and loss functions: CrossEntropy (XE), MaxEnt (ME), and Quasi-MaxEn (QME), with and without Iterative Magnitude Pruning (IMP). Results are averaged over 5 seeds and 10 pairs.
FMNIST
Model
R
IMP
loss
Merge
Entropy
Rewarded Acc
Logit Weight
0.003 (0.001)
0.340 (0.096)
XE
0.011 (0.074)
0.499 (0.007)
No
Logit Weight
2.296 (0.003)
0.985 (0.020)
ME
2.238 (0.104)
0.840 (0.165)
Logit Weight
2.106 (0.015)
0.971 (0.027)
QME
1.605 (0.279)
0.746 (0.173)
Logit Weight
0.180 (0.071)
0.180 (0.120)
XE
0.200 (0.230)
0.505 (0.027)
Yes
Logit Weight
2.294 (0.006)
0.987 (0.021)
ME
2.134 (0.250)
0.940 (0.052)
Logit Weight
2.091 (0.008)
0.956 (0.033)
QME
1.475 (0.358)
0.781 (0.141)
Logit Weight
0.112 (0.057)
0.746 (0.130)
XE
0.150
(0.118)
0.688 (0.119)
No
Logit Weight
2,288 (0.005)
0.966 (0.018)
VGGbn
ME
1.866 (0.577)
0.744 (0.146)
Logit Weight
2.104 (0.012)
0.964 (0.016)
2
QME
1.890 (0.196)
0.710 (0.154)
Logit Weight
0.056 (0.042)
0.722 (0.124)
XE
0.034 (0.020)
0,551 (0.124)
Yes
Logit Weight
2.282 (0.011)
0.967 (0.028)
ME
1.898 (0.383)
0.748 (0.132)
Logit Weight
2.091 (0.013)
0.956 (0.021)
QME
1.795 (0.379)
0.745 (0.152)
Logit
0.691 (0.052)
XE
Weight
0.477 (0.145)
No
Logit
0.910 (0.010)
ME
Weight
0.610 (0.102)
Logit
0.911 (0.005)
5
QME
Weight
0.594 (0.097)
Logit
0.662 (0.027)
XE
Weight
0.524 (0.096)
Yes
Logit
0.913 (0.005)
ME
Weight
0.583 (0.084)
Logit
0.858 (0.124)
QME
Weight
0.525 (0.158)
40

---

Published as a conference paper at ICLR 2026
IMP
MNIST
FMNIST
Merge Method logit
0.8
Cardinality
0.6
0.0
CressEntropy
MaxEnt
QuasIMaxEnt
CressEntropy
MaxEnt FMNIST
QuasiMaxEnt
MNIST
1.0
Cardinaliy 2
0.4
0.2
0.0
CressEntropy
MaxEnt MNIST
QuasiMaxEnt
CressEntropy
MaxEnt FMNIST
QuasiMaxEnt
0.8
Cardinality =5
0.6
0.4
CressEntropy
MaxEnt
QuasiMaxEnt
CressEntropy
MaxEnt
QuasiMaxEnt
Figure 26: Rewarded accuracy of Shallow MLP on MNIST and FMNIST across cardinalities, loss functions, and merge strategies (logit vs. weight), with and without IMP.
41

---

Published as a conference paper at ICLR 2026
Deberiie
HAR
Random classifier
YEAST
Merge
No
togit
Yes
sum
Cardiralty
0.4
0.2
CrossEntropy
MaxEnt
QuasiMaxEnt
CressEntropy
QuasiMaxEnt
HAR
YEAST
1.0
0.8
Cardinaliy
0.4
0.2
0.0
CressEntropy
QuasiMaxEnt
CressEntropy
QuasiMaxEnt
Figure 27: Rewarded accuracy of Shallow MLP HAR and Yeast across cardinalities, loss func- tions, and merge strategies (logit vs. weight), with and without IMP.
42

---

Published as a conference paper at ICLR 2026
IMP
MNIST
FMNIST
Merge Method logit
0.8
Cardinality
0.6
0.0
CressEntropy
MaxEnt
QuasIMaxEnt
CressEntropy
MaxEnt
QuasiMaxEnt
MNIST
FMNIST
1.0
Cardinaliy 2
0.4
0.2
0.0
CressEntropy
MaxEnt MNIST
QuasiMaxEnt
CressEntropy
MaxEnt FMNIST
QuasiMaxEnt
1.0
0.8
Cardinality =5
0.6
0.4
CressEntropy
MaxEnt
QuasiMaxEnt
CressEntropy
MaxEnt
QuasiMaxEnt
Figure 28: Rewarded accuracy of Deep MLP on MNIST and FMNIST across cardinalities, loss functions, and merge strategies (logit vs. weight), with and without IMP.
43

---

Published as a conference paper at ICLR 2026
Daberiie
HAR
Random classifier
YEAST
Merge
No
togit
sum
Cardiralty
0.4
0.2
CrossEntropy
MaxEnt
QuasiMaxEnt
CressEntropy
MaxEnt
QuasiMaxEnt
HAR
YEAST
1.0
0.8
Cardinaliy
0.4
0.2
0.0
CressEntropy
QuasiMaxEnt
CressEntropy
QuasiMaxEnt
Figure 29: Rewarded accuracy of Deep MLP HAR and Yeast across cardinalities, loss functions, and merge strategies (logit vs. weight), with and without IMP.
44

---

Published as a conference paper at ICLR 2026
IMP
MNIST
FMNIST
Merge Method
logit
Mes
Cardinality =1
0.0
CressEntropy
MaxEnt
QuasIMaxEnt
CressEntropy
MaxEnt
QuasiMaxEnt
MNIST
FMNIST
1.0
0.8
Cardinalty 2
0.6
0.4
0.2
0.0
CressEntropy
MaxEnt MNIST
QuasiMaxEnt
CressEntropy
MaxEnt
QuasiMaxEnt
FMNIST
Cardinality =5
CressEntropy
MaxEnt
QuasiMaxEnt
CressEntropy
MaxEnt
QuasiMaxent
Figure 30: Rewarded accuracy of CNN on MNIST and FMNIST across cardinalities, loss functions, and merge strategies (logit vs. weight), with and without IMP.
45

---

Published as a conference paper at ICLR 2026
C.7 COMPLETE BASELINES
reporss the full comparison across all loss functions for different architectures, cardinalities, and datasets, with and without IMP. Across all settings, MaxEnt loss (ME) achieves the highest rewarded accuracy and the entropy closest to the theoretical maximum log((C)) on non-rewarded classes, confirming that its asymmetric treatment of rewarded and non-rewarded inputs is essen-
tial for isolating class-specific subnetworks. CrossEntropy with Confidence Penalty (CP) and with Label Smoothing (I.S) consistently underperform ME, demonstrating that ME is not mere re- parameterisation of standard regularisers.
46

---

Published as a conference paper at ICLR 2026
Table 9: Average rewarded accuracy and non-rewarded entropy of pairwise-merged (weight-space) sub-modules across architectures, cardinalities |R], and loss functions: CrossEntropy (XE), Con- fidence Penalty (CP), Label Smoothing (LS), MaxEnt (ME), and Quasi-MaxEnt (QME), with and without IMP. Results are averaged over 5 seeds. ME consistently achieves the highest rewarded accuracy
FMNIST
MNIST
Yeast
Model
[R]
IMP
Eatmpy
Rowarded Ace
Entropy
Rewarded Ace
Eatropy
Rowarded Ace
Entropy
Rewarded Ace
010445 (3.54)
0.723 (3.16)
0.001 1000)
0.492 pa2e
0.318 (1.00)
0.11 (235
0.176 (x260
0.559 (0140
CP
0.066 s1.0ts
0.753 11.211
0.19 1025)
0.506 suac
0.320 yu4e0
0.817 s026
0.177 ac303
0.539 51141
No
LS
0.334 (0.773)
0.503 10.001
0.338 00600
0.519 suo
0.120 sx24
0.315 X010
ME
2.205 (0:04)
0 1.500 10:101
2.300 9.00)
0.969 porn
1.512 worhs
0.666 womt
0,804 (1160)
0.829 51190
QME
2,008 10,044
0.909 11.901
20000 pum0
0.968 s0C
0.902 sute
0.974 m01
0.828 punz)
0.792 sue9
XG
0.269
10.341
0.813 13.251
0.452 (1155
0.679 sUDR
0.571 MM2
0.866 sue
1.367 sun
0.660 sue4
CP
01300
00500
0.000 p.500
0371 9175
0.504 au3
0.622 au6-
1,368 (012)
0.666 (U60)
Yos
1.S
0.786
703
0.9461 919771
0,891
0.742
0.580
1.368 swnh
0.666
2,279
0.980
0.991
1,555
0.883
1.287
0.845
QME
2,047
2.081 p/00
0.973 pa10
1076
(v09)
0.737
0.064
0.555
0.564
0.152
0:457
(ue0)
0.577 suMe
0.780 sun
0.438 sumo
0.263 (.09))
0.530 (094)
0.749 (0100)
0.699 ja09
(337))
0.784 (us
0.437 (unes)
ME
2.288 (aa
0.734 (1177)
2.297 p0x01
0.957 y0m
1.521 j0mc
0.945 2u15)
0.549 yuren
2
QME
1.959 (017
0.881
10000
1.656 (0200
0.954 (022)
0.286
(227))
0.929 suos)
0.556 10012)
XB
0.309 a.0
0.772 1011
0.334 (000)
0331 1000)
0.411 1330
0.864 auen
0.419 suos)
Shallow MLP
CP
0.347
0.773 (0:00
0.460 (039
0863 p00
0.368 y005
0.822 p0xt)
0.418 putto
Yes
LS
0.778
0.507 101831
0.912 11291
0,825 s000
0.587 M11
0.723 (ued)
0.418 suots
ME
2.252
11552 (0025)
2.264 pao
0.580 jume)
1.509 2020
0.945 suis)
0,549 sunc-)
QMF
1.773
0.918
198
0.963
0.577
0.919
0.475
XB
0.451
0.554
CP
0.424
0.580
I.S
0.527
0.690
10.00
0.937 s08t7
QME
0.713 (0.05)
0.891 juzz
0.424 (0054)
0.831 noth
CP
0-436 11/901
0.832 y0m0
Yes
LS
0.521 10.201
0.848 s0077)
ME
0.827 (0014
0.945 p0m0
QME
0.752 (0.00)
0.909 s0012
XB
0.016
10:241
0.544 121151
0.000 11:0)
0.490 suG2
0.218 11277
0.797 uz2
0.019 suec)
0.509 sueet
CP
0.022
01568 (21441
0.000 200))
0.490 ja22)
0.253 (.23)
0818 a23
0.020 jumt)
0,509 jum
No
I.S
0.360
1.545
0.333 pm
0.494 yain
0.320
0.838
0.181 p0s6
0.519
2.293
0.886
2,300
0,333
1.305
0.979
0.865
0,550
QME
1,836
0.916
p01
0.975
0.792
0.965
0.594
0.787
0.216 (818
0.486
0,668
0.543
0.674
0.228 HIt
0.788 11.24)
0.564 (022)
0.550 uato
X14
mm))
1.375 suen
0.669 suen
YES
LS 0.718 (154
0.655 (1188)
0,925 (030)
0365 (.3
0.553 (012
2:237 (a00))
0.973 yoarts
0.991 mao
1.08 yust
0.982 pu0t)
awn))
0,839 y119
QME
1.544 ma3
0.858
(01531
2.062 suzz)
0.960 wuch
0.779 (uz7)
0.923 (08))
1.178 sunt)
0.807 sL12
XB
0.006
(34
0.622 mi1
0.133 summ
0.580 (12
0,141 (157)
0.668 (i6
0.438 suos)
CP
0.074 ma54
0.627 p110
0.133 (009)
0.559 am
0.123 911
0.633 (1177)
0.46f] 2015
No
LS
0.306 111
0.575 (100)
0.446 112191
0.551 xuee
0.329 stze1
0.805 1122)
0.476 succ
ME
2.291
2314
0.798 suam
2.295 sum
0.085 juzz)
1.219 2210
0.935 auo3)
0.619 surec)
2
QMF
1.966
0.862
1,825
0.900
0.238
0.919
0.582
XF
0 0.226
748
0. 194 p019
0.724 yuad
0.400
0.821
0.372 sueth
Doep MLP
CP
0.227
0.729
0.438
0,812
0.213
0.705
0.405
Yas
0.636
0.648
0.831
0.763
0.634
0.407
2.225
0.930
2.272 (080
0.975 somm
0.672 3154
zuss)
0.598 sumt
QME
1.347
(.40))
0.546 101121
1.158
(036))
0.910 juett
0,205 jur
0.868 (uss)
0.486 (u/00)
0.356 0030)
0.578 (.13))
CP
0.558 9110
0.607 y119
No
LS
0.566 10.00
0.551 suMes
ME
0.485 10/07)
(1639 pach)
QME
0.601 11.31
0.784 sume
XE
0.500 11:34
0.687 sume
CP
0.437
0.718 juzzo
Yok
1.5
0.486
0.806 yuarss
ME
0.500
0.919
QME
0.649
0,807
0.045
0.992
0.077
0.517
0.547 1111
0.064 (1147)
0.513 stren
0.197 a.9
0.510 (u00)
0.225 (0340)
0.518 (um
MF
2.226
m000
0.950 (019)
2.102 9m
0.972 purtt
QME
1.272
(:24)
0.906
1210 pum
0.970 sue
XB
0.022
(34)
0.576
11:51
0.023 (000
0.520 sumb
CP
0.004
0.575 (0151
0.002 9000)
0.509 path)
Yes
LS
0.169
0.582 11151
196 (12191
0.527 sue
ME
2,189
D1966
2.021 (027)
0.544 juzz
QME
1.006
0.996
0.969 9m77
0.952
XF 0.081
0.664
0.095 post
0.311 91
CP
0.092
0.671
0.135
0.731
0.618
0.235 p.12
0.683
2.147 (err
0.384
0.948 pis
2
QME
1.188 mim
0,009
0.743 (070
0.936 ju60
XB 0.047 (33
0.300 (000)
0.022 (00
0.68 ju0
CNN
CP
0.054
(8:4
0611 (1
0.66 (0
0,63 yr
Yes
LS
0.183
0.587 10:21
0.233 (L1
0.611 su20
ME
2,055
0:00
8.926 2047)
164 2460)
0.964 nart
QME
0.604
y 7721
0:808 (196)
0 366 no
0.945 p01s
XE
0.462 11:54
0.708 sume
CP
0.475 1177111
0.738 juzz
No
I.S
0.407
0.641 ym
MC
0.903
0.915
QME
0.767
01884
0.453
0.376
0.535
0.794
Yes
0.528 (3565
0.554 (uTT))
0.791 p110))
0.943 jastn
QME
0.750 11:00)
47 0.909 su84

---

Published as a conference paper at ICLR 2026
C.8 IMAGENETTE
To evaluate the generality of our framework on large-scale image data, we apply our pipeline to Im- agenette (Howard 2019), challenging 10-class subset of ImageNet (Deng et al. 2009) comprising visually diverse categories (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute). We train ResNet18 with the MaxEnt loss, using pruning ratio of 0.6 whenever IMP is applied, following the typerparametrisation recommended by the dataset authors.
Pairwise merge. Table 10 reports pairwise merge results across cardinalities [R| E {1, 5)
Logit-space merging consistently outperforms weight-space merging across all settings. For [R| = 5, when the union of rewarded sets covers the entire dataset, logit-merged modules match or sur- pass the CrossEntropy baseline trained on the full dataset in the standard way (0.894 + 0.002 vs. 0.866 + 0.011). Weight-space merging degrades at higher cardinalities, consistent with our find- ings on CIFAR-10 and underscoring that logit merging is the reliable composition rule for complex architectures.
Table 10: Pairwise merge results on Imagenette (ResNet18). Mean and std over 5 seeds. The CrossEntropy baseline trained the full dataset achieves 0.866 + 0.011.
R
IMP
Merge
Entropy
Rewarded Acc
No
Weights
1.899 + 0.367
0.628 + 0.105
No
Logits
2.293 + 0.002
0.952 + 0.021
1
Yes
Weights
2.064 0.258
0.641 = 0.086
Yes
Logits
2.288 + 0.006
0.942 = 0.024
No
Weights
2.152 + 0.154
0.452 = 0.063
No
Logits
2.282 0.004
0.926 = 0.021
2
Yes
Weights
2.127 = 0.186
0.409 = 0.058
Yes
Logits
2.282 0.008
0.928 + 0.020
No
Weights
0.334 + 0.075
No
Logits
0.886 + 0.006
5
Yes
Weights
0.206 + 0.009
Yes
Logits
0.894 0.002
Complete merge. Table 11 shows the accuracy of the fully merged model at the final step of the menplete-merge sequence (one class-specific module added time until all 10 classes are covered). Logit merging achieves 0.849=0.007 with IMP and 0.845=00.00 without, remaining only slightly below the CrossEntropy baseline (0.866 + 0.011). Weight-space merging collapses to near- random performance, confirming that the composition difficulty observed on CIFAR-10 extends to larger-scale datasets.
Table 11: Final step of the complete merge on Imagenette (ResNet18). CrossEntropy baseline: 0.866 + 0.011.
IMP
Merge
Accuracy
No
Weights
0.113 + 0.014
No
Logits
0.845 + 0.005
Yes
Weights
0.103 + 0.013
Yes
Logits
0.849 + 0.007
Taken together, these results show that MaxEnt + sparsity training reliably identifies class- -pesi-c submodules at the scale of InageNet-derived data, and that logit-space merging produces general- ist model that closely matches a monolithic CrossEntropp baseline.
48

---

Published as a conference paper at ICLR 2026
C.9 CIFAR-100
To probe the limits of compositionality on a dataset with a large number of classes, we evaluate ResNet18 with MaxEnt on CIFAR-I00 (Krizhevsky et al. 2009 All experiments pruning ratio of 0.6 (when IMP is applied).
reports results for |R| 50, i.e., two submodules each specialising 50 disjoint classes that are then merged. Logit merging again outperforms weight merging in both the pruned and unpruned setting. The logit-merged model (No-IMP: 0.644 0.002; IMP: 0.521 + 0.005) approaches the CrossEntropy baseline (0.661 + 0.006), whereas weight merging degrades substantially.
Pairwise merge at cardinality 50. Table 12
Table 12: Pairwise merge at R = 50 on CIFAR-101 (ResNet18). CrossEntropy baseline: 0.661 + 0.006.
IMP
Merge
Rewarded Accuracy
No
Weights
0.107 + 0.011
No
Logits
0.644 + 0.002
Yes
Weights
0.020 + 0.002
Yes
Logits
0.521 + 0.005
Complete merge with 100 modules. We conducted a stress test useful to detect whether we can merge more than 10 class-specific sub-modules together or if there is bottleneck. To such extent, we tested, for CIFAR-100, the complete merge experiment (i.e., merging 100 sub-modules each specialized single class). This the most challenging scenario. Table rebports at representative steps of the complete-merge sequence (steps 1, 25, 50, 75, 100), run on a single seed due computational cost. Each step adds one additional single-class module the merged model. We observed no sudden drop in performance each time we added new model (till reaching 100 steps). The rewarded accuracy decreases gradually from nearly 100% (initial modules) to roughly 50% the end of the merge sequence. Importantly, logit merging maintains above 60% rewarded accuracy for roughly the first 50 merged modules before gradually decaying, reaching 0.533 (with IMP) and 0.509 (without IMP) step 100 compared to the CrossEntropy baseline of 0.661. These results confirm that MaxEnt + sparsity training scales to datasets with high visual complcxity and large numbers of classes. Logit-space merging remains the reliable composition strategy, recovering a substantial fraction of monolithic performance even when composing 100 independently trained modules.
reports accuracy
49

---

Published as a conference paper at ICLR 2026
Table 13: Complete merge on CIFAR-100 (ResNet18), single seed. CrossEntropy baseline: 0.661 + 0.006.
IMP
Merge
Step
Accuracy
0.992
25
0.654
Logits
50
0.581
75
0.536
100
0.509
0.992
25
0.046
Weights
50
0.019
75
0.011
100
0.009
0.985
25
0.671
Logits
50
0.606
75
0.560
Yes
100
0.533
0.985
25
0.021
Weights
50
0.012
75
0.009
100
0.009
50

---

Published as a conference paper at ICLR 2026
C.10 TEXT CLASSIFICATION
To evaluate the generality of our framework beyond vision tasks, we conduct a comprehensive study on two standard text-classification benchmarks: IMDB (Maas al. 2011) (binary sentiment clas- sification) and 20 Newsgroups (Lang] 1995) (20-way topic classification). These datasets differ substantially in size, difficulty, and label-space cardinality, allowing us to assess the robustness of modular training and merging in a qualitatively different regime.
Our evaluation mirrors the setup used in the main paper: models are trained specialize on subsets of classes via the MaxEnt objective, optionally combined with IMP, and then merged using either weight-space summation or logit-space averaging (Table T4). We compare the resulting merged models to CrossEntropy baselines trained on the full dataset (Table 15p.
Across both datasets and both architectures (shallow and deep MLPs), we find that merged models attain performances comparable to, and in the cases with biggest cardinality often even exceeding, the full-dataset CrossEntropy baselines. We can notice this in IMDB where logit-merged modules without IMP consistently reach 0.88 accuracy, for shallow MLP and 0.87 for deep MLP). For 20NG, logit-merge again produces strong merged models. For example, when merging modules with cardinality 10 and without using IMP, shallow MLP reaches 0.70 accuracy, on par with the baseline, whilst deep MLP obtains 0.68, surpassing the 0.64 of the corresponding baseline.
par with or slightly above the CE baseline (0.88
51

---

Published as a conference paper at ICLR 2026
Table 14: Performance of merged submodules on text datasets (20NG and IMDB). Mean and std over seeds.
Dataset
Model
[R]
LTH
Merge
Entropy
Rewarded Acc
No
Logit
2.986 (0.004)
0.878 (0.069)
No
Weight
2.833 (0.032)
0.703 (0.091)
Yes
Logit
2.977 (0.007)
0.886 (0.083)
Yes
Weight
2.856 (0.090)
0.879 (0.083)
Logit
2.917 (0.016)
0.739 (0.024)
20NG
Deep MLP
No
Weight
2.248 (0.128)
0.403 (0.035)
Yes
Logit
2.905 (0.019)
0.729 (0.022)
Yes
Weight
2.678 (0.064)
0.719 (0.022)
No
Logit
0.677 (0.005)
No
Weight
0.368 (0.021)
10
Yes
Logit
0.667 (0.004)
Yes
Weight
0.654 (0.008)
Logit
2.984 (0.003)
0.872 (0.074)
No
Weight
2.764 (0.051)
0.810 (0.096)
Yes
Logit
2.981 (0.005)
0,859 (0.085)
Yes
Weight
2.946
(0.009)
0.857 (0.083)
No
Logit
2.926 (0.012)
0.756 (0.020)
20NG
Shallow MLP
No
Weight
2.126
0.528 (0.060)
5
Yes
Logit
2.933 (0.013)
0.734 (0.024)
Yes
Weight
2.753 (0.043)
0.728 (0.025)
No
Logit
0.704 (0.002)
No
Weight
0,599 (0.020)
10
Yes
Logit
0.673 (0.002)
Yes
Weight
0.670 (0.001)
No
Logit
0.877 (0.001)
No
Weight
0.684 (0.049)
IMDB
Deep MLP
Yes
Logit
0.848 (0.003)
Yes
Weight
0.849 (0.005)
No
Logit
0.880 (0.001)
No
Weight
0.870 (0.002)
IMDB
Shallow MLP
Yes
Logit
0.848 (0.002)
Yes
Weight
0.849 (0.002)
Table 15; CrossEntropy baselines for reference in text classification. Mean and std over 5 seeds.
Datasct
Model
|R|
Loss
Acc
Std
20NG
Deep MLP
20
XE
0.639
0.005
20NG
Shallow MLP
20
XE
0.702
0.001
IMDB
Deep MLP
2
XE
0.867
0.002
IMDB
Shallow MLP
2
XE
0.876
0.006
52


Reviews:
[{"review_id": "fvs12uMceb", "reviewer": "Reviewer_g43A", "summary": "The authors describe a method for composing independently trained sparse neural subnetworks into a single model without joint training or fine-tuning. Each subnetwork (“module”) is trained on a subset of classes called the rewarded set, using a modified loss based on the maximum-entropy principle: the model is encouraged to make confident predictions only for its rewarded classes and uniform (high-entropy) predictions for all others. Sparsity is induced through IMP to limit parameter overlap between modules. Once trained, these subnetworks (each specialized to different classes) are combined by simple weight summation, producing a merged model that can classify across all classes while preserving much of each specialist’s performance.\n\nThe authors empirically evaluate this approach on MNIST, Fashion-MNIST, and two tabular datasets, using small MLPs and a LeNet-style CNN. They report that modules trained with the MaxEnt objective achieve functional isolation, and that merged networks retain high per-class accuracy and high entropy on non-rewarded classes, indicating low interference. Mode-connectivity analyses further suggest that module parameters lie in nearby basins, explaining why their linear combination yields coherent behavior. The work claims that enforcing entropy separation, combined with sparsity, enables modular composition of neural networks in weight space.", "questions": "-  Please, if there is enough time in the rebuttal phase, compare the MaxEnt loss directly to label smoothing and confidence-penalty objectives, tuned so that non-rewarded classes approach a uniform distribution. This is essential to verify that MaxEnt provides more than a re-parameterization of existing regularization methods\n- Conduct a 2×2 ablation (MaxEnt vs. standard cross-entropy) × (with vs. without IMP) while keeping final sparsity fixed. This would clarify whether isolation arises mainly from the loss, from pruning, or from their interaction.\n- can the method compose modules on datasets with shared features, such as CIFAR-10 or CIFAR-100? Can you have some results on a small ResNet at similar sparsity?\n- As you know weight summation can amplify parameter norms. Did you test weight rescaling, re-centering, etc before merging? Please report any normalization strategy (or justify why none is needed?)\n- Quantify inter-module overlap. for example Jaccard similarity of masks, cosine similarity of activations, or participation coefficients. This will help support your claim of isolated modules.\n- in the real world we often have overlapping or hierarchical class groups. Can this approach handle partially overlapping rewarded sets without large interference?\n- How would the method extend to architectures with BatchNorm or LayerNorm? Did you attempt merging such models?\n- beyond non-rewarded entropy, report per-class confusion or leakage rates, calibration error and loss-barrier heights as quantitative measures of interference after merging\n- I would love to see some comparisons against model soups, Fisher-weighted merging, and Git Re-Basin under the same initialization and pruning regime to position your approach within current weight-space composition literature.\n- Add legends and unit definitions in Table 1. specify the dataset split and permutation protocol in complete-merge experiments. plz give implementation details for IMP (rounds, pruning ratios, weight resets) and temperature settings for all architectures", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- simple, interpretable composition mechanism, avoiding complex re-alignment or joint optimization.\n- clear/principled loss function that explicitly enforces confident predictions on rewarded classes and uniform predictions elsewhere, promoting functional isolation\n- demonstrates that iterative pruning improves mergeability, showing a plausible connection between sparsity and reduced interference across modules\n- mode-connectivity analysis indicating that isolated modules occupy nearby low-loss basins, lending some theoretical (but limited) support for weight-space composition\n- related to modular and compositional learning. In my opinion, this is an important and promising direction in DL these days.", "weaknesses": "Major issue: experiments are limited to simple datasets and small architectures (MNIST, Fashion-MNIST, shallow MLPs, LeNet)\n- The scalability and effectivenes to realistic settings (e.g., CIFAR-10/100, ImageNet, Transformers) is unclear (probably it would not work in my opinion)\n\n- no comparison to strong baselines such as model soups, Fisher-weighted merging etc\n- The interaction between pruning and the MaxEnt loss is not disentangled. It is unclear which component drives functional isolation and mergeability\n- Summation without alignment or normalization can be tricky to work\n- the MaxEnt objective may suppress shared features and degrade generalization when classes share structure, limiting applicability beyond disjoint one-vs-all setups\n- Claims of “modular structure” are not supported by structural analysis. There is no measurement of mask overlap, activation similarity, or connectivity between class modules\n- The method lacks quantitative ablations and uncertainty metrics (e.g., calibration, leakage, loss barriers vs. number of merged modules).\n- The writing and figure clarity varies a bunch. Some tables lack legends and experiment protocols (e.g., dataset reuse and merge order) are not sufficiently described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaND7KnCqJ", "reviewer": "Reviewer_oJ2u", "summary": "The paper examines whether independently trained class-specialized sub-networks can be combined into a single network that accurately identifies all classes.\n\nThe authors propose a method combining the maximum entropy principle with iterative magnitude pruning. The approach defines an objective function that uses KL divergence during module training to identify specific class subsets, while enforcing uniform distribution predictions for samples from other classes. The method employs iterative magnitude pruning with weight rewinding to enhance both functional specialization and module composability.\n\nThe authors evaluates the method using two image classification and two tabular classification datasets. The evaluation encompasses three architectures: a shallow MLP, a deep MLP, and a CNN. Results demonstrate that the specialized sub-network modules successfully identify specific class subsets and perform well when combined to handle all classes. The authors also conduct mode-connectivity analysis, revealing that the modules are mode-connected with a local minimum at the linear path's midpoint.", "questions": "- What factors contribute to the degradation of composability in deeper networks? This issue requires further investigation and discussion to understand its implications.\n- Why does iterative magnitude pruning show greater effectiveness in deeper networks compared to shallow MLPs? Additional insights would be valuable.\n- How does this work relate to broader research trends addressing increasingly complex tasks beyond classification?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents its hypotheses, questions, details, and conclusions with exceptional clarity.\n- The proposed approach offers a novel and well-justified contribution.\n- The results convincingly demonstrate the method's advantages over traditional training approaches, with mode-connectivity analysis providing valuable insights into local minima and composability outcomes.", "weaknesses": "- The method's application is restricted to classification tasks.\n- The approach only identifies end-to-end modules and requires manual class assignment, limiting its utility for complex tasks where classification is just one component and intermediate labels may be unavailable.\n- The experiments are confined to small-scale settings, and composability deteriorates with increased network depth, raising concerns about the method's scalability to deeper networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oVCZtyXC43", "reviewer": "Reviewer_gGJL", "summary": "The proposed method creates modular NNs for classification by training each subnetwork to recognize only a subset of the classes. When the input is from another class the network is trained to output a uniform probability vector. Sparsity is achieved by iterative pruning. Subnetworks are then combined by weight summation, which is justified by a modified mode-connectivity analysis. Experiment results show the subnetworks learn their tasks, the merged models succeed at the combined tasks, and pruning leads to better merging.", "questions": "Would there be a benefit of end-to-end fine-tuning after merging? Fine-tuning just the last linear layer could reduce interference between subnets and wouldn't impact modularity or interpretability.\n\nThe Quasi-MaxEnt loss will bias predictions of merged models for the reason explained in the final paragraph of sec 3.1. As an alternative baseline, what about giving each subnetwork an “other class” output for $\\mathcal{C}\\setminus\\mathcal{R}$ which is then discarded in the merged model?\n\nCombining the above two comments, I bet last-layer fine-tuning would eliminate the difference between QME and ME, since the targets of these two loss functions are linear combinations of each other (in logit space). The same should be true of one-vs-all formulations. If the primary question is whether an effective modular representation can be learned then it seems fair to free up the final weight layer, which is just an affine map from this representation to logits.\n\nMerging models requires a choice of correspondence between neurons (or filters) because of permutation invariance. The naive approach of identifying neurons in different models that have the same nominal index amounts to a random permutation. Can you do better by choosing a permutation, for example to minimize overlap in weights?\n\nRather than merging models, you could compute each model separately and sum their logits. This would avoid questions of mode connectivity. In fact you’re effectively doing this already when pruning is high (e.g., the 99% pruning for the MLPs).\n\nL193: I'm not certain what is meant by the first term on the RHS. Is it $\\mathcal{L}(f\\_{\\theta\\_1}(\\mathcal{D}))$ or is $(1-t)$ included? Why the asymmetry, basing the threshold only on model 1?\n\nTable 1: It would help to state the number of classes for each dataset (I had to look up HAR (6) and yeast (10)). Why is entropy so low for yeast when it is near maximum for the other datasets? The dataset is imbalanced (I calculated entropy 1.729) but the target should still be uniform implying entropy 2.302. Also it would help to state $|\\mathcal{R}|=1$ since that is given only later in the text.\n\nFig 2: what are the units of the entries? I would have guessed they’re predicted probability summed over the test set, but the MNIST test set is balanced and the row sums here differ across rows (though they look to be consistent across tables).\n\nThere’s a claim that merged models maintain near-maximal entropy for non-rewarded classes (line 308) but Fig 2 suggests a significant bias toward rewarded classes. I calculated the entropy for classes 1 and 2 in fig 2a and got 1.97 and 1.78, much lower than the reported 2.28. Is this an atypical example?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Clever training objective to make each subnet an expert on its classes but to be agnostic (i.e., not interfere) on other classes. The combined loss is elegant, relative entropy from either a one-hot or a uniform. Experiment results show the method works as intended.", "weaknesses": "The promise of modular and interpretable networks makes sense, but the paper would be stronger with some direct demonstrations of downstream applications that could not be done with a standard monolithic network.\n\nThe mode connectivity analysis is not well justified since (as the paper notes) it evaluates a combined loss that neither model was trained on. The results in fig 5 are not surprising, because each separate model does terribly on the other model’s task. The more important question is how the combined model’s loss for classes in $\\mathcal{R}\\_{\\rm A}$ compares to model A’s loss on those classes. In the ideal case, where summing parameters results in summing the logits, these losses will be equal because model B’s logits will be uniform for items in $\\mathcal{R}\\_{\\rm A}$. This is the rationale for the method, but how close to you come to that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4xrg37SkI", "reviewer": "Reviewer_EE1e", "summary": "This paper introduces a method for training sparse networks that accurately classify only a designated subset of classes while remaining deliberately uncertain on all others, functioning as class-specific subnetworks.", "questions": "Please refer to the weaknesses, especially more experiments on larger models (e.g., ResNet, ViT) and more complex tasks (e.g., CIFAR, ImageNet).\nIf all my problems are addressed, I will raise my rating.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The topic this paper focused on is very interesting.", "weaknesses": "1. The motivation of using Iterative Magnitude Pruning is not clear enough. Authors should explain clearly why using such pruning is a necessity.\n\n2. Authors only conduct experiments on simple networks such as MLP and LeNet, making results not convincing enough. Can the proposed method be applied to other widely-used DNNs, such as ResNets, transformer-based models, VIT?\n\n3. Can the proposed method be applied to large-scale and complex image dataset, such as imagenet or CUB200? MNIST is too simple.\n\n4. Can the proposed method be generalized to other tasks, such as text classification and other tasks rather than classification?\n\n5. What is the pratical usage of the proposed method? Authors can add a discussion for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)