How Can Knowledge of a Task’s Modular Structure Improve Generalization and Training Efficiency?
Authors: Shreyas Malakarjun Patil
Cameron Ethan Taylor
Constantine Dovrolis
Venue: Accepted by TMLR ()

Abstract: Many real-world learning tasks have an underlying hierarchical and modular structure, composed of smaller sub-functions. Traditional neural networks (NNs) often disregard this structure, leading to inefficiencies in learning and generalization. Prior work has demonstrated that leveraging known structural information can enhance performance by aligning NN architectures with the task’s inherent modularity. However, the extent of prior structural knowledge required to achieve these performance improvements remains unclear. In this work, we investigate how modular NNs can outperform traditional dense NNs on tasks with simple yet known modular structure by systematically varying the degree of structural knowledge incorporated. We compare architectures ranging from monolithic dense NNs, which assume no prior knowledge, to hierarchically modular NNs with shared modules that leverage sparsity, modularity, and module reusability. Our experiments demonstrate that module reuse in modular NNs significantly improves learning efficiency and generalization. Furthermore, we find that module reuse enables modular NNs to excel in data-scarce scenarios by promoting functional specialization within modules and reducing redundancy.

Published in Transactions on Machine Learning Research (08/2025)
How Can Knowledge of a Task's Modular Structure Improve Generalization and Training Efficiency?
Shreyas Malakarjun Patil
sm papplallgateecch..mmomm edu
School of Computer Science, Georgia Institute of Technology
Cameron Taylor
cameron.tarloor@ggaechh....
School of Computer Science, Georgia Institute of Technology
Constantine Dovrolis
constantine@gaaech..de c.dovrolis@cyj.accc
School of Computer Science, Georgia Institute of Technology
Computational Science and Technology Research Center (CaSToRC), The Cyprus Institute
Reviewed OpenReview: https: /ttp::///ppnrevviceee net/ formm? id=A0NTOOO
Abstract
Many real-world learning tasks have an underlying hierarchical and modular structure, composed of smaller sub-functions. Traditional neural networks (NNs) often disregard this structure, Icading to inefficiencies in learning and generalization. Prior work has demonstrated that leveraging known structural information can enhance performance by aligning NN architectures with the task's inherent modularity. However, the extent prior structural knowledge required to achiove these performance improvements remains unclear. In this work, we investigate how modular NNs can outperform traditional dense NNSs on tasks with simple yet known modular structure by systematically varying the degree of structural knowledge incorporated. We compare architectures ranging from monolithic dense NNs, which assume no prior knowledge, to hicrarchically modular NNs with shared modules that leverage sparsity, modularity, and module reusability. Our experiments demonstrate that module reuse in modular NNs significantly improves lcarning efficiency and generalization. Furthermore, we find that module reuse enables modular NNs to excel data-scarce scenarios by promoting functional specialization within modules and reducing redundancy.
1 Introduction
Real-world learning tasks often exhibit inherent hicrarchical and modular structure, where complex target function can be decomposed into smaller, hierarchically organized sub-functions (imon 1991 Traditional neural networks (NNs), such as multilayered perceptrons (MLPs), however, treat target functions as undifferentiated input-output mappings, disregarding any underlying modular structure. This leads to higher training costs and increased data requirements. By exploiting the task structure in NN architecture design, efficiency and generalization can be significantly improved. For instance, domains like vision and language, prior structural knowledge such as spatial locality and temporal coherence are well understood. These priors, when encoded using convolutional or recurrent architectures, are known to significantly improve generalization performance and training efficiency.
On the other hand, the precise hicrarchical and modular structure underlying a given task is often unknown. To address this, previous work has explored explicit hierarchical and modular archilectures that incorporate partial knowledge of the task structure (Fernando et al. 2017 Rosenbaum et al., 2017 Shazeer et al. 2017 Kirsch et al. 2018 Goyal al. 2019 2021 Ponti al.] 2022). These architoctures break the NN into sparsely connected sub-networks or modules, each learning a distinct iub-function and organize them hicrarchically,
where the organization itself is learned.

---

Published in Transactions on Machine Learning Research (08/2025)
Monalthic dense
Nondlthic random sparse
Herarchizally modular Leamned inter module connectivity
Hitrarchicall modular Fixed inter module connectiviy by
Inter-odile ocnnectbity
Herarchcaaly moduler with shared modules
Herarchically modular with shared modules Fxedd imer-module connectivity and modulss selection
Learned Inter-module connectivity and module selection
Module locations
Intor-module connerdivity
Module selector
Shared
Shared modules
Filed with modure
modules
Empty
Unused
Figure 1: Overview of the models, highlighting varying levels of structural knowledge assumed at initialization. The associated task for these architectures is depicted in Figure 2 (depth 2). From top left to bottom right: Monolithic dense NN: Unknown task structure. Monolithic random sparse NN: Sparsity in task structure. Hicrarchically modular NN: Modular sparsity pattern. Hicrarchically modular NN (fixed inter- module connectivity): Modular sparsity with known sub-function connectivity. Hierarchically modular NN with shared modules: Modular sparsity and module reusability. Hierarchically modular NN with shared modules (fixed inter-module connectivity and module selection): Modular sparsity, module reusability, known sub-function connectivity and reuse.
The variety of specific architectural choices tailored to different tasks raise questions about the extent of prior knowledge required for performance improvements. For instance, architectures designed for tasks like visual question answering (VQA) often depend a high degree of prior knowledge about the task's modularity (Andreas et al. 2016 Hu et al.] 2017). Conversely, generic configurations like mixxuure-of-experts (MoE) or
routing networks excel in domains such as mulli-ttsk learning (Rosenbaum et al. 2017 Purushwalkam et al,
2019
Hazimeh et al.] 2021 Ponti et al. 2022 Chen et al.) 2023). transfer continual learning (TTerehoov
ct al. 2015; Wang et al. 2020: Veniat et al.)
2020 Mendez & Eaton 2020 Ostapenko et al. 2021). and
compositional generalization Bahdanau el al.
2018 Lake & Baroni 2018 Chang el al., 2018 Hupkes el al.
2020 Rahanan et al. 2021). In these domains, where sub-functions are dynamically organized or selected based input-dependent cues, modular designs enable flexible function representation by adapting the organization or selection of modules for different inputs. Even in these settings, previous analysis suggests that knowing specific input components influencing module organization (Mittal et al. 2022), as well as understanding module architecture (such as input dimensions) or identifying input bottlenecks, improves
generalization Bahdanau et al. 2019 Goyal et al. 2021 Ostapenko et al. 2022
Our work contributes to this line of inquiry PPathak et al. 2019 D' D'Amario et al. 2021 Mittal et al. 2022
Ostapenko et al., 2022; Schug et al., 2023 Jarvis et al. 2024 Lippl & Stachenfeld 2024 by systematically studying how different degrees of structural knowledge impact NN generalization and training efficiency. We consider tasks with fixed and known hierarchically modular structure, particularly those derived from Boolean functions. Unlike prior works that focus on input-dependent modular structure, static modular structure provides a controlled environment for evaluating key architectural principles such as sparsity, modularity, and module reusability, while isolating those factors from principles used to learn input-dependent hicrarchical organization. Additionally, the synthetic Boolean tasks with completely known structure allows for systematically varying the degree of prior knowledge encoded in the architecture, from assuming no prior knowledge to complete structural knowledge.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Honerchical
Herarchica
Heerarhhical depti::
OR
AND
XOR
Transfer
Figure 2: Function graphs with varying complexity used to generate the truth tables.
We start with dense NNs that make structural assumptions, followed by random sparse NNs that assume sparsity without specific pattern, classifying them monolithic NNs. We then examine hierarchically modular NNs, where modules are explicitly defined and organized hierarchically (Fernando et al.) 2017 Ostapenko et al., 2021). In these networks, both module weights and inter-module connections are Icarned without assuming prior knowledge of specific connectivity. We extend this exploration to modular NNs with fixed inter-module connectivity, where the exact sub-function connectivity known. Finally, we introduce module reusability, where the same module can be used in multiple locations within the hierarchy, reflecting the idea that sub-functions may recur throughout the task (Goyal et al., 2021 Ostapenko et al. 2022). In this setup, the NN must. dynamically learn both connectivity and module selection from shared pool of modules, We also explore a variant with fixed connectivity and module selection, representing complete structural knowledge.
All architectures studied are implemented using multi-layer perceptrons (MLPs) and involve learning functional components, regardless of the degree of structural knowledge incorporated. We systematically analyze the effects of sparsity, modularity, and module reusability on generalization and training efficiency. Our findings demonstrate that hierarchically modular NNs with shared modules consistently outperform dense N.ss, especially in data-scarce scenarios. Furthermore, module reuse through sharing facilitates the accurate learning of inter-module connectivity while promoting functional specialization within modules, ultimately leading to improved generalization. These results are demonst on simple, hierarchically modular tasks with fully known structure, primarily Boolean functions and are further validated on a visual recognition task using the MNIST dataset.
2 Preliminaries
2.1 Hierarchically Modular Boolean Functions
In this work, we construct hierarchical and modular tasks using Boolean functions (Malakarjun Patil et al. 2024). A Boolean function f (0, 13" 10,1) 1)m maps n input bits to m output bits. The set of gates G includes V. (AND, OR, XOR), with edges representing direct connections.
A function graph for a Boolean function is represented as directed acyclic graph (DAG) consisting of n input nodes with zero in-degree, k gate nodes with non-zero in-degree (associated with gates from G),, and m output nodes with zero out-degree. truth table is a tabular representation of a Boolean function that enumerates all possible input combinations and their corresponding outputs.
A sub-function sub-task corresponds gate node within the function graph that applies an operation on its specific inputs. Sub-functions are organized hierarchically, with outputs from certain sub-functions serving as inputs for others. These sub-functions have three fundamental properties:
1. Input Connectivity and Separability: Sub-functions operate on outputs from previous sub-functions or input nodes. This connectivity is sparse-eade each sub-function relies on a subset of preceding outputs. In our experiments, each sub-function takes exactly two inputs to maintain uniformity across graphs
2. Output Connectivity and Reusability: Outputs produced by sub-function can be reused by multiple sub-functions at higher hierarchical levels, similar to feature reuse in NNs.
3

---

Published in Transactions on Machine Learning Research (08/2025)
3. Sub-Function Reusability: The functional operation of sub-function can be reused at multiple locations
in the function graph. For instance, an XOR gate might be reused in various parts of the graph.
A task's hierarchical structure or sub-function organization is defined by the relationships among these underlying sub-funee fubefinction.
2.2 Neural Network Architectures
In this section we describe various NN architectures used in our experiments.
Monolithic NNs: We consider dense multi-layer perceptrons (MLPs) and sparse MLPs. Sparse MLPs are created by random pruning to introduce sparsity, leveraging a notion sparsity in the function graph but. without its specific pattern.
Hicrarchically Modular NNs (modular): The modular architccture is arranged in L hierarchical layers, each containing Mr modules, denoted as mi,, where i represents the module's position within layer 1. Each module includes a small MLP and an input selection vector sf RM-11, The MLP learns the functional component, while the input selection vector governs inter-module connectivity by selecting module inputs from the outputs of modules in the previous layer. The final outputs of the NN are selected from the set of all modules in the last layer using an output selection vector.
We explore two scenarios: 1. Modulor: Both inter-module connectivity and module MLP weights are learned. 2. Modular-FC: Inter-module connectivity is fixed, inferred from the function graph, while module MLP weights are learned.
Learning Inter-Module Connectivity: Given outputs from layer - 1, æ E RM-1, the Sigmoid function is applied to si to produce a score for each potential input, selecting the top-k inputs. We use the straight- through estimator (Bengio et al., 2013) to propagate gradients through non-differentiable selections, facilitating effective learning of moder- connectivity. See Appendix [ for additional implementation details and Appendix F. for experiments related to learning inter-module connectivity.
Hierarchically Modular NNs with Shared Modules (modular-sharel)); The modulor-shared architecture extends the modular architecture, treating module positions as slots filled by modules from shared pool of M modules. Each slot has an input selection vector 8 E RMM-+ and a module selection veclor of E RM which determines the module used in that slot. The final outputs are selected from the set slots in the last layer using an output selection vector.
We explore two scenarios: 1. Modular-shared: The network learns both inter-module connectivity and module selection dynamically, alongside module weights. 2. Motuular-harre--FCMM Connectivity and module selection are fixed, with only the module weights being learned.
Learning Inter-Module Connectivity and Module Selection: Outputs from slots at layer - 1, 21-1 E TR: Mi-M, are selected as inputs similar to the previous architecture. The chosen inputs are passed to a module determined by the selection vector of E IRM,, which is first transformed via a Sofumax function to assign probabilities each module in the shared pool. The module with the highest probability is selected (top-1). The straight-throuuhh estimator is used to compute gradients for both sf and of (see Appendix Aj.
Hereafter, "hierarchically modular NNs" refers to both modular and modular-shared unless otherwise specified.
3 Learning Modular Boolean Tasks
We evaluate the performance of different NN architectures by learning Boolean functions represented function graphs. The truth table derived from these graphs serves as the dataset. Our evaluation focuses on the models' generalization and learning efficiency when only a fraction of the truth table is available for training.
We use three function graphs, each with input nodes and output nodes, as depicted in Figure 2 The complexity is controlled by the number of hierarchical levels levels-greater depth implics increased dependence intermediate sub-functions, making the task more intricate.
Experiment Details:
4

---

Published in Transactions on Machine Learning Research (08/2025)
Hiera cecrachhccal
Hlerarchical
Heraachical depth:
02
30
boaning
Ratio barking
accuracy
depth:
acouracy deoth:
accuracy depth:
Robio of buth DMMe Merringeoo
Rodo buth tolle tainin do12
Rato of bibr in trariinng 0o6o
Monalichic
Honolithic: 255 Density
Morolitnic: 10% Density
Modular
Modular FC
Modular shared
Modular shared FCMS
Figure 3: Test and train accuracy of different NNs relative to training size. For each datapoint, we report the mean and combined standard error (shaded region).
The NN architectures are trained on different portions of the truth table, with training sizes ranging from 0.7 of the total rows. The remaining rows are split evenly between validation and test sets. For each training size, we create three random partitions of the truth table, and each partition is trained with three additional random sceds, resulting in total of nine training runs per architecture per training size. We report the mean and combined standard error, calculated from the three separate means corresponding to the three dataset partitions. To enhance robustness, random noise from (0, 0.1) is added to the inputs during training for data augmentation. All models are trained using the Adain optimizer for 1000 epochs.
To ensure consistency, modular and modular-FC architectures use the same number of modules as there are gate nodes at each level of the function graph (Mittal et al., 2022). Similarly, modular-shared and Moouular-havred-FCM.
architectures align the number of slots with the number of gate nodes, and the count of shared modules matches the number of distinct gates in the graphs. Appendix C demonstrates that varying the number of modules or slots has minimal impact on performance. All module MLPs have uniform structure, with 2 input units, a hidden layer with 12 units, and output unit. We fix the number of inputs to modules in hierarchically modular NNs to 2, ensuring structural consistency while preventing module collapse and optimizing module utilization (Goyal et al.] 2021 Ostapenko et al., 2022). Additional experiments, presented Appendix D explore varying module input dimensions. These results demonstrate that precisc knowledge of the module input size has minimal impact on generalization performance.
Monolithic NNs are configured with 1, 3, or 5 hidden layers match the depth of the hierarchically modular NNs, with each hidden layer containing 36 units. Further architectural and training details can be found in Appendix
3.1 Generalization Performance
Comparing Architectures: Figure shows the generalization performance of different NN architectures
relative to the training size.
1. Monolithic Dense us. Monolithic Sparse: The performance of monolithic NNs is heavily influenced by parameter count. As sparsity increases, test. accuracy declines significantly.
2. Modular us. Monolithie: Modular NNs possess only 30%, 7.1%, and 5.9% of the weights compared monolithic dense NNs for functions with depth 1, 2, and 3, respectively. They outperform monolithic sparse NNs with similar parameter counts (see Appendix Table 1 for details). As the parameter count of monolithic
5

---

Published in Transactions on Machine Learning Research (08/2025)
Traiing efficency Hlerarchical cesth:
Taining efficenc Hlerarciicll depth:
Taining efficecy Hierarciical cesth:
51
63
Monalithic: 100% Denaity
Monolithic: 25% Dersity
Modular
Modutarshhrre
Traning officency Hiorarchiczl cesen:
Triring eftcercy.. Heerarchica deoth.
Traning efficenyy Hierarchical cesth
agari tathitek
Monolihic:
100% Density
Modular
Modulan-shared
Modulan-FC
Modulaa-shared-CCS
Figure 4: FLOPs required to train various NNs as compared to the ratio of truth table available.
NNs increases, their test accuracy converges with that of modular NNs, indicating that prior knowledge of modular sparsity is most beneficial when parameter counts are similar.
3. Modular-Shared us. Monolithic and Modular: Modular-shared NNs consistently outperform both monolithic and modular NNs, benefiting from module reuse that artificially increases the number of samples per module and leads efficient learning of sub-functions. For depth-1 and depth-2 functions, modular-shared NNs significantly outperform the others. However, for depth-3 functions, the performance gap narrows, suggesting that for highly complex tasks, the benefits of modularity and reuse diminish, especially with limited training data.
4. Fixed Connectivity and Module Selection: Modular-FC and modular-shared-Fe FCMS NNs demonstrato superior performance compared other NNs. Motullr-sharrd-PCMM NNs are particularly effective with minimal training data, highlighting the value of reusability and fixed structure, which reduces the complexity associated with concurrently learning both the sub-functions and their organization.
Train Accuracy vs. Test Accuracy: Modular and modular-shared NNs tend to have closely aligned train and test accuracy, while monolithic NNs often exhibit overfitting, particularly with limited training data. This suggests that the inherent inductive bias of hierarchically modular NNs effectively prevents overfitting by aligning the learned representations with the true task structure.
Generalization Relative to Function Complexity: As function complexity increases, all NNs require larger training sizes for effective generalization. A minimum threshold of training data exists for each architecture, determined by its level of prior knowledge, generalize to unseen samples. This threshold increases with complexity, but decreases with greater structural knowledge, such as module reuse and fixed connectivity.
3.2 Training Efficiency
We next evaluate training efficiency based on the number of floating-poin operations (FLOPs) required for training. FLOPs are calculated considering the number of training iterations to reach peak validation accuracy, training size, and the number of weights used in forward and backward passes, as well as in optimization updates. We compare architectures in two groups, including only those that generalize effectively (see Appendix for further details).
Comparing Architectures: Figure presents the FLOPPs required for training different. NNs.

---

Published in Transactions on Machine Learning Research (08/2025)
Structure compatison Illerarchical depth:
Structure comparison Hierarchical depth:: 2
Structure comparisson Hierorchical degth:
Mudvilrsshree
Moduler
Moduler-shaeed
Moduler
Meduter shared
Grgph cdl- disdrce
1
02
25
04
61
02
03
64
6.5
0.6
64
0.
43
641
6.5
5.6
Figure 5: Minimum graph edit distance between the learned connectivity in modular and modular-shared NNs and the ground truth function graphs.
1. Monolithic Dense us. Monolithic Sporse: Monolithic sparse NNs achieve better training efficiency due to reduced parameter counts. There exists range of densities for which sparse NNs match the generalization of dense NNs while reducing training costs.
2. Modular Monolithic: Modular
NÑs use only 30%, 7.1%, and 5.9% of the weights of monolithie dense NNs for depths 1, 2, and 3, respectively, leading to improved training efficiency. However, with sufficient training data, monolithic dense NNs can converge more quickly for simpler tasks, ultimately showing better efficiency, as seen for the depth-1 and depth-2 functions at high training sizes. Monolithic sparse NNs (25% density) can also match the generalization performance of modulor NNs with similar training efficiency but. lack the convergence speed-up seen in dense NNs, likely due reduced parameterization. See Appendix for NN convergence performance.
3. Modular-Shared vs. Monolithic and Modular: Modular-shared NNs consistently demonstrate superior training efficiency compared to monolithic dense NNs, although dense NNs eventually catch up with increasing training size for simpler tasks. Modular-shared NNs require similar FLOPs to monolithic sparse and modular NNs but outperform them in generalization.
Also, hierarchically modular NNs do not achieve faster convergence with larger training size possibly because they need to explore and determine the inter-module connectivity.
4. Fized Connectivily and Module Selection: Modular-FC and modular-shard--CCM NNs achieve the highest NNs perform particularly well, underscoring the computational
training efficiency. Modular-shared-FF advantage of focusing on learning only the sub-functions as compared to learning the sub-functions along with exploring and determining their organization. Further analysis in Appendix shows that structural parameters require higher learning rates than module MLPs.
Efficiency Relative to Function Complexity: As function complexity increases, training all architectures requires more operations. For low-complexity functions, monolithic dense NNs are more efficient compared to others. However, as complexity grows, NNs with prior structural knowledge (e.g,, sparsity, modularity, and reuse) achieve better efficiency. This improvement is directly tied to their ability to generalize effectively as compared to dense NNs with increasing task complexity. Modular-FC and Moddula-sharrd--FCC NNs particularly benefit from prior knowledge of connectivity and module selection, significantly enhancing training efficiency.
3.3 Factors Influencing Generalization in Modular NNs
In this section, we analyze two key factors influencing the generalization of modular and modular-shared NNs: learning the sub-function organization through inter-module connectivity and achieving functional specialization within modules, particularly under limited data conditions.
Learning the Sub-Function Organization: Unlike monolithic dense NNs, which directly learn inpul- output mappings, modular and modular-shared NNs need to identify the underlying task structure to perform effectively. We mcasure how well these NNs capture the true task structure using minimum graph edit distance (Abu-Aisheh al. 2015), comparing the learned inter-module connectivity to the ground truth function graph.
7

---

Published in Transactions on Machine Learning Research (08/2025)
Function specialization (NDR) Herarchical depth:
Function specialization (1OR) Hierarchica depth:
Commation contrrttttteeee crapstude 61
Function specialization (XDR) Hierarchical depth:
Nodalar
Muculu
Muchianshhroe
Nodula
Commaction coAfe WTT
Rat of Iuth table training data
Ratio of table trainin cata
Rato of inthh table in training data
Figure Magnitude of the correlation coefficient wetween NN module output and the XOR truth table
output. Larger values indicate greater functional specialization.
Test accurety Unseen combinattions
Test accuracy Seen combinations
Batoo of dight comtinatiion ontiiinnneeei training dats
Rato ofdigt continations miiiiingg data
OR
AND
XOR
Tieenferr
Monolèhic
Meduler
MedularFFC
Modularshhred
ModrannshamesSCCM5
(a.) MNNST task
(b) Generalization performance of various NN architectures.
Figure 7: Hierarchically modular task based on MNIST and generalization performance of various NN architectures on unseen digit combinations and seen digit combinations.
The learned connectivity is represented as a graph with NN imput units, output units, and all modules as nodes. The input and output nodes must match, while modules can align with any gale node at the same hicrarchical level, ensuring permutation invariance. Figure 5 shows that modular-shared NNs consistently achieve lower graph edit distances compared to modular NNs, indicating a closer match to the ground truth. Notably, the graph edit distance zero when both modular and modular-shared NNs achieve 100% train and test accuracy.
Learning the Underlying Sub-Function: The superior generalization of modular-shared NNs, particularly with lower truth table ratios, suggests an advantage due to module reuse across multiple locations, enabling modules to learn iobsfunctions more effectively with fewer samples. To quantify functional specialization, we use a metric basod on Pearson's correlation coefficient between module outputs and truth table outputs for a specific sub-function.
Let X represent all truth table rows for specific sub-function. We collect the corresponding module outputs and calculate the correlation coefficient p between these outputs and the ground truth. A higher magnitude of indicates greater alignment. between the module's function and the ground truth sub-function.
Figure shows the correlation coefficients for the XOR sub-function in the first hierarchical level of all three functions. Modules in modular-shared NNs exhibit consistently higher correlation with the ground truth compared to modular NNs, with p values closely aligning with generalization performance.
4 Visual Modular Task Based on MNIST
We present results for a modular task constructed using the MNIST handwritten digits dataset, as shown in Figure 7a In this task, two MNIST images, each selected from digits between 0 and 7, serve as input. These images are classified into their corresponding 3-bit binary representations, which are then concatenated and passed to Boolean task. The NNs must first classify each image independently before performing additional operations.
8

---

Published in Transactions on Machine Learning Research (08/2025)
accuracy Modula"
Test accuracy Modulan
Test accuracy
Modular shared
Test accuracy Modularshhared-FCMS
Norbor verernee percrneenn
b6.32
66. 74
61:03
05.36
5B.29
618. 78
06.56
69
70.85
68. 71
75.69
84.95
50
65.21
68.32
17.87
66.16
70.33
77.85
76.29
79.81
90.01
70.03
81.86
96.44
2000
68,
79.
87:13
76.38
85.12
91.07
79.49
82:67
94.94
8.
5.6
D-1
PA
0.4
0.
R0 commtlationn trawiin
Rato of triiiinng data
5A60 tombo traringg data
of dight...ooooo..ooooooooooo comD tiiiing
Figure 8: Generalization performance of different NNs trained on the hierarchically modular MNIST task, evaluated across varying proportions of digit combinations used for training and different numbers of samples per combination. Plots show results for: a. modular NN, b. modular-FC NN, c. modular-shared NN, and d. modular-ssarrd--FC... NN.
We vary the ratio of unique digit combinations used for training. For each training size, a random subset of all possible digit combinations is chosen, with the remaining combinations evenly divided between test and validation sets. We also present results on a test set constructed using seen digit combinations but with unseen digit images. Each training combination contains 1000 samples, while the test and validation sets contain 100 samples per combination. We also examine the effect of different numbers of samples per combination for training.
We adapted the NNs for handling image inputs: in modular NNs, the first hierarchical level contains MLP modules designed for image processing, with 784 input units, two hidden layers (128 and 64 units), and 3 output units. Each module processes one of the two images, and the outputs are concatenated and passed to higher levels, where modules learn Boolcan functions, similar to previous experiments. In modular-shared NNs, two sets of shared modules are employed one set for image processing and another for Boolean functions. The first layer contains two slots, each selecting an image-processing module from the shared pool, with outputs concatenated and passed to higher layers that use shared Boolean modules. For monolithic NNs, the imput size was increased to 784 x 2, and the architecture was adjusted to match the depth and mumber of hidden units in the modular NN. All models were trained for 200 epochs using the Adam optimizer, with three dataset splits and three different seeds for each split (see Appendix Section for additional details).
Generalization on Unseen Digit Combinations: Figure 7b shows the generalization performance on both unseen and seen digit combinations. At training size of 0.4, all NNs show random test accuracy. As the training size increases, modular, modular-shared, modular-FC; and Moodular-sharad-FCM.. NNs start to generalize at different rates, while monolithic NNs do not improve and their test accuracy declines. This decrease may be due to over-fitting on seen digit combinations given that, for larger training sizes, the test accuracy of monolithic NNs on seon digit combinations improves.
The ability to generalize to unseen combinations, known as combinatorial generalization, is a persistent challenge for monolithic NNs (Keysers et al.) 2019 Csordás al.] 2020). However, for Boolean tasks, monolithic NNs demonstrated the capacity to capture the underlying function and generalize well modular NNs, suggesting that simpler tasks and higher data availability can enable generalization.
Modular-shared NNs outperform monolithic, modular, and modular-FC NNs for the MNIST-based task. Additionally, modular and modular-shared NNs closely track their fixed-connectivity counterparts (modular- FC and modular-shared. FCMS, respectively), with small accuracy differences. This may be attributed to the large sample size used in the task, which facilitates better generalization.
Figure S presents the generalization performance of various NNs across different sample sizes per digit combination. We observe that modular-shared--FCM. NNs consistently outperform other architectures when the sample size is reduced to 500 and 100, supporting our previous hypothesis. Additionally, madular-shared NNs exhibit superior generalization compared to modular and modular-FF FC NNs, emphasizing the advantage of module reusability in low-sample training scenarios.
Generalization on Seen Digit Combinations: Figure 7b illustrates the generalization performance on seen digit combinations. It is noteworthy that the validation set contains combinations distinct from both the training and test sets. The overall trends observed in previous results consistent. here. With
9

---

Published in Transactions on Machine Learning Research (08/2025)
larger training sizes, monolithic NNs demonstrate effective generalization. Notably, the modulor-FC and modular-shared--CNN architectures outperform the modular and modular-sharel NNs at larger training sizes, emphasizing the advantage of leveraging predefined connectivity to facilitate learning of the complete task, encompassing both seen and unseen combinations.
Training Efficiency: The FL.OPs required by various NNs during training are shown in Figure 9 Monolithic dense NNs show lower training costs smaller training sizes, primarily because they reach their highest validation accuracy after just epoch for sizes 0,4, 0.5, and 0.6, indicating limited learning. While monolithic NNs match the training efficiency of other NNs at larger training sizes, they still fall short generalization performance.
Train ii efficency MNIST task
NaculandC
Aporosmmtte
For larger training sizes, modular and modular-shared NÑs achieve training efficiencies comparable to those of modular-FC and NNs. This suggests that when sufficient data is available, the advantage of knowing the underlying sub-function organization has a limited impact on training efficiency. However, in data-scarce scenarios, this urructural knowledge becomes crucial for effective training and generalization.
Rabio mibinationss
modular-shared--FCn
Figure 9: FLOPs required to train vari- ous NNs on the hierarchically modular MNIST task.
Learning the Sub-Function Organization: Similar the Boolean function graphs, we compare the learned inter-module connectivity in modular and modular-shared NNs for the MNIST- based (ask by computing the minimun graph edit distance between the learned connectivity and the ground truth function graph.
Structure comparison MNIST task
Moduler
Moduler-shared
Groph edit oistennc
The task graph inchudes two input nodes (one for cach image), cach connected to three intermediale nodes in the finst hierarchical level, corresponding to six output bits (three per image). These nodes then connected to the rest of the Boolean function graph. To construct the learned connectivity graph, represent each image module as three nodes, with incoming connections from the input nodes and outgoing connections to subsequent modules. Our results (Figure 10) indicate that modular-shared NNs more
Ratio of digit combinations in training data
Minimum graph edit dis- tance between learned connectivity and ground truth task connectivity accurately capture the underlying connectivity compared to modular NNs.
Figure 10:
Learning the Underlying Sub-Function: Next, evalu- ate the functional specialization of modules in both modular and modular-shared NNs, focusing on the image classification modules. We assess specialization using the magnitude of the Pearson cor- relation coefficient. Each module produces three outputs, while the ground truth classes are represented 3-bit vectors.
Function specialization MNIST classification Modular
Cormaatton coufiiennt machiaute 15 1
Modultn-chared
We determine the optimal permutation of the output units by maximizing the correlation coefficient and report the highest value achieved. Our results, shown in Figure 11 reveal that modules in modular-shared NNs exhibit greater degree of functional specializa- tion compared to those modular NNs, reinforcing the advantage of module reusability.
combirations training
Figure 11: Function specialization in im- age modules for modular and modular- shared NNs.
5 Conclusion
This work explored how varying degrees of structural knowledge about a task's hicrarchical modularity impacts NN generalization performance and training efficiency. Through experiments involving Boolean functions and hierarchically modular MNIST task, showed that NNs incorporating modularity and
10

---

Published in Transactions on Machine Learning Research (08/2025)
module reusability significantly outperform monolithic and sparse networks, particularly in data-limited scenarios. The improved performance stems from the ability of these NNs to exploit task structure effectively by learning both the specific sub-functions and their organization. Our findings emphasize the importance of explicitly incorporating task structure through modularity and module reusability into NNs, indicating a promising direction for scalable and efficient. learning systems.
Our study is deliberately grounded in tasks with fully known, static modular structure, enabling isolation of architectural factors such sparsity, modularity, and module reuse. Extending these insights to more complex, real-world domains-wiih where task structure is implicit and must be inferred-remaine an important direction for future work. Future work could further explore the theoretical foundations of task learnability with respect to function graph complexity and the degree of encoded structural knowledge. Additionally, applying modular architectures in transfer learning settings and investigating how structural similarity between pretraining and target tasks influences efficiency and generalization may offer valuable directions for scalable learning systems.
Appendix Summary
The appendix provides additional details, results and experiments supporting the claims presented in the main part of the paper.
Technical Details: Appendix A provides implementation details for the datasets and NN architcc- lures, along with link to the codebase. Appendix B describes how training operation counts are computed for each architecture.
Hicrarchically Modular NN Ablations: In the main results, modular and modular-FC architcc- tures use the same number of modules as gate nodes in each level of the function graph. Likewise, modular-shared and modular-sharrd-FCM. match the number of slots to gate nodes, and the number of shared modules to distinct gates. Appendix shows that varying the number of modules slots has minimal impact on performance. All module MLPs share a fixed architecture, and the number of inputs to each module is kept constant. Appendix D further shows that generalization is robust. lo changes in module imput dimensionality.
Multi-task Learning: Appendix E explores a multi-task lcarning setup where NNs are trained to solve multiple Boolean functions simultaneously. This enables task-conditioned modularity, allowing NNs to adapt connectivity and module usage based on a task identifier. Hierarchically modular NNs naturally benefit from this flexibility. Our results show that in the multi-task case, both modular and modular-shared NNs outperform dense monolithic NNs.
Design Choices and Hyperparameters: Appendix F presents additional experiments supporting the architectural and training choices for modular and modular-sharer NNs. These include comparisons of input selection strategies in modular NNs, and module selection mechanisms in modular-share NNs. We also examine the effect of different learning rates for structural and functional parameters, finding that higher learning rates for structural parameters tend to improve generalization.
Additional Results: Appendix G provides supplementary visualizations offering alternative per- spectives generalization performance and training efficiency across the different architectures.
References
Zeina Abu-Aisheh, Romain Raveaux, Jean- -ves Ramel, and Patrick Martineau. An exact graph edit distance algorithm for solving pattern recognition problems. In 4th International Conference on Pattern Recognition Applications and Methods 2015, 2015.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEFE conference compuler vision and pottern, recognition, pp. 39 99-48, 2016.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Hum Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what is required and can it be learned? or preprinl arXiv:1811.18899 2018.
Damitry Bahdanau, Harm de Vries, Timothy J O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:1912.05788 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXis:1808.3339, 2013.
Michael Chang, Abhishek Gupla, Sergey Levine, and Thomas L Griffiths. Automatically composing representation transformations as a means for generalization. orXiv preprint arXiv:1807.046400 2018.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G Learned-Miller, and Chuang Gan. Mod-squad: Designing mixtures experts as modular multi-task learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion, pp. 11828-11837, 2023.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through diffcrentiable weight masks. arXiv preprint arXiv:2010.00066 2020.
Vanessa D' D'Amario, Tomotake Sasaki, and Xavier Boix. How modular should neural module networks be for systematic generalization? Advances in Neural Information Processing Systems, 34:23374-23388 2021.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint ar Xio: 701.08834, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arxiv preprint Xix::9999.. 10893, 2019.
Anirudh Goyal, Aniket Rajiv Didolkar, Nan Rosomary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Curtis Mozer, and Yoshua Bengio. Neural production systems. In Advances in Newrul Information Processing Systems, 2021.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335 29347, 2021.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 804-813, 2017.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalisc? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Devon Jarvis, Richard Klein, Benjamin Rosman, and Andrew M Saxe. On the specialization of neural modules. arXiv preprint w=Xiv:2409.149811 2024.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Mcasuring compositional generaliza- tion: A comprehensive method on realistic data. arXie preprint urXiv:1912.0991 2019.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. A Aduacces in neural information processing systems, 31, 2018.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-seqquecee recurrent networks. In International conference on machine learning, pp. 2973-2882. PMLR, 2018.
Samuel Lippl and Kim Stachenfeld. The impact of task structure, representational geometry, and learning mechanism on compositional generalization. In ICLR 2024 Workshop on Representational Aligument, 2024.
Shreyas Malakarjun Patil, Loizos Michael, and Constantine Dovrolis. Neural sculpting: Uncovering hierar- chically modular task structure in neural networks through pruning and network analysis. Advances in Neural Information Processing Systems, 36, 2024.
Jorge A Mendez and Eric Eaton. Lifelong learning of compositional structures. 2rXiv:2007.07332 2020.
arXiv preprint
Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie. a modular architecture enough? Advances in Neural Information Processing Systems, 35:28747 28760, 2022.
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Contimual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298-30311 2021.
Oleksiy Ostapenko, Pau Rodriguez, Alexandre Lacoste, and Laurent Charlin. Attention for compositional modularity. In NeurIPS'22 Workshop on All Things Attention: Bridging Different Perspectives on Attention, 2022.
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control self-assembling morphologies: a study of generalization via modularity. Advances in Newral Information Processing Systems, 32, 2019.
Edoardo M Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining modular skills in multitask learning. Xiv preprind arXiv:2202. 13914, 2022.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. Task-driven modular networks for zero-shol compositional learning. In Proceedings of the IEEE/CVE International Conference 2019.
on Computer Vision, pp.
Nasim Rahaman, Muhammad Waleed Gondal, Shruti Joshi, Peter Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. Dynamic inference with neural interpreters. Advances in Newral Information Processing Systems, 34:10985-10998, 2021.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239 2017.
Simon Schug, Seijin Kobayashi, Yassir Akram, Maciej Wolezyk, Alexandra Proca, Johannes Von Oswald, Razvan Pascanu, João Sacramento, and Angelika Steger. Discovering modular solutions that generalize compositionally. arXiv preprint arXiv:2312. 15001, 2023.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsoly- gated mixture-of-cxperts layer. orXiv preprint urXis:1101.00533 2017.
Herbert A Simon. The architecture of complexity. In Facets of systems science, pp. 457-476. Springer, 1991.
Alexander V Terekhov, Guglielmo Montone, and J Kevin O'Regan. Knowledge transfer in deep block-modular neural networks. In Biomimetic and Biohybrid Systems: 4th International Conference, Living Machines 2015, Barcelone, Spain, July 28-31, 2015, Proceedings 4, pp. 268-279. Springer, 2015.
Tom Veniat, Ludovic Denoyer, and Mare' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint rxXi:2202.12633 2020.
Jianan Wang, Eren Sczener, David Budden, Marcus Hutter, and Jocl Veness. A combinatorial perspective transfer learning. Advances in Newral Information Processing Systems, 33:918-999 1299,,999-99 2020.
13

---

Published in Transactions on Machine Learning Research (08/2025)
A
Implementation and Training Details
The GitHub Modular-NNa
repository
can
be accessed here:
This section presents the implementation details for datasets, NN architectures, and training settings for all architectures.
A.1 Dataset Construction and Hyperparameter Tuning
For Boolean functions, we generate a truth table rows (6 inputs). Rows are split into training, validation, and test based on the specified training ratio and dataset seed. Each training size has three different splits corresponding to three different dataset seeds.
Boolean Functions:
The MNIST task uses pairs of digits (0-7), resulting in 64 combinations. These combinations are divided into training, validation, and test sets, similarly to the Boolean functions. For each combination, we randomly select image pairs in the MNIST training set based on the specified sample size per combination. Test sets use image pains from the MNIST test split. Datasct splitting is performed using three different seeds. When varying the number samples per combination, the validation and test sets remain consistent.
MNIST Task:
Training and Hyperparameter Tuning:
The NNs are trained using the Adam optimizer for 1000 epochs for Boolean functions and 200 epochs for the MNIST task. The loss function is bitwise cross-entropy with Sigmoid activation. A grid search over learning rale, batch size, and weight decay is used to select optimal hyperparameters based on validation accuracy. We use seeds (40, 41, for dataset splits and 1, for NN initialization and training. We independently tune the hyperparameters for each dataset split and training size by maximizing the validation accuracy, averaged over the three training seeds.
A.2 MLPs and Random Sparse MLPs
We MLPs with ReLU activations at the hidden layers, Sigmoid at the out layers, and Xavier weight initialization (Glorot & Bengio 2010). Sparsity in monolithic NNs is achieved by pruming edges based on uniform random score.
Architecture Details:
Boolean functions with depths of 1, 2, and 3 use MLPs with 1, 3, and 5 hidden layers (36 units each). The MNIST-based task uses MLPs with 784 x 2 input units, 2 output units, and 6 hidden layers with 256, 128, 64, 36, 36, 36 units.
Hyperparameter Sets: For Boolean functions, and weight decay f0.001, 0.0001). For MNIST, we test learning rates f0.01, 0.001), batch sizes {128, 256, 512), and weight decay {0.001, 0.0001}.
learning rates (0.1, 0,000.11,0000 0.001), batch sizes (4,64),
A.3 Hierarchically Modular NNs
The architecture has L hierarchical layers, each with Mi modules. Each module mf has functional parameters (MLP) and structural parameters (input selection vector of E MR--)). The input selection vector, initialized with values from standard normal distribution, determines module input connectivity.
Overall Architecture:
Module Input Selection: For module mf. we apply the Sigmoid function to the input selection vector sf to get pi, then select the top-2 values to generate one-hot encoded binary masks by E {0, 1] M and by E {0, 1] Mi- These masks isolate inputs from æ1 using dot products, resulting in inputs xn (1, = by o x0-1 and 22 i) = b2 2 æ1-1. The trraighttthrough estimator is used to estimate gradients.
For image modules, the Softmax function is applied to sf and one input image is selected. A binary mask, b E (0,1) 1 is generated and applied to each pixel position across the two images.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Task
Hierarchical depth
Hierarchical depth 2
Elerarchicall depth
MNIST
Nomber of weighto
F
B
U
B
U
F
B
U
B
Mouclihic
268
255
258
2650
2850
2550
6472
6472
5472
442,744
442.744
442,744
Modular
100
100
56
306
335
356
108
Modular-FC
72
72
72
180
180
180
258
256
256
217,002
217.662
217,652
Sewblaaeehhored-CCCCC PCMS
72
72
36
180
180
72
256
298
108
217,032
217,652
106,808
Table 1: Number of weights involved in forward pass (F), backward pass (B) and gradient based update (U) of various NNs for different tasks.
Forward Pass: Each module's input is processed by its MLP, and outputs are concatenated before passing to the next layer. This is repeated until the final layer, where outputs are selected from the last set of modules using input selection vectors.
Module MLP Architectures: For Boolean sub-functions, module MLPs have 2 input units, output unit, and a hidden layer with 12 units. For MNIST, module MLPs have 784 input units, 3 output units, and 2 hidden layers (128 and 64 units). Xavier initialization is used for weights.
Hyperparameter Sets: Hyperparameters include learning rates (0.1, batch sizes {4, 64), and weight. decay (0.001, 0.00011 } for Boolean functions. MNIST uses learning rales (0.01, 0.001 batch sizes {128, 256, 512), and weight decay {0.001, 0.00013. Structural and functional parameters use separate learning rates, and activation function temperature (r) for input selection vectors is also tuned ({1.0, 2.0, 5.0)).
The learning rate values lested here for Boolean functions subset of the one used for monolithic NNs while the batch size and weight decay values the same. The learning rates and temperatures used here are selected from broader range of values based on results in Appendix and
Fixed Connectivity: In fixed inter-module connectivity, imput selection vectors are fixed, and gradients are not computed for them. A single learning rate is used for functional parameters, and the hyper-paramcter sets tested are consistent with the setup for monolithic NNs.
A.4 Hierarchically Modular NNs with Shared Modules
The architecture ure consists of L layers with Mt slots, filled by modules from a shared pool of M modules. Each slot has an input selection vector (s)) and a module selection vector (uf). Both vectors are initialized randomly with samples from the standard normal distribution.
Overall Architecture:
Input selection follows the same procedure as for standard hierarchically modular NNs. For module selection, the Softmax function is applied to of to select module from the pool and a binary mask, b E {0, 1 is constructed. The inputs to the slots are passed through all M modules, and the slot output is computed using a dot product between the module outputs and the binary mask. The ttraight-through estimator is used for gradient calculation. For image slots, the module selection mask is applied independently at each module output position.
Input and Module Selection:
Each slot processes inputs using a selected module, and the outputs are concatenated and passed to subsequent layers. The module MLP architecture, training, and hyperparameters are consistent. with those used for hicrarchically modular NNs.
Forward Pass:
Fixed Connectivity and Module Selection: In this variant, both input and module selection vectors are fixed, with no gradients computed. The same hyperparameters are used as for monolithic NNs.
B Training Efficiency Details
This section describes the methodology used to calculate the number of floating- point operations (FLOPs) required during training across various NN architectures. In practice, GPU hardware performs dense matrix multiplications, even when the weight matrix is sparse. However, in the hierarchically modular architectures, computational savings arise from modules that are small, independent MLPs cach with significantly fewer parameters than their monolithic counterparts. These smaller MLPs naturally lead to reduced FLOP counts, irrespective of GPU implementation details. In the case of sparse monolithic hic NNs, we report FLOP reductions
15

---

Published in Transactions on Machine Learning Research (08/2025)
Namber of herations Herarthical depth:
Number of to retionn Herarchical depen:
Number of itferstione
Herarchic depth:
ithc: 100% Dansity
Monelmic: 35W, Danctty
Nodular
Medulla-shardd
Nimber of horations
Herarchital depth:
Number of to terotions Hereronicall depen:
Number of iteerstionn Herarch deeth:3
Moduler
Vidu anstaared
ModulanFCC
Modulen-thare--FCCC
Figure 12: Number of weight updates (training iterations) for various NNs to reach peak validation accuracy different Boolean functions compared to the ratio of truth table available.
based on parameter sparsity; however, these do not translate to actual computational gains on GPUs due lo the lack of native support for sparse matrix operations.
The three main factors determining the FLOP count are: the number of samples training size, the number of parameters, and the number of training iterations (or weight updates) needed to reach the highest validation accuracy (i.e., early stopping). Figures 12 and 13 present the number of training iterations required by different NN architectures, while Table D summarizes the weights involved in forward, backward, and gradient update processes for various tasks.
Number of terations MNIST task
Modi
Mode inshorddtrOs
X00
natio digf combrnatiore trarnin
In each training iteration, given a batch size b, there are forward passes, b backward passes, and one weight update. For dataset of size D over epoch, this results in D forward passes, D backward passes, and [D/b] + 1 weight updales. The total FLOP count is computed by multiplying the number of epochs by the operations performed during all forward passes, backward passes, and weight updates per epoch. FLOPs related to activation functions and biases are excluded from this calculation.
Number of weight up- dates (training iterations) for various NNs to reach peak validation aceu- racy the MNIST-based task.
Figure 13:
Let W be the number of weights in a dense, monolithic NN. The FLOP count for single forward pass through the lincar layers is 2 x W. For the backward pass, this count is 4 x W. Weight updates using the Adam optimizer require 18 x W operations.
Monolithic NNs:
For random sparse monolithic NNs, the number of weights is scaled according to the network's density, and FLOP calculations follow the same approach as for dense NNs.
Hierarchically Modular NNs: In hierarchically modular NNs, the total FLOP count. includes both operations from the forward pass through each module and those from the module input selection mechanism. For Boolean function modules, input selection involves two dot products; for image modules, it involves 784 dot products, effectively introducing additional units (2 for Boolean modules and 784 for image modules) to process the full output of the previous layer. The parameters for input selection vectors are also considered in weight updates, and the FLOP counts for the forward and backward passes incorporate 2x or 784x the parameters for the input selection vectors for Boolean and image processing modules, respectively. The rest. of the calculations follow the previously described procedures.
In the variant with fixed inter-module connectivity, we do not include any FLOPs related to input selection and only consider the weights within the module MLPs for the FLOP calculations.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Hierarchically Modular NNs with Shared Modules: For hierarchically modular NNs with shared modules, each slot processes its specific inputs through all shared modules. A dot product is performed between the activated module selection vector and the outputs of all modules, increasing the number of parameters involved in the forward pass.
Let W. represent the number of weights in each module MIP,, N, be the number slots, and M the number of shared modules. The number of weights used in the forward pass is Wm N, M. In the backward pass, the number of active weights is reduced to W., x N, because the module selection mask is binary, leading to zero gradients for unselected modules in each slot.
For accounting for FLOPs associated with input selection vectors during forward and backward pass we utilize the same procedure as described for hierarchically modular NN. Module selection involves a dot product between the outputs of all modules and the module selection mask, effectively adding units (1 for Boolean modules and for image modules) at the top of cach slot. The forward and backward pass incorporates 1 3x the parameters for the module selection vectors for Boolean modules and image processing modules respectively.
Finally, for weight updates using the Adam optimizer, we account for the parameters in the input selection vectors, module selection vectors, and all shared modules.
The number of weights involved in the forward pass, backward pass, and optimizer updates is scaled according to the respective operations (refer to details for monolithic NNs). The total FLOP count then obtained by multiplying these operations by the number of training epochs.
For the variant with fixed inter-module connectivity and module selection, FLOPs related to input and module selection are excluded, and only the weights in the module MLPs are considered. During the forward and backward passes, the number of active modules equals the number of slots, while weight updates are applied only to the shared modules.
C Experiments with Arbitrary Modular Architectures
In this paper, we initialize modular NNs with number of modules that matches the function graph at each hierarchical level. For modular-shared NNs, we use the same number of slots per hierarchical level as in the function graph, while the number of shared modules corresponds to the number of distinct gates in the function graph. However, an important question arises: how effective are these hicrarchically modular architoctures when such structural information is unknown, and an arbitrary number of modules or slots are initialized?
In this section, present results for scenarios where vary the number of modules/slots at each hierarchical level, as well as the number of shared modules for the moduldr-shared architecture. These experiments provide insights into the flexibility and robustness of hierarchically modular NNs under less structured initialization conditions. We consider the Boolean function shown in Figure 14 and the hierarchically modular NNs are trained following the methodology outlined in Section A
XOR
Transfe
Figure 14: Function graph
C.1 Hierarchically Modular NNs
In the case of modular NNs, as depicted in the function graph in Figure 14 the NN requires three modules at the first hierarchical level and two modules at the second. To evaluate the flexibility of the module count, we experiment with three additional architectures, where each hierarchical level is assigned M modules, varying M from 3 to 9.
Each architecture trained independently using different training size ratios of the truth table, as described in Section A The resulting training and test accuracy values are shown in Figure 15
17

---

Published in Transactions on Machine Learning Research (08/2025)
Test Moduter Mie Varying number modules
Tann Modular Nike Varying number modules
0.1
02
3.3
04
23
0.8
0.7
0.5
02
23
04
0. 0.7
Ground truch
Num modules:
Nurn medules: 6
Num modules: 9
Figure 15: Train and test accuracy as compared to the number of modules defined in each layer of the modular NN. The black lines represent the accuracy for the model with ground truth number of modules in each layer.
Interestingly, the performance across these architectures remains comparable the NN that uses the ground truth number of modules per hierarchical level. Furthermore, our analysis does not show a clear trend where increasing or decreasing the number of modules consistently improves performance, suggesting that the architecture is robust to variations in module count.
C.2 Hierarchically Modular NNs with Shared Modules
For modular-shared NNs, we explore architectural variations along two dimensions, organized in grid. The first dimension is the number of slots per hierarchical level, ranging from (3, 6, 9}, and the second is the total number of shared modules, varying across {2, 8). This setup results in nine distinct architectures, each defined by a unique combination of slot and shared module counts.
Each architecture trained independently using different splits of the truth table, with varying fractions allocated for training. The results are presented in Figure [6] with plots segmented by the ratio of the truth table used for training. The horizontal black line in the figure indicates the performance of the architecture configured with the ground truth number of slots and shared modules.
Consistent with our previous analysis, we observe that these architectures perform comparably to the ground truth setup. However, modular-shared NNs with a larger number of slots outperform the ground truth configuration at smaller training sizes. Additionally, modular-shard NNs with more shared modules than slots (e.g., three slots per hicrarchical level and two shared modules) exhibit reduced generalization performance at larger training sizes, suggesting that an imbalance between slots and shared modules may limit effective module reuse.
D Effect of module input size on generalization performance
Previous research has demonstrated that constraining module input dimensions introducing input bot- tlenecks enhances functional specialization, thereby improving generalization performance. The underlying intuition is that by promoting sparsity-cconiideriin considering only subset of outputs from preceding layers the modules more effectively capture the sub-function structure they are designed to learn. In our experiments, we align the number of module inputs with the dimensions of their corresponding sub-functions: two inputs for Boolean modules and one input for image modules. This section presents the experiment and results obtained by varying the input dimensions of the modules in modular and modular-shared NNs. We consider the Boolean function shown in Figure 14 and the hierarchically modular NNs are trained following the methodology outlined in Section
We evaluate modular NNs configured with 6 and 9 modules at each hicrarchical level, allowing the module input dimensions to vary between 2, 4, and 6. If the number of modules were reduced to fewer than 3 or 2, the maximum possible input dimensions for modules at higher hierarchical levels would be constrained by this reduced number.
Modular NNs:
When the module input dimension matches the number of outputs from the previous layer, we explore two approaches: 1. Feeding the entire output directly into the modules without leveraging the module's input.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Test acourscy Sodubbrshored Detaset spih 0.1
Taln scouracy Notirrrseeddd MN. Dutaset spit: 0.1
6cta each leve
Mumber Metei Nareronce
accurary
sccurary
echiieeacccca
Tist sccurscy Modularshared MN Dstacet spitt 0-3
Talh socuracy Nodularshaned MN Detaget spit: 0.3
Wamrbe ditts each Henarchic love
Mumbe skis each Neraroncal leve
lest eccuracy Modularahhred
spát:
bain accurasy Kodulanatanee Dalasee sprit
Te-t Modidarsbaree Datasse spith 0.5
Tall Nodailentaree Datavee spith 0.5
Mariber 9415 oechiiiacccclll lewe
Mumber skts each Netronical leel
Test accurecy- Hodubbrshared KN Dutasct spilt: 0.6
Taln scouracy Nodubrshhaee KN Detaset spilt: 0.6
Tea
Nambee 6sts each leve
Mumber aabit Neraronicll
eccuracy
spilt
accursty
sn Nenrovical
Ground truth
Num modules:
Num modules:
Num modules:
Figure 16: Train and test accuracy as compared to the number of slots defined in each layer and the number of modules the modular-shared NN. The black curve represents the accuracy for the model with ground truth number of slots in each layer and the ground truth number of shared modules.
19

---

Published in Transactions on Machine Learning Research (08/2025)
(Medular
Train (Modular MN)
Hicrarchical depth: Num modules:
Hierarchical depth: 2, Num modules:
80
15
14
05
Rabio truth bable training data
Ratto of training data
Test accuracy (Modular NN)
Train accuracy (Modular NN) Hierarchical depth: 2, Num modules: 9
Hierarchical depti:: 2, Num modules:
Thit ACTUINYy
Rebro of buith litle in trarnin dala
Nubua of truth Lob%: in braming dota
Input= 2
nout
Input (Permutable)
Input (Fixed)
Figure 17: Train and test accuracy as compared to the number of module inputs in the modular NN.
selection vectors; 2. Utilizing input selection vectors to rank and reorder inputs. Figure 17 presents the training and testing accuracy for modular NNs with varying numbers of modules and input dimensions.
Our findings indicate that a module input dimension of 4 yields the highest training and testing accuracy. However, configurations with 2 inputs (aligned with the sub-functions in the function graph) 6 imputs (matching the overall input dimension of the NN) perform comparably al higher training sizes. Notably, when the imput dimension is set to 2, performance slightly surpasses that of 6 at lower training sizes. Conversely, modules with 6 inputs achieve better training accuracy than those with 2 inputs, indicating that larger input dimensions may facilitate over-fitting
This result indicates that knowledge of the exact module input size (ground truth of 2) does not significantly impact the generalization performance of modular NNs. However, introducing some form of input sparsity appears to provide slight improvement in generalization performance, likely by promoting better specialization and alignment with the underlying task structure.
For modular-shared NNs, we experimented with architectures comprising and 9 slots at each hierarchical level and module input dimensions of 2, 4, and 6. The corresponding results are illustrated in Figure 18
Modular-shared NNs:
The results show that module input sizes of 2 and 4 yield significantly higher test accuracy compared to an imput size of 6 when slots are not allowed to reorder or permute their inputs. However, when slots are permitted to perform input permutations, the advantage of input sparsity (input sizes of and 4) diminishes, with only a slight margin of improved performance over input size 6.
Effect of Permutations with Full Module Inputs: A key observation is that when module's input dimension equals the full output of the previous layer (i.e., no input sparsity), allowing modules to permute their inputs significantly improves performance in modular-shared NNs. For example, consider a module with more than two input units tasked with learning the XOR function for two inputs. If the module learns the XOR gate for two specific input positions, it can only be reused in slots requiring XOR computation for those same positions. By enabling input permutation, the module generalizes and can be reused across any slot where XOR functionality is needed, regardless of input order. This enhanced flexibility improves module reuse and contributes to better generalization.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Test (Modstn-shared MK0
Thin (ffodubenshhreee NK)
Herarchical daplis Num Siets: 6, Num moduless: 2
Hicrarch degtth 2. Num Stats b. Nurm medules;
Sent REOUPNcy
70
06
0.1
Batl truth batle training data
table training data
(Modularahared
rchicai depti: Num Slets: 9, Num modules: 2
Hierarch depth: Num Slats. 9, Num modules;
Thet SECUInCy
Ratio of Inuth bable in training data
Rable brutt table in braining data
Input-
Input-
Input 6 (Permutetle)
Input (Fixed)
Figure 18: Train and test accuracy as compared to the number of module or slot inputs the modular-shared NN.
Overall, the results indicate that knowledge of the exact module input size (ground truth of 2) does not significantly affect the generalization performance modular-shared NNs when input reordering is allowed, facilitating effective module reuse. Neverthcless, introducing input sparsity (e.g., input dimensions of or 4) provides slight improvement in generalization, likely due to better functional specialization and aligument with the task's modular structure. These findings suggest that the superior generalization performance of modular-shared NNs is primarily driven by module reuse, with input sparsity acting as complementary factor that refines specialization.
E
Multi-task learning
We extend our experimental setup to explore multi-task learning scenarios where each neural network (NN) architecture is trained to perform multiple Boolean functions simultancously. This allows us evaluate conditondd modularity by enabling networks to adjust connectivity and module selection based on a task identifier provided with each input sample.
E.1 Experimental set-up
Consider a set of n Boolean function graphs (or tasks), and their corresponding truth tables Te for i E (1,2,...,,)) n.. Consider the experimental setup described in Section 3 of the paper. For each task i, we sample an r-fraction of rows from T,, prepend a one-hot encodod task identifier i to each row, and combine these samples from all tasks into unified training set. This process is repeated to create the validation and test. sets.
We train three architectures:
The input dimension of the architecture is expanded to include the task identifier. The network is then trained using standard methods.
Monolithic dense:
Modular: The modular NN configured similarly the single task setting, with L modular layers, containing Mj modules at layer I. Each module consists of ML.P comprising 2 input units, 12 hidden units, and
21

---

Published in Transactions on Machine Learning Research (08/2025)
Task:
Tank: 2
Task:
OR
AND
XOR
Transfer
Figure 19: Function graphs used in the multi-task learning setting.
output unit, along with an input selection vector. The input selection vector is a twoo-dimensional matrix of size n x Mi11 where n represents the number of lasks.
Given the one-hot encoded task identifier associated with each input sample, the corresponding row of the input selection vector is used to determine the module's inputs or inter-module connoctivity. The input selection vector can be interpreted the weight matrix of single linear layer that takes the task identifies ifier as input and generates the input selection vector specific to the module and task.
Please note that the input selection vectors for all the modules are learned or trained along with all the MLP weights.
Modular-shared. Similar to the single-task modular-shared NN, this architecture replaces individual modules with M2 slots in each hierarchical layer and employs pool of M shared modules. Each slot is defined by input selection vector of size M-1 and module selection vector of size M.
For each input sample, the task identifier determines the corresponding rows of these matrices, specifying both the inter-module connectivity and the module selection at each slot. The input selection and module selection vectors for all the slots are learned or trained along with all the MLP weights.
E.2 Results
We consider the three function graphs shown in Figure 19 in the multi-task setting.
Learning two tasks (1 & 2): First, we present results for learning two of the three tasks. Here, the monolithic dense NN consists of 8 input units, 2 output units, and 3 hidden layers with 60 units in each layer. The modular NN comprises 5 modules in the first layer and 4 modules in the second layer. The modular--hhred NN consists of 4 slots in the first layer, 2 slots in the second layer, and 3 shared modules.
The test accuracy and the training efficiency are presented in Figure 20 column 1). Both modular and modular-shared NNs consistently outperform the monolithic dense NN. The accuracy gap between modular and modular-sharel NNs is low, likely due to the sub-function outpul reuse across the two lasks (e.g., 2 sub-function outputs with XOR gates are common or reused between tasks 1 and 2). Modular NNs with particularly effective in capturing sub-function output reuse both across and
their fixed module positions, within tasks. (See section 2 of the paper for definition)
In terms of training efficiency, both modular and modular-shared NNs require significantly fewer operations to learn the two tasks effectively. Additionally, modular NNs are able to match the training efficiency of shorear-shored NNs.
Learning two tasks (1 & 3): Next, we present results for multi-task setting where tasks 1 and 3 are learned simultancously. The monolithic NN consists of 8 input units, 2 output units, and 3 hidden layers with 72 units in each layer. The modular NN consists of 6 modules in the first layer and 4 modules in the second layer. Finally, the modullr-sharre NN comprises 3 slots in the first. layer, 2 slots in the second layer, and 3 shared modules.
The test accuracy and the training efficiency are presented in Figure 20 column 2). Both modular and modular-shared NNs again outperform the monolithic dense NN in terms of test accuracy and training
22

---

Published in Transactions on Machine Learning Research (08/2025)
accuracy Number tasks: Tasks;
Number tasks: Tasks: 1
Test Mumber of
Modee Modd
Hecltr Hoc reculonraadd
55
E1 Ranio
31
60
Ratio
tairis
Ratto tmininng
accuracy
Train acouracy Number of tadis 2, Tasks: &
Thin accuraty Number of tasks
740 K05.19
Vassini Vollinstard
6-5
En
47
Traiiin efficency Number
Teining efficency
Tainigg efficency
Agr Sent menoea
CA
Tatit
Figure 20: Test accuracy, train accuracy, and training efficiency results of various NN architectures relative training size in the multi-task learning sct-up. For each datapoint, we report the mean and combined standard error (shaded region).
efficiency. However, there is a more significant performance gap between modular-sharee and modular NNs in this case. Since tasks 1 and 3 do not share sub-function outputs, the performance advantage of modular NNs due common sub-function outputs is absent, leading to a larger difference in accuracy and efficiency.
Finally, we examine the multi-task setting where all three tasks are learned simultaneously. The monolithic NN has 9 input units, 2 output units, and 3 hidden layers with 72 units per layer. The modular NN configured with 6 modules in both hierarchical layers. The modular-shared NN has slots in the first layer, 2 slots in the second layer, and 3 shared modules.
Learning three tasks:
Both modular and modular-shared NNs significantly outperform the monolithie dense NN in terms of generalization performance and training efficiency.
Across all training sizes, modular and modular-shared NNs show comparable test accuracy. This similarity can be attributed to the substantial amount of sub-function output reuse, which modular NNs can exploit duc to their fixed module positions. However, modular-shared NNs consistently achieve superior training efficiency due to their reduced number of trainable parameters and the abundant sub-function operation reuse across the three tasks. (See section of the paper for definition)
E.3 Observations and implications
The input or musk-identifier-conditoo inpul-output, function can adapt based on the task. In a single-task setting, where the input-output function is fixed, this advantage is less apparent, For this reason, we focused on a single-task or static setting earlier and found that modularity without module reusability does not improve generalization performance. However, in multi-task setting where the task identifier is available, even modular NNs outperform monolithic dense NNs.
connectivity inherently favors hicrarchically modular NNs, as their
23

---

Published in Transactions on Machine Learning Research (08/2025)
accuracy Modular
Train accuracy Modular
1o-- Sumbel
lep- Gumbel
Tes accurary
0.5
treining
Figure 21: Train and test accuracy of modular NNs as compared to the ratio of the truth table used for training. The varions bars indicates the addition of Gumbel noise or direct top-k for module input selection.
F
Hyperparameter tuning and selection
In this section, we present additional results to support the architectural and training choices for modular and modular-shared NNs. These experiments are based on the function graph shown Figure [4] with varying proportions of the truth table used for training. Dataset details remain consistent with those described Section A
We perform grid search over learning rates for both structural and functional parameters, well weight decay values. Tested learning rates include (0.1, 0.01, 0.001), while weight decay values are (0.001, 0.0001 }. The batch size is set to use all available training samples in single batch, and all networks are trained for 1000 epochs using the Adam optimizer.
F.1 Connectivity and Module Selection
In Section we described the process learning structural parameters in modular and modular-shared NNs. Here, we present experiments to justify the use of the top-k operation for input connectivity and module selection in both network types.
Test accuracy Modular NN, Top-k
Trin accurecy Modular NN, Top-k
Taan SCCU
Test
0.1
0.3
Ralio of Iruth table in training dela
Patio of truth table in training date
Test accuracy Mocular NN, Gumbel
Train accuracy Modular NN, Gumbel
Tan
0.1
0.5
Radio of bruth rable in trairing data
Ratio of truth cable in training data
Tau: 0.1
Tau: 0.5
Tau: 1.0
Tou: 2.0
Tau: 5.0
Figure 22: Train and test accuracy of modular NNs as compared to the temperature (tau) values used for module input selection using Gumbel noise.
24

---

Published in Transactions on Machine Learning Research (08/2025)
Test accuracy Medular-charad
accuracy Modular-charod
look 20 Gumbel
lop-k Gumbe
Thain eccureyy
2
0.3
testing
traitinn
Figure 23: Train and test accuracy of modular-shared NNs as compared to the ratio of the truth table used for training. The varions bars indicates the addition of Gumbel noise or direct top-k for module input selection.
F.1.1 Hierarchically Modular NNs
Consider the input selection vector for module mj, denoted as sf- The goal is to use this vector to score and select & indices along with their corresponding input values for the specific module. First, the Sigmoid function applied the vector, yielding pi - as))).
Previous training methods for hierarchically modular NNs have enhanced the exploration of different connectivity patterns by adding Gumbel-distributed noise to the input selection vector before applying a normalization function. This process allows for the effective selection of the top-k indices, promoting exploration during training.
We investigate a variant of this process where Gumbel noise is added to si before selecting the top-k indices from pl. A grid search over the temperature parameter (r) used to normalize the vector afler adding Gumbel noise also performed to identify the best configuration. This approach aims to balance exploration and exploitation, reducing the likelihood of premature convergence to suboptimal input configurations while improving lcarning capability.
The results are shown Figure 21 We observe that the standard top-k selection method significantly outperforms the Gumbel bosee variants. We also evaluate the effect temperature on input selection performance. As depicted in Figure [2] higher temperature values yield better results. Increased temperature facilitates uniform exploration of the input selection vector, contributing to improved learning outcomes.
F.1.2 Hierarchically Modular NNs with Shared Modules
We now present the results for the modular-shared architecture. Let st and of denote the input and module selection vectors for given slot, respectively. The goal is to select inputs and one module for each slot. The Sigmoid function is applied to sf and the Softmax function is applied to of to compute the selection scores.
We compare direct top-k selection to a variant that uses Gumbel noise to enhance exploration. In this variant, Gumbel noise added to the selection ion vectors before applying the normalization functions, which aims to avoid immediate convergence to a specific set of inputs or modules, promoting broader exploration during training.
Figure 23 demonstrates that Gumbel noise-based variants perform worse than the standard top-k selection. Furthermore, as shown in Figure 24 higher temperature values during input selection improve the model's performance, consistent with the findings for hicrarchically modular NNs. This effect is due to increased exploration, preventing ure convergence and enhancing learning outcomes.
F.2 Learning Rate Analysis for Structural and Functional Parameters
We analyze the impact of learning rates on both the structural and functional parameters in hierarchically modular NNs. For both the modular and modular-shared architectures, we begin by fixing the learning rate combinations and then fine-tuning other hyperparameters, including weight decay and temperature values. The batch size is kept constant so that all available samples used in each training iteration.
25

---

Published in Transactions on Machine Learning Research (08/2025)
accuracy Modufar-chared Top-k
accurary Modulla-charrd Top-k
20
Thain eccUrecy
0.3
training
training
Test accuracy Modular-snared Gumael
Madular-chared Gumbel
ROCUFSCY
0.1
0.5
25 training
0.1
0.3
U.S
U.5 training
Figure 24: Train and test accuracy of modular- shared NNs as compared the temperature (tau) values used for module input selection using Gunbel noise.
Ioan Eoen-ditaoet onllor 0.1
Ban dotieet Fatio 0.3
Via Oibieet lalo 3.3
The acouracy as learring piiet.
13.30
69.73
63:20
74.40
75.85
26.57
15.15
25:35
$5.39
$4.44
75:00
70.03
Maschtna fournnng siid
Stochard foanning Mii
Sonutara foarning sele
Syociand bor rele
Thain Taln-dalzeet deleaet natroc d.1
Tuan delases natioc 0.3
Tha delasie wator 0.3
Trai dabaset nutlo 6.7
12.04
7b:02
60.39
990.0
er.35
PAR NEN0S loating 1dl
24.25
91.42
$1.44
PA PAPCORA (earent
$2.16
25:04
00.55
Po MOCOCC
50
50.50
T6.04
75.55
20:21
Stn chund Marning
Smuctua leamning
Senctain Tearning
Figure 25: Test and train accuracy of modular NNs compared to learning rates for structural and functional parameters. The various columns show the results for different truth table ratios for training.
Figures 25 and 26 show the training and test accuracy for various learning rates across different training sizes for the modular and modular-shared NNs, respectively. We consistently find that the best-performing combination of lcarning rates is 0.1 for structural parameters and 0.01 for functional paramcters across both architectures. This suggests that learning inter-module connectivity and module selection requires a more aggressive optimization strategy compared to learning sub-functions within the modules. Structural parameters seem to benefit from a higher learning rate, which may be due to the need for broader exploration during training.
Moreover, larger learning rates generally improve performance on tasks involving Boolean functions. Thus, for both modular and modular-shared NNs, we focus on learning rate combinations of 0.013. This approach reduces the complexity of the hyperparameter search space while still achieving high performance across different training data sizes.
26

---

Published in Transactions on Machine Learning Research (08/2025)
yalt Train datanet racec
Tran dataset latel
Iran dibaeet oe di
Test accuroo leamiing gits
F6.73
50.75
$1.45
13.06
00.23
23.44
71.20
T2.11
10.50
73.67
00
M6.35
18.52
12.92
11
62.22
6%
eccuracy ye leaminng
0.3
accurazy learning patnn
lesriig ratea
71.30
12.23
04
23:24
71.30
302,00
23.50
$2.32
703
20.31
46
TOUE2
$0.41
70.32
m1
EC:03
71.11
62.22
13.33
sariing 1996
learning
laaming
Structure leariing
Figure 26: Test and train accuracy of modular-shared NNs as compared to learning rates for structural and functional parameters. The varions columns show the results for different truth table ratios for training.
G Additional results: generalization and training efficiency
In this section, we present additional visualizations that provide alternative perspectives the generalization performance and training efficicncy of the various architecturos.
anTiEAyy Seen conccluttion Medicle
Ted cruuryy..o comtinationn Morddenf
That accuracy Seen combinationn: Wodufarddhwee-Daa
4
74.73
73.43
80.49
82.34
91.53
80.00
83.93
84.89
94.06
83.97
84.08
90.25
86.34
87.91
95.87
60.66
91.13
94.23
94.40
96.11
97.15
65.83
90.00
95.69
86.93
96.82
96.60
91.57
93.63
94.82
94.37
96.01
97.09
6.6
64
Rato comtendeer
combinations of different NNs trained on the hierarchically modular MNIST task, evaluated across varying proportions of digit combinations used for training and different numberr of samples per combination. Plots show results for: a. modular NN, b. modular-PG NN, c. modular- shoreed NN, and modular-shie shared swwred-FCaas NN.
Figure 27: Generalization performance
Training El BPx Cog coae) Vad.ua
Twirks T1OR (log undel Nockleff
Thiiiig OP (bo kralet Hociden
Triising FLORs scalet: HodvanndhnnndFCCW
11.19
11.63
11.85
11.04
11.53
11.93
11.36
11.79
11.79
11.42
11.83
11.99
12.32
12.39
12.66
12.63
12.74
12.81
13.01
12.23
12.83
12.89
12.68
12.85
12.92
12.75
12.85
12.94
Fazs of wwt.ccrbbraee.......o........... cowbratttooooo..moo Sna
traiiing traiing Dera
natio of tadooe mainting data
AM50
wtt.://.torrrraaooooo.o sraiiing Dica
Kalu- contiaations
Figure 28: Training efficiency of different. NNs trained on the hierarchically modular MNIST task, evaluated across varying proportions of digit combinations used for training and different numbers of samples per combination. Plots show results for: a. modular NN, b. modular-FC NN, e. modular-shared NN, and d. modular-shonl wwwoed--FCCM. NN.
27

---

Published in Transactions on Machine Learning Research (08/2025)
Test Accuracy
Monolithic Nhs. Density 100.0%
Test Accuracy Modular NNs
Test Accuracy: Moduler shared Nife
Herochicl tree treptt
$1.5
74.1
97.3
98.9
97.6
90.1
90.9
Meachaloos
73.9
91.7
Bb..5
98.9
100.0
100.0
100.0
Hanncheal cteadt
04-6
97.0
100.0
100.0
100.0
100.0
100L0
He
69.2
72.2
67.9
86.9
92.7
90.2
96.7
71.8
68.2
23:1
87.5
89.0
94.4
03.9
73.0
78.8
87.9
96.9
100.0
100.0
100L0
$1.5
54.5
65.2
10.0
74.0
74.0
71.7
60.1
67.11
25-1
12.2
72.8
27.4
00.0
89
69.7
79.2
76.7
80.6
85.5
B5.6
0.2
6.1
04
67
2.5
Rutio of bruhh in baining dal.
Putio trttt triining dute
futio of bubt in buining date
Toiin Accurwey Manolitii MNs, Deniity 100.0%
Train Arcuracy: Mociuler NNs
TFwin Accunwcy: Mudhlarrsheedd hitss
Heronbhea depits
93.5
100.0
100.0
100.0
100.0
97.7
87.4
100.0
Henoncied dapits
100.0
100.0
100.0
100.0
100.0
99.8
100.0
88.3
89.5
100.0
100.0
2.5
2.5
Ratio
Ratio
training
Operations
Monoithis NN,, Derwiy
Operatioo
Modular
Opunations Muslalershheed
Herueten depiis
6.8
6,4
6.1
6.1
6.1
6.2
6.9
6.7
6.6
Henur Soal biepth
6.2
6.4
6.7
6.6
2
7.0
8.3
8.5
8.5
7.8
Horrcciaa
6.5
6.9
7.4
F5
7.0
7.0
3.6
6:1
7.4
1.3
1:2
7.1
7.1
7.9
8.6
9.5
8.6
8.9
8.6
9.0
7.0
7.1
7.5
7.5
7.6
7.7
7.8
8.8
7.5
7.7
7.7
7.8
7.9
7.9
64
2.5
0.2
0-4
0.6
0.7
6/
64
2.5
Ratio truch table training data
training data
Retio fruth tab training data
Figure 29: Test accuracy, train accuracy and FLOPs count of NNs compared to the complexity hierarchical depth of the Boolean function graphs and the ratio of truth table used for training. First column indicates the results for dense monolithic NNs, second column for modular NNs and third column for modular-shared NNs. We can clearly see Irend where the lop right of the heatmap has better values indicating that larger training size and lower complexity functions are an easier combination to learn.
28


Reviews:
[{"review_id": "gyuBWaI3Iz", "reviewer": "Reviewer_wzuG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is very well written; the experiments are clearly described and easy to follow.\n- The supporting figures effectively illustrate the experimental setup and results.\n- The claims are mostly well-calibrated and supported by the experimental evidence.\n- The training and evaluation framework is well designed and clearly explained.\n\n**Weaknesses**\n\n- As a fundamental study, the main limitation lies in the simplicity of both the tasks (synthetic Boolean compositions) and the model architectures (basic MLPs). This makes it challenging to assess how well the findings would generalize to more complex or realistic settings, such as modern deep learning architectures and real-world tasks. While the MNIST task helps extend the findings beyond synthetic data, it is still a relatively simple and artificial scenario.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates how incorporating knowledge of a task’s modular structure and enabling parameter sharing can improve generalization and training efficiency. The main experiments use multi-layer perceptrons (MLPs) with dense, sparse, or modular architectures, trained on tasks composed of layered Boolean functions (AND, OR, XOR), with difficulty increasing with depth.\n\nThe authors show that leveraging the task’s modular structure improves generalization and reduces the data and computation required for training. Allowing module sharing further enhances these benefits.\n\nAdditional analyses show that module sharing helps networks better capture the underlying task structure, and that shared experts specialize to represent specific Boolean functions more effectively.\n\nThese findings are further validated on a visual modular task based on MNIST.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical changes needed to secure my recommendation for acceptance**\n\nWhile the reliance on simple, artificial tasks and MLPs is understandable given the paper’s goals and fundamental nature, and I believe the community may still find the work valuable, I think some improvements to the main experimental setup are necessary for the paper to be suitable for acceptance.\n\nCurrently, the study uses only three function graphs, shown in Figure 2, all of which follow a fixed, manually imposed structure: the first layer consists solely of XOR functions, the second of AND functions, and the third of OR functions. This predefined structure raises concerns about the generality of the results, even within the chosen artificial setting.\n\nTo address this, I strongly recommend modifying the experiments to use randomly constructed graphs, where each node (as in Figure 2) is assigned a randomly selected Boolean function (AND, OR, XOR), rather than fixing one function type per layer. These experiments should be repeated with different random seeds for each number of layers to ensure robustness, and the results should be included in the main paper.\n\nThis change will likely increase task difficulty, which is desirable in my opinion. Currently, the best-performing models (e.g., Modular-shared-FCMS) reach near-perfect accuracy with relatively little training data. Identifying the point at which these models begin to fail would provide valuable insight into the scalability and limitations of the proposed approach.\n\nIf the models continue to perform well even under random function assignment, task difficulty could be further increased by adding more layers.\n\n- As a separate point, I believe a dedicated Related Work section is necessary to properly situate the paper within existing literature. In particular, this work appears closely related to (Malakarjun Patil et al., 2024). The authors should explicitly acknowledge this connection and clearly articulate the differences and novel contributions of their own work.\n\n**Changes that would help strengthen the work**\n\nWhile not critical for acceptance, I believe the following revisions would significantly improve clarity and completeness:\n\n- Briefly introduce the concept of a truth table at the beginning of Section 3, as it is central to the evaluation framework. Not all readers may be familiar with it, and it would serve as a helpful refresher even for those who are.\n- In Figure 4 (bottom), either separate the sparsity and modularity experiments or clearly distinguish them. At the very least, include the monolithic 100% baseline in the bottom plot, as it is an important point of comparison.", "broader_impact_concerns": "This paper investigates how incorporating knowledge of a task’s modular structure can improve generalization and training efficiency. The work is fundamental in nature and aims to advance our understanding of the inner workings of artificial neural networks. As such, there are no specific ethical concerns that would necessitate a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aVXAZ5MBOs", "reviewer": "Reviewer_QVTi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper is well-written and easy to follow\n- The authors conduct a comprehensive set of empirical experiments to investigate the premise\n\nWeaknesses: \n- The core idea of NN architectures designed to exploit a task’s structure working better than dense monolithic NNs is already a well-established idea. For example, CNNs are designed to make use of the structure in image data which is why they perform significantly better than dense NNs for image related tasks with fewer data samples. Similarly, LSTM style models perform very well on time sequencing tasks as they make use of the structure in time data. So the idea of networks designed for hierarchical tasks performing well on hierarchical tasks over monolithic networks is more or less expected. So I would like the authors to add more discussion on how their paper is adding further value to this core premise that we are already aware of.\n- The authors mention that for more complex depth-3 functions, the performance gap starts narrowing. It would be nice to see a more detailed investigation on why this is the case. Is it purely due to lack of data? If so, then what happens if we provide ample data to all the architectures; is the performance gap still the same? Or are there any other factors involved?\n- For the MNIST style task, I would like to see a comparison between the hierarchical MLP based network designed by the authors vs a CNN style network which first processes the images using convolutions, followed by linear layers to predict the Boolean functions.\n- Expansion of the idea to larger datasets: The tasks used in this paper are very simple and the same idea may not extend to more complex tasks (as evidenced by the depth-3 functions). The authors should add an discussion on how this idea can be extended to larger datasets.\n- Similar to point 1, in the conclusion part, I would like to see more discussion on how the results of this paper add further value to the core premise that we are already well aware of.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates how incorporating structural knowledge into neural network architectures enhances their generalization performance and efficiency. The authors compare the following architectures with varying degrees of structural knowledge \n\n- Monolithic dense NNs (no knowledge)\n- Monolithic sparse NNs (sparsity assumption)\n- Hierarchically modular NNs (hierarchical pattern assumption)\n- Hierarchically modular NNs with  shared modules (hierarchical pattern, module reusability assumption)\n\nThe authors then perform extensive empirical experiments on Boolean function tasks and a visual recognition task using MNIST dataset. \n\nThe main conclusion of the paper is that incorporating modularity and module reusability gives significant gains compared to monolithic NNs especially in data scarce scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend the authors to address at least some of the concerns mentioned in the weaknesses section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o0wOIlCKbZ", "reviewer": "Reviewer_gvrp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The work performs an exhaustive comparison of the models, given the considered task. The Authors compare five types of the models (fully-connected and modular with/without fixed connectivity with/without shared modules) through the lens of at least four factors (accuracy, compute requirements (FLOPs), similarity to ground truth (graph edit distance), and generalization) where they rank the models' results.\n\n- The text is well-written and easy to follow. It is well-structured, first introducing the models and then, one by one, investigating their properties. The figures are clear and instrumental in conveying the message.\n\nWeaknesses:\n\n- [Broader impact concern] The experiments are constrained to a limited domain that may not generalize beyond the examined scope. In ground-truth models, the \"blocks\" to be discovered by the models are represented by logical gates - the smallest computational units possible. The depth of their hierarchy does not exceed three layers. While the results reported here appear correct, the community's interest in the modularity is driven by possible speedups in the training of large models or in the better accuracy on data-limited real-world datasets. While the Authors touch on the second aspect of these two, the utility of the proposed model does not automatically transfer to either of these cases and thus has to be additionally shown, should the Authors want to expand the scope of their claims. However, this is not the requirement for TMLR submissions.\n\n- [Requested change] The clarity of the text can be improved in a few places. Specifically, provided that the ground-truth networks are comprised of logical gates, it would be important to highlight, early in the text, that the neural networks that are trained here to learn it are actually the MLPs. It is also important to further outline the scope of the claims starting from the abstract and then throughout the text (i.e. that the results apply to few-layer models with logical gates as blocks).\n\n- The FLOPs claim is not particularly relevant as, once the models are run on GPUs (and everything of size where the FLOPs argument may matter in run on GPUs), they will still multiple the full-size matrices, equating the computation between the considered models. I suggest adding the related discussion.\n\n- In the text, please explain the drops in the training accuracy in Figure 3. Surely, it's not overfitting as suggested in the text, because overfitting would result in a higher accuracy in training splits.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the Authors examine how the modularity in neural networks affect their accuracy, convergence, generalization, and compute requirements. To this end, they perform a series of experiments where they train neural networks featuring varied degrees of modularity to reproduce synthetic functions comprised of several layers of logical gates. On these tasks, the Authors present an exhaustive comparison of modular MLP architectures with/without shared modules and with/without fixed inter-module connectivity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the [Requested change] section above. Conditioned on that change, in light of the correctness of the work, which is the main criterion for the TMLR, I recommend the acceptance of this paper.", "broader_impact_concerns": "Please see the [Broader impact concern] section above.\nWhile the results may not generalize and thus may not be super useful, they are technically correct and thus fit the TMLR criteria.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)