Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning
Authors: Anna Soligo
Pietro Ferraro
David Boyle
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
RL
Interpretability
Modularity
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We show how functionally relevant modules can be induced in RL policy networks, and develop methods to detect them and empirically interpret their functions.

Abstract: Interpretability is crucial for ensuring RL systems align with human values. However, it remains challenging to achieve in complex decision making domains. Existing methods frequently attempt interpretability at the level of fundamental model units, such as neurons or decision nodes: an approach which scales poorly to large models. Here, we instead propose an approach to interpretability at the level of functional modularity. We show how encouraging sparsity and locality in network weights leads to the emergence of functional modules in RL policy networks. To detect these modules, we develop an extended Louvain algorithm which uses a novel `correlation alignment' metric to overcome the limitations of standard network analysis techniques when applied to neural network architectures. Applying these methods to 2D and 3D MiniGrid environments reveals the consistent emergence of distinct navigational modules for different axes, and we further demonstrate how these functions can be validated through direct interventions on network weights prior to inference.

Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning
Anna Soligo
Pietro Ferraro' David Boyle
Abstract
However, this breadth of impacts also raises wide-ranging concerns related to topics of safety, reliability and bias, among others. It is thus crucial that the behaviour of RL agents can be properly characterised to the extent that can be reasonably verified that their impacts align with human values. As reflected in the EU's AI ethics guidelines: sys- tems should allow for human oversight, accountability and transparency (European Commission & High-Level Expert Group AI, 2019).
Interpretability crucial for ensuring RL systems align with human values. However, it remains challenging to achieve in complex decision mak- ing domains. Existing methods frequently attempt interpretabilityy at the level of fundamental model units, such as neurons or decision nodes: an ap- proach which scales poorly to large models. Here, we instead propose an approach to interpretability at the level of functional modularity. We show how encouraging sparsity and locality in network weights leads to the emergence of functional mod- ules in RL policy networks. To detect these mod- ules, we develop an extended Louvain algorithm which uses a novel 'correlation alignment" metric to overcome the limitations of standard network analysis techniques when applied to neural net- work architectures. Applying these methods to 2D and 3D MiniGrid environments reveals the consis- tent emergence of distinct navigational modules for different axes, and we further demonstrate how these functions can be validated through direct in- terventions on network weights prior to inference.
Currently, there remain fundamental challenges to achieving this, and RL systems rarely afford sufficient interpretability. One factor is the ambiguity regarding what constitutes a suitable 'explanation" of a model. Lipton (2016) consid- ers two parallel concepts: 'simulatability", the ease with which human can predict a model's output from its in- put and explanation, and 'decomposability" the extent to which constituent components of a model are themselves interpretable. Doshi- Doshi-Velez Kim (2017) reiterate this with the concept of 'cognitive chunks', emphasising the need for model explanations to be tractable to human interpreters.
More concretely, interpretability can be considered in terms of the affordances it provides. In some cases, interpretability enables formal verification of safety-relevant capabilities (Bastani et al., 2018). However, in complex, incompletely- defined scenarios, it can instead offer insights which enable downstream safety-relevant tasks, such as system auditing or direct interventions to reduce undesirable behaviours (Kohler al., 2024; Delfosse et al., 2024).
1. Introduction
Reinforcement learning (RL) has emerged as a powerful ap- proach to improve performance in complex decision-makinn domains. Learning policies directly from interactions can offer improved flexibility and performance, whilst avoid- ing challenges faced by classical model based control ap- proaches (Song et al., 2023). The growing body of RL research is demonstrating its potential to positively impact diverse real-world domains, from battery manufacturing (Lu et al., 2020) to the design of medical treatment regimes (Coronato al., 2020), applications which directly impact critical issues such as climate-change and human-health.
When scaling these affordances to complex domains, in- terpretability at the level of neurons, or other fundamental model units, becomes problematic: both due to their sheer quantity and because individual neurons are rarely seman- tically meaningful in isolation (Elhage al., 2022). We address this challenge by taking a modular approach to in- terpretability. Modularity is fundamental in diverse biologi- cal architectures, including physical structures of the brain (Gazzaniga et al., 2018). Similarly, can be considered as decomposable into modular processes (Eppe et al., 2022), suggesting that modularity may offer a natural framework for enhancing human understanding of complex systems.
making
'Imperial College London. Correspondence to: Anna Soligo Sanna. @iciggo@@@iii amperialac..k>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Inducing, Detecting and Charactcrising Neural Modules for Functional Interpretability in RL
Motivated by this, we demonstrate how training modifica- tions can encourage the emergence of functional modules within RL policy networks. We further propose methods to detect these modules and characterise their behaviour. In doing so, we aim to establish a suitable level of abstraction for aligning model interpretations with our internal decision making frameworks.
of nodes with denser intra-community connections than inter-community ones. Detection of these communities is NP-hard problem (Fortunato, 2010), and a multitude of methods have been developed to avoid the brute force approach, including hierarchical clustering, non-negative matrix factorisation (Lee & Seung, 2000), and the Lou- vain algorithm (Blondel et al., 2008). For the purpose of interpretability, we are further interested in functional mod- ularity: the presence of components which show level of independence and specialisation in their function (Fodor, 1985; Stembeerg 2011).
1.1. Contributions
Considering interpretability at the level of functional mod- ules, our work makes the following contributions
Functional modularity in the brain arises alongside its 'small-world' architecture: a combination of high clustering and short path length hypothesised to have evolved partially to satisfy spatial and energy constraints (Margalit al.. 2024). Recent works have investigated applying analogous constraints to neural networks. Liu et al. (2023); Achterberg et al. (2023); Margalit et al. (2024) demonstrate that penal- ising parameter 'connection length' in neural networks can lead to clustering and improved interpretability of network visualisations. We primarily build on the brain-inspired modular training approach proposed by Liu et al. (2023). We extend the concept of distance weighted regularisation to the RL context and further propose methods to extract and characterise functionally relevant modules from these regularised networks, enabling scalable interpretability in a decision making context.
We extend recent algorithms for encouraging locality in neural networks (Margalit et al., 2024; Liu et al., 2023; Achterberg et al., 2023) to an RI. context, demonstrat- ing that penalizing non-local weights facilitates the emergence of functional modules within policy net- works (Section 4.2). These modules offer scalable unit for decomposing decision making, moving beyond interpretability at the level of neurons.
We propose an extended Louvain algorithm for commu- nity detection which addresses the limitations of con- ventional community detection methods when applied to neural networks (Section 3.5). We thus demonstrate the ability to automatically identify functionally cohe- sive neural modules (Section 4.3) in a manner which enables the scaling of module based interpretability to complex networks.
3. Methods
Utilising this approach module detection, demon- strate how targeted modifications of network param- eters prior to inference can be used to characterise module behaviour, offering an empirical understanding of their functionality (Section 4.6).
3.1. Spatially Aware Regularisation
Regularisation approaches encourage sparsity by penalising the magnitude of model parameters. Following (Liu et al., 2023; Achterberg et al., 2023), we extend this to encourage local connectivity by projecting the neural network into Eu- clidian space and scaling weight penalties by the 'distance' between the neurons they connect.
2. Background
Following (Glanois et al., 2024), and to avoid confusion aris- ing from the inconsistent use of terms in the literature, we denote interpretability as the extent to which model's 's inner workings can be examined and understood. We distinguish this from explainability, which we define as an external understanding of model behaviour generally arising from post hoc attempts explaining input-output relations. In- terpretability and explainability present two approaches to obtaining information which can be used to form explana- Specifically, this work takes
For network with L weight layers, we denote neuron layers N, for I E o,... L,, and weight matrices W, for E 0, O.... 1. Each Wi E Rm connects adjacent neuron layers, where w/y links the 8th neuron in N to the jth neuron in N Each neuron, n/. is assigned a 2D coordinate. To preserve their sequential nature, neurons within each layer N share a fixed y-coordinate yf := I. Initial x-coordinates are uniformly spaced as of m..
tions for direct interpretability approach, learning an inherently more interpretable model architecture
Standard L1 regularisation, ΣN |weele promotes sparsity by penalising the sum of absolute weight values. However, this scales linearly with weight magnitude, such that two weights of size x incur the same penalty as a single weight of 2r. We thus introduce a logarithmic sparsity loss, Σ," log[wss|+ which provides greater sparsity incentive by incurring a penalty which scales with (x + 1)* rather than kz. We
Structural modularity, a property well-studied in network analysis, is characterised by the presence of communities
All code is available at: http://gghhhh.c.ommmmmsslggggg/
2

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
provide further explanation and analysis in Appendix A. Scaling sparsity by distance gives the 'connection cost' loss:
where the community of node n' is denoted C(n') E P. modularity Q is defined as:
m--1
1 2m
kiki 2m
Lice Ace
Σ
log((d;) - d.)) d.)lw((+1)
(I)
Q
Aig
(2)
DJEP
where
where m = Ens w/ is the sum of all edge weights, A is the network adjacency matrix, kū is the weighted degree of node i and the binary 8(c,, Cÿ) is a binary variable which equals 1 if nodes i share a community. The null model ki models random connectivity given node orders and acts 2m
d, V (oy - 2,,2 + (yi + yz)²
Ace is the regularisation scaling factor, and dg adjusts the relative impacts of weight 'length' and magnitude.
as the non-modular baseline. This equation forms the basis of the heuristic Louvain algorithm (Blondel et al., 2008), which optimises P to maximise Q through hierarchical local node reassignments.
3.2. Neuron Relocation
To further minimise Teees neurons are periodically relocated during training, following (Liu et al., 2023). Within each layer, neurons are ranked by their weighted degree w(n)= = Σ [wmm|+ +E Woael The top k neurons are optimised within their layer by exchanging their position with the alternative neuron position which leads to the greatest reduction in Lee, as detailed in Algorithm 1. This has the effect of changing the relative 'cost' of weights, such that weights with greater impact on performance can be retained with a lower relative connection cost. We discuss this further in Appendix E.2.
We take the Louvain algorithm as a baseline partitioning ap- proach due to its efficiency (Inlog(z))), automatic detec- tion of community number, and relative simplicity. However application to neural networks reveals limitations arising from the differences between NNs and traditional networks. Primarily, the high fan-out connectivity of input features and constrained layer- wise connectivity in MLPs results in Louvain partitions which fail to span sufficient weight layers and violate the directionality of NN information processing. We further discuss and provide of this in Appendix B.
Algorithm 1 Neuron Position Optimization
1: Every Tooop training steps:
3.4. Modularity in Feed-forward Neural Networks
2: for each layer in [1, Lj do
To address these challenges, we propose two metrics to guan- tify neural network modularity while accounting for their architecture and the specific utility of modularity for inter- pretability. Firstly, we consider module isolation. High iso- lation implies minimal inter-module connectivity, resulting in stricter decomposability and enabling more independent module analysis. For a single module, we define isolation I(C) as:
3: Calculate weighted degrees: w(n) = Σ |wea| + Σ Wootl
for all E Ni
4: Select top & neurons with highest wr(n)
5: for each candidate neuron nf do
6: Compute baseline cost Lee using Equation!
7: Ibest + Lec. 12 best + None
8:
for each neuron ni in layer do
9:
Calculate The after swapping positions of ni and mif
10:
if L'ce Lbest then
W in
ICC)
(3)
11:
T.best +- Tiues
bratt ni Ti
W + W
12:
end if end for
where Wint = Eijec |w Wen = Eiec | + Eiecsec |wm represents the sum of intra- and inter- community weights respectively. We extend this to the isolation of a network partition P:
13:
14:
if bestt is not None then
15:
Swap positions of nf and mi
best
16:
end if
17:
end for
18:
end for
0 | ICE
i
I(P)
(4)
otherwise
3.3. Structural Modularity in Networks
In network analysis, structural modularity is quantified by comparing the strength of intra-community links with their expected strength in a random 'null model', such that high modularity indicates stronger connectivity within a set of defined modules than would occur by chance (Clauset et al., 2004). Given the network partition (C1, .z,..., C
Secondly, we consider the alignment between structural and functional modularity by considering correlations between neuron activations. Neuronal correlations have been used to study functional architectures of biological neural networks (Cohen & Kohn, 2011), as well as similarities between artificial neurons (Li et al., 2016). We calculate the Pearson
3

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
correlation coefficients rd between each pair of neurons i, based their activations n nt) and n² over T samples.
Algorithm 2 Interpretability Fine-tuning of MLP Modules
1: Initialize corrretttt and Pr using the Louvain algorithm each of S.current and Pr, assign each input neuron ná the community which it most strongly connected. Calculate initial modularity score M [(Ps,euvrent) +
EL (n' (t) n (')(n' (t) r2)
(5)
1-= (1) ( -2))==// Et (nt n2)a
(R.(Ps.. PE)
4: repeat
for each pair (P,, P2) of adjacent modules do
These correlation values form the adjacency matrix of functional network graph, Gr, in which, unlike the weight- based structural network graph, Gs. connections are not constrained to adjacent layers. Given two Louvain parti- tions, PF = {F1, F2, .... F and P3 = {Si, S2. ..., S, of these graphs, the Adjusted Rand Index (ARI) quantifies their similarity and thus the "correlation alignment' of Ps:
Calculate M,₃ for merged modules
7: end for
8:
if Quax(Ma) > M then
Update Psceureent with best merge
10:
M max(Ma)
11: end
12:
until max(Ma)) < M
2 2 (72 ARI Ps)
(x ( ( E, 4 2i (*:²)
(6)
where nej is the number of nodes shared between modules i of Pr and j of Ps, and and sj the total numbers of nodes in modules and j respectively.
Figure 1: The Dynamic Obstacles (DO) and Go to Key (G2K)
Environments.
3.5. Detecting Modules in Neural Networks
interpretable and functionally relevant components?
Utilising these isolation and correlation aligmment metries, we propose 'fine-tuning' stage which improves the ini- tial Louvain partition Ps by iteratively merging modules to maximise the structural and functional modularity of the resulting partition. As detailed in Algorithm 2, functional and structural partitions, Pp and Ps, are initialised using an 'internal' variation of the Louvain algorithm. This cludes input layer nodes in the initial partitioning, then sub- sequently assigns them to the community which they are most strongly connected, mitigating the challenges the input layer poses to module detection. Ps is evaluated according to its modularity score M = T(Ps) + ART(Ps Ps, Pr), and adjacent modules are merged in the manner that maximises M, until no further improvement is obtained.
4.1. Experimental Setup
Experiments are conducted in three Minigrid environments (Chevalicr- Boiiseeier-Boivvert et al., 2023; Pignatelli et al., 2024), shown in Figure 1: Go-to-key (G2K), where an agent must navigate to of two keys in 4x4 grid; dynamic obsta- cles (DO), where agent must reach goal in a 4x4 grid whilst avoiding three moving obstacles; and 3D dynamic obstacles (3D-DO), which extends dynamic obstacles to a 3x3x2 grid. These are encoded into a symbolic observation of entity coordinates relative to the agent. The action space consists of left, right, up, down, and, in the 3D case, for- ward, backward steps. Following Pignatelli et al. (2024), Markov reward function offers sparse reward of when the target is reached and 0 otherwise. The reported returns thus represent both the mean episode return and the success rate. Episodes terminate on goal completion, collision with obstacles, reaching the incorrect key, or exceeding the 100 step limit.
4. Experiments
We evaluate the proposed methods with respect to 4 main research questions. These examine the ability to induce (RQ1), detect (RQ2) and interpret (RQ4) modularity, while considering auxiliary impact on model performance (RQ3).
RQL. Does spatially aware regularization lead the emergence of modular structures in RL policy networks? RQ2. Can emergent modular structures be detected using the proposed extended Louvain algorithm?
We train all policies using Proximal Policy Optimisation (Schulman et al., 2017) due to its stability and simplicity. The actor and critic networks are implemented as MLPs with two hidden layers of 32 neurons and hyperparameters (Table 1) optimised via grid search. We focus on decision making interpretability. and apply distance weighted regular- isation the actor network. The regularisation coefficient
RQ3. What is the quantitative relationship between RL
policy modularity and performance?
RQ4. Do detected network modules correspond to

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL.
Acc
0.00
0.01
0.02
0.1
0.14
DO
Returns
0.96
0.90
88
0.76
0.75
30-00
Returns
0.96
0.82
0.77
0.73
0.72
G2K
Returns
1.00
0.99
0.97
0.95
0.92
Figure 2: Distance weighted regularisation induces the emergence visual modularity. As Aee is increased, increasingly isolated modular structures are observed in the policy networks of the DO (top). 3DO (middle) and G2K (bottom) environments. A moderate decrease in mean return is also observed, as annotated below each network plot.
Apc is increased linearly from 0 to its target value between 20 and 30% of training steps. Agents are trained for 4M environment frames, pruned by removing weights and neu- rons with magnitudes below 1 of the maximum values and orders in their respective layers, then fine-tuned without Lee regularization for 2M frames. This regularization sched- ule and two-stage training methodology yields improved returns and modularity metrics, detailed in Appendix C. The PPO agent and environment are implemented in JAX (Bradbury et al., 2018) and trained using a NVIDIA RTX 4090 GPU.
Left Right
Up
Down
Fwd
Bwd
Down
Up
Left
Right
81X B2X
B3X
GY
B2Y
822
81Z
K1V
K2Y
10
K1X K2X
GX
B3Y
B1Y
B9Z
Figure Neuron relocation causes input features and output tions align spatially by function. the 3D-DO (left) and G2K (right) policy networks, object coordinates align spatially with the actions controlling movements along their corresponding axes.
4.2. Emergence of Visual Modularity (RQ1)
4.3. Module Detection (RQ2)
Structural modularity emerges as the strength of distance weighted regularisation is increased. As shown in Figure 2, module independence initially emerges in the second and third weight layers, while feature sharing persists in the first. Across all environments, neuron relocation causes the input features and output actions to reorder in manner that reflects their relevance. Figure 3 shows how feature x, y and z coordinates align vertically with the actions controlling movements on the x, and axes respectively. In the DO and 3D-DO environments, modules become fully indepen- dent at high Aee values, and networks increasingly prioritise goal features over obstacles. Conversely, the target key ID in the Go to Key task remains used by both navigational modules at all Aoe values, reflecting its necessity to solve the task. Figure 4 shows the importance of both the connection cost loss and neuron relocation in inducing this modular- ity and we provide further ablation results isolating their impacts Appendix
We benchmark our proposed fine-tuned internal-Louvain approach against the standard Louvain algorithm, and fur- ther evaluate the isolated impacts of the finettning and internal-Louvain modifications. Fine-tuning (FT) signifi- cantly increases the average isolation (16.8%) and correla- tion alignment (33.7%) of the detected modules across all Aee values (as detailed in Appendix G). Initialising with the internal Louvain (FT Int.) increases isolation by a further 2.4% but decreases correlation alignment by 1.6%. Notably, this disparity in ARI between the FT and FT Int. methods predominantly arises from differences in the G2K results. While the FT method gives partition ARIs that reach max- imum value at A = 0.11 before decreasing, the FT Int. method gives monotonically increasing ARI, which ex- ceeds the FT. ARI for Aoe > 0.07. Figures 5 and exemplify these improvements in performance at varying Ace values.
The G2K networks retain feature sharing between modules
5

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
Weight Regularisation
Regularisation
Distance Weighted Regularisation
Distance Weighted Reg. with Relocation
Figure 4: The combination of distance weighted regularisation and neuron relocation results in the most modular networks. Ablating the distance weighting (left and second left) the relocation (left and second right) does not achieve the same level of modularity. We provide further quantitative ablation results Appendix E.2.
in highly regularised networks. This, as detailed in Section 3.3 and Appendix B, presents failure case for the standard Louvain algorithm, whereby it fails to distinguish modules within the first layer, as can be seen in Figure 6. Since the correlation partitions are less clearly segregated in networks with higher connectivity, and occasionally exhibit this input layer failure, we deem ARI to be a less reliable indicator of modularity lower Ace values, and therefore adopt the fine-tuned internal method.
17%, which, as detailed in Appendix D.1, is largely due to the regularisation component.
Despite this performance trade-off, note that our regu- larisation approach offers an auxiliary benefit by yielding much smaller networks. At the 'modularity emergence' stage, our final DO, 3D-DO and G2K networks have 90.5%, 96.5% and 89% fewer parameters, respectively, than those trained without regularisation. This offers a means of signif- icantly reducing computational overhead during inference in addition to further potential interpretability benefits.
Louvain (Weights)
Fine-tuned Internal Louvain
4.5. Functional Interpretability (RQ4)
While visualisation of network modules offers level of insight into the structure of decision making, it relies on subjective assessment and lacks scalability. Consequently, we use targeted modification of parameters as an empirical means of interpreting module functionality.
Do, Acc 0.02
For the example networks shown in Figure 9, we system- atically modify module parameters in two ways: negative saturation, in which we replace all values with large nega- tive value of -50; and negation, where we reverse the sign on all parameters. The former aims to effectively disable a module, while the latter aims to perturb it. We evaluate the subsequent behavioural changes over 10,000 episodes, measuring the frequency actions and their corresponding outcomes: success, failure, or continuation of the episode.
3D-DO, Acc 0.02
Figure 5: The modified Louvain algorithm is able identify func- tional modules across multiple layers. Our fine-tuned internal Louvain method (right) successfully detects cohesive neural mod- ules, whereas the standard Louvain (left) incorrectly subdivides modules in DO (top) and 3D-DO (bottom) policy networks due to limitations handling MLP architectures.
Foremost, find that negatively saturating any individual module strongly inhibits actions along specific axis. This validates that the detected structural modules correspond to axis specific navigation. Intervening on community 0 in Figure 9a, for example, reduces the frequency of for- ward/backward actions by 83%, while intervening on com- munity 0 in Figure 9b reduces the frequency of up/down actions by 95%. These results replicate, with slightly re- duced functional independence, in less regularised networks like Figure 9c, where community O intervention results in 91 % decrease in up/down actions, while community I intervention results a lesser 42% decrease in left/right. Notably, while the overall success rate decreases when we saturate modules in the dynamic obstacles environments,
4.4. Quantification of Modularity (RQ1, RQ3)
Applying the FT. Int. partitioning method, we find module isolation and correlation alignment increase with Apes as shown in Figure 7. The induced sparsity, does, however, impact negatively on return. As shown in Figure this impact varies significantly by environment. At the point of module emergence, returns decrease by average of 11.4% and 12.5% for DO and 3D-DO respectively (Aco s 0.02, 0.04), compared to just 0.8% for G2K (Age s 0.02). We also find that implementing regularisation and neuron relocation results an average training time increase of
find
6

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
Louvain (Weights)
Louvain (Correlations)
Internal Louvain
Fine-tuned Louvain
Fine-tuned Internal Louvain
Figure 6: The internal Louvain and fine-tunine tunintunng stage result modules which more isolated and better aligned with the activation based network partition. Utilising the internal Louvain improves module assignment in the input layer, particular the lesser-reuularieed example (top). while the fine-tuning stage reduces the subdivision of modules between layers.
Relationship between and Isolation
Relationship between ècc Adjusted Rand Index
Adjusted Rand Index
0.9
0.8
Isolation
0.8
0.7
DO
0.2
G2K
G2K
0.4
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.02
0.04
0.06
0.08
0.2
0.12
0.14
Acc
Figure 7: Increasing regularisation strength results in increased module isolation and correlation alignment (ART). The and standard deviation (n 10) isolation (left) and ARI (right) increase with Aee.
Relationship between Acc and Return
U
R
0.95
Retum
0.85
02 812 BTY B3V 822 (sz. D1X GY BZY Gx
0.6
100
0 Farward/Baccwward Left/Right
U-Up/Dawn Lofft/Right
U-Up/Down Left/Right
2 Up/DDwn
G2K
(a)
(b)
(c)
0.02
3.05
0.12
Acc
Figure 9: The detected modules specialise in navigation along specific axes. Examples of partitioned policy networks for (a) 3DDO (Aee = 0.06), (b) G2K (ee 0.12), (b) G2K (Aoc 0.02), with the identified module functions described in the legends.
Figure 8: The performance trade-offss observed with increased regularisation strengths vary environments. The mean and standard deviation return (n=10) shows varying levels perfor- mance degradation as Aee is increased the three environments.
the proportion of actions resulting in failures does not in- crease. This shows that with the achieved level of functional independence, we can disable a module while retaining the crission-making ability of those remaining.
Compared to negative saturation, negation has significantly stronger impact on return: rather than minimising actions along a given axes, the agent now acts incorrectly. For the 3D-DO network presented, we observe an average decrease

---

Inducing, Detecting and Charactcrising Neural Modules for Functional Interpretability in RL
in return of 73% when module is negated compared decrease of only 36% when module is negatively saturated. This also reflects in the failure rate: when community 0 in Figure 9b is negatively saturated, we observe that the ratio of success to failure outcomes of up/down actions declines from 37:1 to 9:1. When it is instead negated, this drops to 1:103. Full intervention statistics are given in Appendix H.
identification of features (Templeton al., 2024), concept representations (Zou et al., 2023) or computational 'circuits' (Wang et al., 2022). We share the behavioural focus of circuits work, but rather than attempting interpretability at the level of computations, we aim to characterise the roles of higher-level communities of neurons. Notably, recent work has partially automated the circuit-discovery process in transformers (Conmy et al., 2023). This, like our automated module detection, is motivated by the need to improve the scalability of interpretability techniques.
4.6. Learning Robust Pong Polices
We additionally train a Pong policy using the same PPO training protocol as the MiniGrid experiments. Due to the simplicity of navigation in Pong, this learns a single sparse module rather than multiple modules as we observe in the Dynamic Obstacle and Go to Key tasks. However, we find that that distance weighted regularisation improves visual interpretability of the network, and enables identification of a flaw in the learnt policy. We find that the sparse Pong policy network retains strong connectivity to the opponents position, which is a consequence of the opponent's s 'fol- low ball' policy and was previously observed by (Delfosse et al., 2024). This reliance means the agent is not robust to changes in opponent policy. We consequently retrain robust Pong policies by removing the opponent position from the observation space, and note only minor decreases in performance. We fully detail these results in Appendix F.
Modularity in RL is approached by hierarchical RL, par- ticularly policy tree methods where decision making is de- composed into sub-policies (Pateria et al., 2021). These rely on predefined levels of decomposition, but can afford interpretability when discernible sub-bchaviours, such as motor primitives (Merel et al., 2018), are explicitly learnt. More intentionally, Cloud et al. (2024) recently proposed to localise network computations through selectively mask- ing parameter gradients. In contrast to our approach, this gradient- approach requires user-defined sets of pa- rameters and data points to control the functional localisa- tion process.
Prior work has explored applying biologically inspired con- nection constraints to neural networks. Achterberg et al. (2023) spatially embed an RNN and penalise connection length, demonstrating energy efficiency and clustering in a one-step inference task. Concurrently, Margalit et al. (2024) proposed to encourage local activation correlation in the training of network layers projected onto simulated corti- cal sheets. While these studies target the advancement of neuro-scientific understanding, Liu et al. (2023) aim to im- prove the interpretability of network visualisations. They demonstrate that length-relative weight penalisation reveals structure within regression and classification tasks such as learning mathematical formulae.
5. Related Work
Existing direct interpretability approaches frequently rely making fundamental architectural changes in order to build policies from intrinsically interpretable units. Examples include representing policies with differentiable 'soft deci- sion trees', (Silva et al., 2020), symbolic equations (Hein et al., 2017: Landajuela et al., 2021), or weighted combi- nations of logic rules (Jiang & Luo, 2019; Delfosse al., 2023). Recent works have extended these frameworks to remove previous barriers to their adoption in RL, for exam- ple by enabling on-policy learning of decision trees (Mar- ton et al., 2025), by combining interpretable policies with deep-neural policies to improve performance (Shindo et al., 2024), and by harnessing large language models improve downstream human interpretability (Luo et al., 2024). How- ever, these methods continue to face scalability challenges, becoming computationally prohibitive in complex scenar- ios even when indirect policy distillation frameworks are adopted (Glanois et al., 2024). Moreover, interpretability at the level of fundamental model units is rapidly compro- mised by scaling: a decision tree with an intractable number of nodes, for instance, may be no more interpretable than network with an intractable number of neurons.
Community detection in graphs a significant area of re- search with relevance to multiple disciplines, including computer science and biology (Fortunato, 2010). Although recent works adapt classical clustering approaches to spe- cialised structures such as multiplex networks (Huang et al., 2021), the challenge of detecting communities within neural networks remains largely unexplored. Filan et al. (2021), study the extent to which non-regularised MLPs can be clus- tered, and Hod et al. (2022) extend this to determine whether such clusters are more 'coherent' than random sets of neu- rons. Both rely on spectral clustering (Shi & Malik, 2000), however, which is impractical for large neural networks due to its reliance on computing eigenvectors and predefining the number of communities.
Tangentially, the field of mechanistic interpretability takes bottom up approach to reverse-engineering neural networks, particularly large language models. This can involve the
8

---

Inducing, Detecting and Charactcrising Neural Modules for Functional Interpretability in RL
6. Discussion
the RL context, motivated by the relevance of modularity in decision making, this modular approach could be applied more broadly. We also note that while we induce and detect high level of modularity in our examples, lower level of spatially aware regularisation may he useful to promote spar- sity and functional localisation in a manner which would improve the performance of post-hoe interpretability and explainability approaches.
Interpretability. Our work addresses interpretability at the level of functional modules, targeting a level of abstrac- tion that may offer a suitable balance of tractability and fidelity when scaled to large models. Given model with emergent modules, we demonstrate how modular function- ality can be systematically characterised through targeted weight interventions, enabled by our neural-network spe- cific partitioning approach. We successfully identify module functions, but emphasize that this is a preliminary demon- stration of module characterisation in simple environments. Other methods may offer improved insights, notably through activation rather than parameter modifications. This will become particularly relevant in complex architectures and applications, where we expect to achieve less module inde- pendence, and where preserving module output distributions will be necessary to preserve the downstream functionality.
Currently, the lack of formal metrics for interpretability makes it challenging to comparatively evaluate the utility of different interpretability methods. While certain metries, such as performance, can be objectively measured, the criti- cal notions of interpretation accuracy and tractability still lack rigorous means of evaluation, and the formalisation of these metrics poses an important challenge for advancing interpretable AL In their absence, we discuss performance, tractability and scope of our approach. Future work ex- ploring specific, real-world use cases could examine how verification or user-studies could be used to validate the utility of this interpretability approach.
As introduced in Section 1, interpretability can, in different contexts, enable both formal verification and safety auditing. While we primarily target the latter, due to its broader appli- cability at scale and given incomplete problem definitions, our sparse, modular approach could also advance formal ver- ification. The extraction of relatively independent modules enables verification to be performed in isolation, reducing complexity and simplifying identification of failures.
7. Conclusion
In this work, we have demonstrated how spatially aware regularisation induces the emergence of structural and func- tional modularity in the policy networks of RL agents. We develop a novel approach and metrics to quantify and detect modularity in neural networks, and leveraging this, automat- ically identify and characterise recision-making modules. By addressing interpretability at the level of functional mod- ules rather than fundamental units, offer promising balance between fidelity and human tractability. Future work should explore the broad potential applicability of this approach within different model architectures and its scalability to complex, real-world domains.
Scalability. Scalability poses a fundamental challenge to in- terpretability, and we have aimed to address this at multiple levels. By preserving standard neural network architec- tures and training, we avoid the scaling limitations faced by white-box model approaches such as decision-tree or logic- based policies. By automating the classification of neurons into modules, we remove reliance on manual approaches to module detection. The subsequent characterisation of module functions through parameter modifications further automates the interpretability process, enabling scalability to complex models embedded in a higher dimensions than the two we consider in this work. Finally, by targeting decomposability at the level of functional modules rather than fundamental units such as neurons, we aim to maintain tractability model complexity increases, thereby offering a level of simulatability that scales with network size.
Acknowledgments
The authors thank the IOTA Foundation, Google and UKRI EPSRC [grant numbers EP/Y037421// and EP/X040518/1] for supporting this research.
Impact Statement
Limitations and Future Work. The observed trade-off between interpretability and performance, while common among booitc-box' approaches, is undesirable and a barrier to the adoption of interpretable systems. The Aee scaling factor offers means of tailoring the level of interpretability based on specific requirements, but further investigation into mitigating the performance decline is warranted.
This work advances the interpretability of deep reinforce- ment learning systems, with direct implications for the safe deployment of RL in real-world applications. The proposed methods could contribute to enhancing human oversight and verification of Al systems through increasing understand- ing of decision making processes. In addition to enhancing safety, this aligns with growing regulatory requirements and could help accelerate the adoption of RL in safety-critich domains.
We offer proof-of-come of-cccep-t in three environments, but demonstration in complex domains and agent architectures remains a key direction for future work. While we focus on
9

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
References
doi: URL httpp::/www.cceecceddiie oclenee//rrticee//1///9333333//3
Achterberg, J., Akarca, D., Strouse, D. J., Duncan, J., and Astle, D. E. Spatially embedded recurrent neural net- works reveal widespread links between structural and functional neuroscience findings. Nature Machine In- telligence, 5(12):1369-1338) Dec 2023. ISSN 2522- 5839. doi: 10.1088644255-0033-077444 URL https: ///oi/ org 10w11108///00
Delfosse, Q. Shindo, H., Dhami, D., and Kersting, K. Interpretable and explainable logical policies via neu- rally guided symbolic abstraction, 2023. URL https: / wawxiiv.ogg/aba /ttaa/bs..3.../. 01439.
Delfosse, Q. Sztwiertnia, S., Rothermel, M., Stammer, W., and Kersting. K. Interpretable concept bottlenecks align reinforcement learning agents, 2024. URL https:
Bastami, O., Pu, Y., and Solar-Lezama, A. Verifiable reinforcement learning via policy extraction. CoRR, abs/1805.08328, 2018. URL http::/arxiv.cr 0bs/1805.08328,
/ //arxivv..v////////a/ wwwg//abssst.. /2401 025.2........
Doshi- Doshi-Velez, and Kim, B. Towards rigorous science of interpretable machine learning, 2017. arXiv:1702.08606
Blondel, V. D., Guillaume, J.-L., Lambiotte, R., and Lefeb- vre, E. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008 2008(10):11000S 2008. doi: 17.1548/174--5468/
Elhage. N.. Hume, T.. Olsson, C., Schiefer, N. Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superposition. Trans- former Circuits Thread, 2022. https:/ttransfommrr www.ppu//2222//ooooo////eeeeeee
2008/10/P10008. URL www.h:tp///c. dx dei iecci.o.. org/10. 1088/1742-54 6488/ 0000000////0000000000/000
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL com/ gownn//oolle//eeooeeooeooe / jax.
Eppe, M., Gumbsch, C., Kerzel, M., Nguyen, P. D. H., Butz, M. V., and Wermter, Hierarchical principles of embodied reinforcement learning: A review, 2022. URL https: /atraaaivvv//v////////eaa//aaava org. wtt.:/iww..rr//b////222.... 201.. arXiv: 2012.10147.
Chevalier-Boisvert, M., Dai, B., Towers, M. de Lazcano, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306. 13831, 2023.
European Commission and High-Level Expert Group AI. Ethics guidelines for trustworthy ai,
2019. URL https: http::/ddlgitallsttaeeg ec www.eurrr..uuu.oo wwwurppp..eeee//ibbrrry..// ethics wwtee-uuueeeiietsssttteeeeee
Clauset, A., Newman, M. E. J., and Moore, C. Finding community structure in very large networks. Physical Review E, 70(6), December 2004. ISSN 1550-2376. doi: doi. 10. 10.1103/phyxtvve.70.666111 URL http:// : doi. 066111
Filan, D., Casper, S., Hod, S., Wild, C., Critch, A., and
org/10..103/PPyyyee
Russell, S. Clusterability in neural networks, 2021. URL https: //te://aazeee www.xii.org//e.. httgg/abb/210330 03386.
Cloud, A., Goldman-Wetzler, J., Wybitul, E., Miller, J., and Turner, A. M. Gradient routing: Masking gradients to localize computation in neural networks, 2024. URL
Fodor, A. Précis of the modularity of mind. Behav- ioral and Brain Sciences, 8(1):1-5, 1985, doi: 10.1017/ $0140525X000121XX
https: wrw.aaavvooooo orgg abs / 2410. 04332.
Cohen, M. R. and Kohn, A. Measuring and interpreting neuronal correlations. Nature neuroscience, 14(7):811- 819, 2011.
Fortunato, S. Community detection in graphs. Physics reports, 486(3-5):75-174, 2010.
Gazzaniga, M., Ivry, R., and Mangun, G. Cognitive Neu- roscience: Fifth International Student Edition. Inter- national student edition. W.W. Norton, 2018. ISBN 9780393667813. URL. htttps://boot google. co wwluk//boooo bokkbbokk 21mIIsSDAAnSSAA....
Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability, 2023. URL https::/arxiie org/ abs /204. 14997.
Coronato, A., Naeem, M., De Pietro, G., and Paragliola, G. Reinforcement learning for intelligent health- care upplications: survey. Artificial Intelligence in Medicine, 109:101964, 2020. ISSN 0933-3657.
Glanois, C., Weng, P., Zimmer, M., Li, D., Yang, T. Hao, J., and Liu, W. A survey on interpretable rein- forcement learning. Machine Learning, 113(8):5847- 5890, Aug 2024. ISSN 1573-0565. doi: 10.1007/
10

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
$109944-022-05544www URL hwt.ttttt//o..i. 1007/s10994-024-00000
Lu, R., Li, Y--.., Li, Y., Jiang, J., and Ding, Y. Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management. Applied Energy, 276: 2020. ISSN 0306-2619. doi: http://oo.aggl0...odd.d//eeeeggp
Hein, D., Udluft, S., and Runkler, A. Interpretable policies for reinforcement learning by genetic program- ming. CoRR, abs/1712.04170, 2017. URL http: ///aaaivvv.//////////a//aaa//aaa www.voorg/aae (ttgg/bb/1710 04170.
URL http:://ww..ecieneeddeeee sccerna//rrricee//p////3366666.2
Hod, S., Casper, S., Filan, D., Wild, C., Critch, A., and Russell, S. Detecting modularity in deep neural networks, 2022. URL https: http//pppeeeii..eeeeeeeeee wttpreeiewwwneeeeeeeee nwt/ffrumr id=tFQyjbOza
Luo, L.,, Zhang, G., Xu, H., Yang, Y., Fang, C., and Li, Q. End-to-end neuro-symbolic reinforcement learning with textual explanations, 2024. URL https: //axxiv. org/ abs 2403. 12451.
Huang, X., Chen, D., Ren, T., and Wang, D. A survey of community detection methods in multilayer networks. Data Mining and Knowledge Discovery, 35:1-45, 2021.
Margalit, E., Lee. H., Finzi D., DiCarlo,
Grill-
Spector, K., and Yamins, D. L. A unifying frame- work for functional organization in early and higher ventral visual cortex. Neuron, (2(14):2435-24511 Jul 2024. ISSN 0896-6273, doi: 10. 10.1016/j. 2024.04.018. URL https::t///..... /ww. http//i...oog/1111111111/// i j. 2024 04. 018.
Jiang, Z. and Luo, S. Neural logic reinforcement learning. CoRR, abs/1904. 10729, 2019. URL http://arxiv.
org/abs// 10729.
Kohler, H., Delfosse, Q.. Akrour, R., Kersting, K., and Preux, P. Interpretable and editable programmatic policies for reinforcement learning, 2024. URL https: //arxiv. oorgor/aaa abs/ / 2405 14956.
Marton, S., Grams, T., Vogt, F,, Lüdtke, S., Bartelt, C., and Stuckenschmidt, H. Mitigating information loss in tree-based reinforcement learning via direct optimiza- tion, 2025. URL https:t///// httpp:://aaxvv.ooo org/ abs/2409 08761.
Landajuela, M., Petersen, K., Kim, S., Santiago, C. P., Glatt, R., Mundhenk, N., Petrit, J. F., and Faissol, D. Discovering symbolic policies with deep reinforcement learning. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 Proceedings of Machine Learning Research, pp. 5979-5989. PMLR, 18-24 Jul 2021. URL https: ptttttps:/procceeee pttpe/eicoceeeeeeeeeoeee mlr. press/ v1 v139 haee/aaddaaueeelle
Merel, J., Hasenclever, L.,, Galashov, A., Ahuja, A., Pham, V., Wayne, G., Teh, Y. W., and Heess, N. Neural prob- abilistic motor primitives for humanoid control. CoRR, abs/1811.11711, 2018. URL http::/arxiv..r abs
Pateria, S., Subagdja, B., Tan, A.-h., and Quek, C. Hier- archical reinforcement learning: A comprehensive sur- vey. ACM Comput. Surv., 54(5), June 2021. ISSN 0360-0300. doi: 10.1145/3453160, / //doi.org//ii 1145
html.
Lec, D. and Seung, H. S. Algorithms for non-negative matrix factorization. In Leen, T., Dietterich, T., and Tresp, V. (eds.), Advances in Neural Informa- tion Processing Systems, volume 13, MIT Press, 2000. URL oc wttpeeeefileeeepeeee//////////// pdf.
URL https:
Pignatelli, E., Liesen, J., Lange, R. T., Lu, C., Castro, S., and Toni, L. Navix: Scaling minigrid environments with jax, 2024. URL https: /arxiv.v.vv org/ abs/2407. 19396. arXiv:2407.19396.
Schulman, J.. Wolski. Dhariwal, Radford. A.. and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: htt///xxiv..or http///xxxo.rgabb//////o
Li, Y., Yosinski, J., Clune. J., Lipson, H., and Hoperoft, J. Convergent learning: Do different neural networks learn the same representations? 2016. URL https: //arxiv. //aaxii..rr//bb/e...... abs///ss111111 07543.
7707.06347. arXiv: 1707.06347.
Shi, J. and Malik, J. Normalized cuts and image seg- mentation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 22(8):888-905, 2000. doi: 10. 1.11109944.666888
Lipton, Z. C. The mythos of model interpretability. CoRR, abs/1606.034990, 2016. URL http://arxiv..r abs/1609 03490.
Liu, Z., Gan, E., and Tegmark, M. Sceing is believing: Brain-inspired modular training for mechanistic inter- pretability, 2023. URL https: htttps://aaiiiii //arxivv...// org/ abs/ 2305. 68746. arXiv: 2305.08746.
Shindo, H., Delfosse, Q., Dhami, D. S., and Kersting, K. Blendrl: A framework for merging symbolic and neural policy learning, 2024. URL https: / //aaxiivvvoo../ orrg/ abs/24100 1689..

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RI.
Silva, A., Killian, T., Rodriguez, I. D. J., Son, S.-H., and Gombolay, M. Optimization methods for inter- pretable differentiable decision trees in reinforcement learning, 2020. URL sstps://arkee org/ abs/ 1903. 09338. arXiv: 1903.09338.
Song, Y., Romero, A., Müller, M., Koltun, V., and Scara- muzza, D. Reaching the limit in autonomous racing: Optimal control versus reinforcement learning. Science Robolics, 8(82), September 2023. ISSN 2470-9476. doi: 10.1126fsiiobboiss.dgg44.4. URL http:// / /xxx doi. org/10. http://.....///croootiiiiissss adg1462.
Stermberg, S. Modular processes in mind and brain. Cog- nitive Neuropsyccology, 2563-4) 2011. doi:
10.0880026439922011.55311 URL https:/ /ttp:s:///...../ /doi. /tt.//..10000//4222222444
oxg/10.10801 00000000022433820 2011 557231. PMID: 22185235.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, R., Rees, E., Batson, Jermyn, A., Carter, S., Olah, C.. and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
scaling-monosemannn nonosemantiiiitt//neeee....
Traag, V. A. Faster unfolding of communities: Speeding up the louvain algorithm. Physical Review E, 93(3):032801, 2015.
Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL https: // /aaxxiv..v.. htgg/bs//ea 2211 00593. arXiv: 2211.00593.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang. Z. Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, Z., and Hendrycks,
Representation engineering: A top-down approach ai transparency, 2023. URL https: paops://arxivy org/ atggabs////e /2300 01405. arXiv: 2310.01405.
12

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL.
A. Sparsity Methods
The L1 norm Ei Ti) is known to induce sparser solutions than the L.2 norm (V(E aF) gradients with respect to parameter magnitudes. In contrast the gradient of the L2 norm decreases with its magnitude, resulting in an optimisation landscape that preferentially reduces larger parameters rather than promoting sparsity through the elimination of near-zero parameters. However, while an LI loss function does not directly discourage reducing near-zero parameters, nor does it favour it: the norm optimises for minimal total parameter magnitude, rather than a low count of non-zero parameters, which is what we desire in sparse model.
due to its having constant
N log([x;) + 1), has gradient aa; x. This scales inversely with In contrast, proposed log-bases sparsity loss, Σ,
2
parameter magnitude, thus explicitly promoting sparsity. We note an alternative formulation exp(E log([æs] + 1)), with gradient 8w; = [ 3i which similarly directly encourages sparsity as result of having relatively higher gradients for parameters whose magnitudes are low in the parameter distribution. We adopt the former due to its linear scaling with respect to number of parameters.
The comparative behaviour of these functions can also be intuitively understood by observing the gradients and value of the contour plots in Figure 10 when varying x and y-
2 vbx + y?
2
log(1+xX log(1+y)
2 expl log(l+x) log(1-y)
Figure 10: Contour plots showing the results of L2, L1, log and exponential log loss functions for two parameters.
In our work, we find that utilising the log rather than L1 based regularisation results in a preferable relationship between return and isolation, and between return and ARI, as shown in Figure
Relationship betweon Return and Structural Modulo Isolation with L1 and Log Sparsities
Relationship between Roturn and Adjustod Rand Index betweon Structural and Functional Modules with L1 and Log Sparsities
Idjusted Fand Index
0.8
Solation
0.8
0.7
DO log
DO loy DO L1
0.5
00
0.7
0.75
0.8
0.85
0.9
0.95
0.7
0.75
0.8
0.85
0.9
0.95
Figure 11: The relationship between return and module isolation (left) and between return and ART (right) for models trained with LI and log based regularisation, showing that log based sparsity results in more isolated and functionally aligned modules.
13

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
B. Applying the Standard Louvain Algorithm to MLP networks
The constrained layer-wise connectivity MLPs conflicts with the null-model assumption of uniform connection probability between nodes. The Louvain equation (Equation 2) compares the magnitude of network connection with its expected strength within a random network with the same node orders (kak) ). Unlike the arbitrary sub-graphs observed in traditionally modular networks, neural modules form continuously across multiple adjacent layers, with connections only present between consecutive layers. They thus generally exhibit lower connection density, leading to modules becoming subdivided. Additionally, neural network input layers frequently show high fan-out connectivity patterns where feature information is distributed to multiple downstream neurons. The resulting areas of high connectivity satisfy Q optimization, despite spanning single weight layer and violating the directionality of information processing in the network.
We illustrate these issues in Figure 12, which contrasts a classically modular network architecture with examples of modular MLP networks. While the Louvain algorithm performs well for the former, several 'failure cases' arise in the latter. The green modules exemplify the challenge of distinguishing modules when feature sharing and fan out connectivity occur the input layer. Furthermore, few of the Louvain detected modules span the full depth of the network, despite it being visually apparent that the network modules do so, which results in the network modules being subdivided.
sog
00
Figure 12: Examples of the clustering results of the Louvain algorithm when applicd to modular network without layer-wise structural constraints (left), and to modular MIPss. Note the modules (in green) which span multiple modules in the input layer, and the modules which fail to span the full module depth.
14

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RI.
C. Hyperparameter and Training Choices
Table I: PPO Hyperparameteers
ARCHITECTURE
HIDDEN SIZE
NUMBER LAYERS
TRAINING
PARALLEL ENVIRONMENTS
STEPS PER ENVIRONMENT
128
MINIBATCHES
EPOCHS
16
LEARNING RATO
5E-4
MAX GRADIENT NORM
0.5
CLIP
0.2
ENTROPY COEFFICIENT
0.01
VALUE FUNCTION COEFRICIENT
0.5
REGULARISATION
d, (EQUATION
0.95
X (SECTION 3.2)
10
RELOCATION INTERVAL (SECTION 3.2)
C.1. Pruning and Fine-tuning
We prune our networks after 80% of the training steps, and train for the remaining 20% with Ace = 0. Pruning involves setting all weights with value below 1% of the maximum absolute weight their layer to 0, and removing all hidden-layer neurons with order (total sum of incoming and outgoing weights) of less than 1% of the maximum in their layer. Gradients for the removed parameters are masked 0 for the remainder of training. The pruning fixes sparse and modular architecture, and we find the fine-tuning with no regularisation improves performance (Figure 13), does not compromise the isolation the modules (Figure 15), and slightly increase their ARI (Figure 16). We select a 1% pruning level because this does not result in a significantly reduced return compared to lower pruning levels, offers a higher level of resulting sparsity (Figure 14), and results in the modules with the highest ARI.
Finetuned Return Initial Return for Dynamic Obstacles
Fine-tuned Return Fraction of Actor Weights Pruned for Dynamic Obstacles
0.95
0.0
Finetued Reeurn
of Weghtt
0.9
0.8
0.86
0.7
grune (000000.01, 0.001)
prunc toce)0.005, 0.003)
pnno tra--00.01 0.03)
0.5-
-pnns maz-10 606, 0.005)
-punc nac= 001, 0.001)
Face D.)
(ucc 01, U.33)
0.70
retur intal moum
0.5
0.76
0.8
0.85
0.95
0.75
0.8
0.86 Ein-tuned Return
0.0
0:55
Iniial Return
Figure 13: The relationship between initial return (before pruning and fine-tuning) and post fine-tuning return for different pruning thresholds applied DO networks with Aoo [0.005, 0.1].
Figure 14: The fraction of actor weights pruned and the fine-tuned return achieved with different pruning levels applied to networks with Xoe E [0.005, 0.1].
C.2. Selecting d,
As introduced in Section 1, the value of d, varies the relative significance of distance and weight in regularisation. Primarily, given the range of possible distances between neurons in adjacent layers d, E [1, v2], value of d, = means that a weight connecting two vertically aligned neurons contributes 0 to the total connection cost. This is apparent in Figure 17, where networks with d, = 1 have a high number of these 'vertical' weights. In contrast, as d, is decreased, we see an increasing number of distant connections the network.
15

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
Fine-tuned Return vs Isolation for Dynamic Obstacles
Relationship between Fine tuned Return and ARI for Dynamic Obstacles
0.0
Fine Sreetireed soleton
Firc-tmont
0.4
0.3
Trcc-00
0.2
mac=10 01
-0.no Trac-00.00
prune nace 0.000)
-0.00 1A0-(0 no08
0.1
pario faa- 0.006)
srurer Facem000000 01, 0.01)
prrre frec= 01, 0.)3)
orune_ 2arp___omm0011. 1, 0.1]
0
puree puune_haccpp0 0.3)
0.75
Finc tuned Return
Fine-tuned Return
Figure 15: The relationship between fine-tuned return and isolation for different pruning levels. With a pruning level we do not set Ace = 0 for fine-tuning, as this allow all weights in the fully connected network to increase, compromising modularity.
Figure 16: The relationship between fine-tuned return and ARI for different pruning levels.
d. 0.8
d, 0.9
d. 0.95
d. 1.0
Figure 17: The impact of de on the structure of the emergent modules (pre-pruning and fine-tuning). The networks shown have varying Ace values, as the scale of the connection cost varies when d, is varied, but where selected to show networks with equivalent returns: top z 0.75, and bottom row = 0.82.
We run all experiments in the main body of our work with d, 0.95, which was selected by comparing the relation between return, isolation and ARI of networks at different values. The results are plotted in Figure 18 and show that higher d, values result in higher isolation. Despite their high isolation score, we find that the modules detected in d, = networks align poorly with the correlation partitions: the resulting ARI values do not exceed 0.3 and do not increase monotonically with regularisation. We find that d 0.9 offers the greatest ARI relative to return, but the difference between 0.8, 0.9 and 0.95 relatively minor, so we select d, 0.95 for its significantly higher isolation scores.
Relationship Between Returnn and Structural Module Isolation
Relationship between Return and Adjusted Rand Index of Structural and Functional Modules
Jub 0.9
sub
d Sub 0.58
sub 0.9
d sub 1.0
0.0
sub 0..5
0.8
-. sub 1.0
bollation
0.6
0.7
0.8
0.4
0.5
0.2
0.65
0.25
0.35
0.7
0.75
0.95
Return
Figure 18: The impact of varying d. on the relationship between return and isolation (left) and between return and ARI (right) of the resulting networks' partitions.
16

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
C.3. Aec Scheduling
For the pruning fraction and d,. select the Ace schedule based on the resulting relationships between return and isolation, and return and ARI. We select linear introduction of the regularisation loss between 20% and 30% of training steps due its high relative performance with respect to these metrics, as shown in Figure 19.
Relationship betveen with Different Regularisation Schedules
Module Isolation
Relationship Structural and Functional Modules with Different Regularisation Schedules
Adjusted
1
0.9
Adjussed Rand Index
Isolation
0.0
0.8
10. 0]
[0. 0]
10, 0.1]
0.1]
10, 0.4]
[0, 0.4]
0.2
0.2. 0.6)
[0.2. 0.6
0.5
0.75
0.8
0.85
0.9
0.95
0.75
0.8
0.85
0.9
0.95
Figure 19: The impact of different introduction schedules for the CC loss the relationship between return and isolation (left) and between return and ARI (right) of the resulting networks' partitions. legend value [x, y] indicates that Aee was increased linearly between fractions x and of the total training steps.
17

---

Inducing, Detecting and Charactcrising Neural Modules for Functional Interpretability in RL
D. Runtime Results
D.I. Regularisation and Relocation
Figure 20 shows the average compilation and training times for 4M steps, comparing a vanilla PPO implementation with cases where distance weighted regularisation and neuron relocation are implemented in isolation and combined. We find an overall time increase in the combined case of 17%, which is dominated by the introduction of the connection cost loss.
The Impact of CC Loss and Neuron Relocation Training Times
Vanila PPO
Neuron Relocation
CC Loss
cC
10
20
30
40
50
60
70
80
80
100
Complattio Time
Training Time
Figure 20: The average compilation and training times, for 4M training steps, of the Vanilla PPO implementation compared to PPO with neuron relocation, with distance weighted regularisation, and with both.
D.2. The Extended Louvain Algorithm
The Louvain algorithm is commonly assumed to have a runtime complexity of (1nlog(n)) (Huung et al., 2021) where is number of nodes. However, no definitive analysis of its time complexity has been performed, and other sources instead find a time complexity of (m)) in the number of edges (Traag, 2015). We compare the duration of the Louvain method to our proposed adaptations, with the caveat that this evaluation limited by the relatively small scale of networks examined.
We observe that for our networks, the standard Louvain time complexity appears to match the O(m) assumption. As our internal version simply applies Louvain but with fewer nodes and edges, it follows that it also has linearly increasing duration with m, and this is what we observe in Figure 21. The figure further shows the duration of the fine-tuning stage, which also appears to be linear in m. A larger sample of networks, including larger ones, would be necessary to rigorously demonstrate this, however.
The calculation of the activation correlation matrix dominates the duration of our extended Louvain, and the duration of this itself dominated by the model inference over the 10,000 episodes for which we collect activations, as shown in Figure 21. We note that we parallelise these episodes using JAX, so inference over 10,000 episodes takes negligibly longer than over fewer number of episodes, but 10,000 reach the memory constraints of the 24GB GPU. Duration appears to increase slightly as the network size increases, but with multiple outliers. Should activation collections or correlation computations become prohibitive in terms of memory or computation as we scale to larger models, a number of approaches could be take to reduce the problem size. For example, we could take advantage of the localised nature of our modules, and separately collect and process activations for smaller network regions.
Time Taken Different Stages of the Module Detection with Increasing Numbara Natwork Ediges
Time Taken Calculate Correlation Mattik Increasing Numbers Natwork Edgea
Louvain Finetuning
Thoe
Number Edges
Number Edges
Figure 21: The observed duration of the Louvain algorithm and of the isolated components of our extended version.
(Accessed 9/01/25)
http:///ewseeoooooaieeeeeeeeeeee (Accessed 9/01/25)
18

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RI.
E. Emergence of Modularity
E.1. Additional Modularity Examples
We show two further examples of modularity emergence for each environment in Figure 22, and further show these networks partitioned using the fine-tuned internal Louvain Figure 23. We find that the navigational modules emerge consistently and with the same structure in the DO and 3D-DO tasks. In the G2K task, we observe two structures: one which closely resembles the DO modules, and where action becomes disconnected from the network (but can still be selected based on its relative logit value compared to the remaining actions). As shown in Figure 23, we are still able to separate X and Y navigational modules controlling the remaining three connected actions.
We also show, in Figure 24, the impact of continuing to increase Ace beyond the emergence of fully isolated modules: the prioritisation of goal information increases till these are the only features considered, but at certain threshold we observe a collapse in both sparsity and return.
M X
00:0
G2K
Acc
0.01
0.02
0.04
0.1
Figure 22: The emergence modularity with increasing Ane across two seeds in environments.
19

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RI.
M.Y.
DO
3D-DO
G2K
Acc
0
0.01
0.02
0.04
0.1
Figure 23: The networks shown in Figure 22 partitioned using our fine- tuned internal Louvain approach. The ordering of detected modules varies between Ace values due to randomness in the order in which the Louvain algorithm considers nodes.
ACC
0.2
0.5
1.0
Return
0.73
0.62
0.23
Figure 24: The network structures and returns observed when continuing to increase Aee above the range considered the main text.
20

---

Inducing, Detecting and Charactcrising Neural Modules for Functional Interpretability in RL
E.2. Isolating the Impacts of Regularisation, Distance and Relocation
While the emergence of visually distinct modules is evidently reliant on local connectivity induced by the connection cost loss and neuron relocation, we here analyse how these protocols contribute to the non-visual modularity measures. We conduct ablation experiments comparing networks trained with and without distance weighting and neuron relocation across different regularization strengths, and show how this affects module isolation and correlation alignment (ART) in relation to return (Figure 25).
Naturally, networks without regularization (Ace = 0) have significantly less isolated modules than any regularized variant. We further find that both distance weighting and relocation increase isolation, but with a diminishing impact as regularisation increases. We expect this occurs occurs because strong sparsity constraints force isolated pathways regardless of their locality, The alignment (ART) between weight-based and correlation-based partitions shows different pattern: distance weighting but not relocation increases ARI scores, but in manner that increases as regularisation increases. Examinnigg sparsity, shown in Figure 26, we observe that distance weighting reduces sparsity relative to return, but relocation compensates for this, particularly at high regularisation levels. This suggests that relocation enables important but initially distant weights to be preserved, allowing sparsity to be achieved in a manner that is less damaging for return.
Overall, these results indicate that both distance weighting and relocation contribute to structural modularity, while distance weighting is particularly important for aligning structural and functional modularity. Although these effects dependant on the strength of regularisation, both are observed in the most useful regularisation ranges where modularity is observed, but the impact return is still relatively low. Intuitively, the distance weighting encourages additional sparsity and thus isolation by encouraging computations to be distributed across few weights, since each weight beyond the first weight is necessarily longer and more expensive. The neuron relocation likely enables to network to restructure in a manner that makes the weights with a greater performance impact shorter and thus less costly, promoting greater sparsity among less important weights (although weight not a direct measure of importance it appears to be a relatively good proxy for it). This is particularly important given the connection cost scheduling: by the time sparsity is introduced, the network has already learnt effective computations which wish preserve. Further analysis will be required to fully understand and formalise how these impacts arise.
The Relationship Between Return and Isolation with Different Regularisation and Relocation Protocol
The Relationship Between Return and Adjusted Rand Index with Different Regularisation and Relocation Protocol
0.6
0.8
0.4
Isoaton
0.7
Aduastdd
0.2
0.5
Distance welghted sparsity with relocation
Vanila FPO, Meon
Dister tx-weigf aed sporsly w relocatio
Vanilla PPO: Mean
No distance will lghting
No distancce weighting
No neuron relocation
No neuron re ocadon
0.05
0,8
Figure 25: The relationships between return and isolation (left) and between return and ARI (right) with the distance weighting and
neuron weighting separately ablated, across range of Aec values([0.005, 0.1] where distance weighting is included and |0.0005, 0.01] otherwise, to achieve equivalent regularisation levels and returns). We include the mean values achieved with the Vanilla PPO case, which corresponds to Aee = O and no relocation for comparison.
21

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL.
The Rellaionship Between Return Sparsity
with Different Regularisation and Relocation Protocal
Vanilla PPO,
Destrnce--ette serighttd relocation
relocation
0.25
0.8
0.85
0.9
0.90
Return
Figure 26: The relationships between return and sparsity, defined the proportion of weights with magnitude below 1% of the maximum magnitude their layer, for ablation cases considered Figure 25, The vanilla PPO case has significantly lower mean sparsity 0.028, so is not shown.
22

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
F. Pong Results
In this section we demonstrate the utility of our approach on Pong. We show that the sparse training protocol learns single sparse module, which enables identification of flaw in learnt Pong policies, as previously identified by Delfosse et al. (2024).
We train MLP policy network custom implementation of Pong in JAX, which we modify to return symbolic observations. The observation consists of the agent paddle position, the opponent paddle position, the and positions of the ball and the x and velocities of the ball. The opponent adopts the standard 'follow ball' policy and the agent receives sparse rewards of 11 and 1 when the opponent and agents score points respectively. We fix the regularisation parameters dx l and the relocation intervals to use the same values as the Minigrid experiments, and conduct small sweep over Ace values. Full training parameters are shown in Table
Table 2: PPO Hyperparameters
ARCHITECTURE
NUMBER OF LAYERS
2
TRAINING
PARALLEL ENVIRONMENTS
16
STOPS PER ENVIRONMENT
MINIRATCHES
EPOCTS
16
LEARNING
MAX GRADIENT NORM
0.1
GAE A
0.99
CLIP
0.2
VALUE FUNCTION COPPEICIENT
0.5
TRAIN STEPS
PRUNE FRACTION (APPENDIX C.I)
0.01
FINETUNE STEPS (APPENDIX C.I)
10M
REGULARISATION
d, (EquATiON 1)
0.95
(SECTION 3.2)
RELOCATION INTERVAL (SECTION 3.2)
SCHEDULING (APPENDIX C.I)
0.4-0.41
Unlike in the Minigrid tasks, the Pong agent moves along a single axis. We thus observe a single module in the computational graph, which becomes increasingly sparse as Ace increased, as shown in Figure 27, We show the impact of regularisation on agent performance and the number of network parameters in Figure 28. Up to Ace of 0.05, we observe a negligible impact on agent performance, with all policies with Aee 0.045 achieving a perfect average score of 21. Beyond this we observe variability between seeds and a significant deterioration in performance in some cases. As in the Minigrid experiments, the regularisation and pruning also significantly reduces the number of parameters in the network, and we find we can achieve mean game score of 21 with just 37 parameters.
1.8.1
Aoe
0.03
0.04
0.05
0.06
Figure 27: We observe sparsity increasing with Acc in the Pong policy network. The simplicity of the task means single module is learnt.
Delfosse et al. (2024) train Successive Concept Bottleneck Agents (SCBots) the Pong environment and find a brittle reliance on the opponent position the resulting policies. This an artefact of the opponent policy, which attempts keep
23

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
Average player score for different lambda values
Number of parameters the pruned model far different lamada values
90
Sead CC
Sool E53
Seed
Hoan Player Ecor Carre
80
15
93
Seed 66
serd eo:
Send 66-3
0.990
0055
D040
1,045
0.220
0.03
0.000
1L053
0,050
0033
0.040
0.43
2,000
0.005
0.000
2083
Lambda CC value
Lambda CC value
Figure 28: Left: The impact of Ace regularisation on the mean player score at game end, with 21 indicating a 100% win rate. Right: The impact of Ace regularisation the number of network parameters post pruning, where the baseline non-ree ron-lagulrr network uses 435 parameters.
the paddle centre aligned with the centre of the ball. While this results in high performance in training, is an example of proxy gaming and undesirable: the opponent policy changes, the agent loses the ability to perform its target task of returning the ball.
Based on this observation, retrain set of policies with the opponent position removed from the observation space. The results, shown in Figure 29, show an increase in score variability at lower Àee values, but still demonstrate the ability to achieve an 100% win rate up to Acc 0.55. By necessity, this policy relies solely on agent and ball information and is thus robust to the more realistic scenario of variable opponent policy.
Average pleyer score for afffrrent lambda values urszen uppoment
Moan Scon at Game End
15
Seed
661
0.053
0.055
0.040
0.04
0.50
0.095
0.000
D085
Lambda valar
Figure 29: The impact of Ace regularisation on the mean player score at game end, when the opponent position removed from the observation space during training.
24

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
G. Module Detection Method Data
Table 3: The Isolation and Functional Alignment (ART) actor network modules detected using the weights, correlation, internal, fine-tuned and fine-tuned internal versions of the Louvain algorithm. Results are averaged across 10 seeds in each of the DO, 3D-DO and G2K environment.
ISOLATION
W L.V.
C.V.v
INT.
FT INT.
0.01
0.325
0.434
0.575
0.742
0.775
0.03
0.671
0.667
0.770
0.836
0.899
0.05
0.699
0.669
0.741
0.866
0.924
0.09
0.778
0.671
0.787
0.937
0.943
0.795
0.812
0.13
0.787
0.728
0.795
0.950
0.949
0.15
0.803
0.708
0.822
0.953
0.949
FUNCTIONAL ALIGNMENT (ART)
Agc
W.V.
CLV.
INT.
INT.
0.01
0.165
0.140
0.305
0.237
0.03
0.273
0.256
0.639
0.462
0.05
0.282
0.273
0.675
0.571
0.07
0,310
0.321
0.676
0.625
0.343
0.11
0.385
0.436
0.750
0.804
0.13
0.378
0.478
0.724
0.834
0.15
0.383
0.489
0.721
0.825
Table 4: The Isolation and Functional Alignment (ARI) of actor network modules detected using the weights, correlation, internal, tne-ttned and fine-tuned internal versions of the Louvain algorithm. Results are averaged across 10 seeds the G2K environment only.
ISOLATION
Age
LvV.
CLV.
INT.
INT.
0.01
0.398
0.423
0.420
0.592
0.656
0.03
0.508
0.549
0.535
0.744
0.754
0.05
0.556
0.517
0.579
0.750
0.764
0.07
0.577
0.517
0.622
0.760
0.812
0.624
0.11
0.648
0.541
0.683
0.800
0.791
0.15
0.707
0.539
0.762
0.837
0. 0.820
FUNCTIONAL ALIGNMENT (ARI)
Aee
W LV.
C
INT.
FT
FT INT.
0.01
0.145
0.111
0.215
0.03
0.180
0.180
0.618
0.398
0.05
0.170
0.195
0.514
0.439
0.07
0.184
0.253
0.560
0.466
0.09
0.196
0.220
0.510
0.519
0.13
0.149
0.499
0.311
0.697
0.15
0.145
0.494
0.287
0.652
25

---

Inducing, Detecting and Characterising Neural Modules for Functional Interpretability in RL
H. Intervention Data
present full action statistics for the networks interpreted Section 4.6 showing the frequency of actions and their outcomes for the unmodified network, and for versions where modules are modified through negative saturation or negation. We bold the data corresponding the axes the targeted community is associated with.
Table 5: Action Statistics for 3D-DO network (Aue 0.06, Figure 9a).
Directions
Freq-
Failure
Success
Continue
Initial Network
up/down
21.33%
8.62%
29.67%
61.71%
Return 0.77
left/rght
42.55%
8.29%
21.70%
70.01% 18:
fwd/bwd
36.12%
9.81%
37 47.48%
52.71%
Negative Saturation
Community 0
upidown
40.74%
2.11%
1.20%
96.69%
Return
leftright
52.92%
2.51%
1.23%
96.26%
twd/bwd
6.34%
3.12%
6.80 %
90.07%
Community
up/down
22.59%
3.47%
2.81%
93.73%
Retum
left/right
6.18%
5.72%
fwd/owd
71.23%
3.49%
2.03%
94.48%
Community 2 Retum
upidown
6.02%
4,01 %
15.16%
80.84 %
left/right
22.91%
5.59%
6.37%
0.4%%%%
fwd/twd
71.07%
3.32%
2.82%
93. 93.86%
Negation
Community Rotum = 0.14
upidown
5.16%
2.79%
1.60%
61
loft/rght
11.14%
2.43%
1.12%
96-45%
fwd/bwd
83.70%
Community Retum = 0.09
upidown
0.99%
10.03%
4.47%
50%
left/right
97.01% %
1.11%
0.01 %
98.88%
fwd/twd
2.00%
12.36%
Community Retur 0.39
upidown
89.59%
0.01
99.00%
leftrright 3.53%
8.48%
13.27%
78.26%
fwd/bwd
6.87%
5.92%
8.15%
35.93%
Tahle 6: Action Statistics for G2K network (Acc 0.12, Figure 9h)
Directions
Freg.
Failure
Success
Continue
Initial Network
upidown leftlight
51.03%
1.16%
42.99%
55.85%
Return
3.57%
28.96%
Negative Saturation
Community 0
upidown
2.38 %
1,38%
12.57%
86.05%
Return 0.35
left/right
97.62%
1.18%
0.36%
98.46%
Community
upidown
97.05%
1.23%
0.41%
98.36%
Return 0.39
lefU/ight
2.95 %
1.36%
13.64%
85.00%
Negation
Community 0
upıdown
98.64%
1,03%
0.01 %
98.96%
Return = 0.14
left/rght
1.36%
4.25%
12.22%
83.53/8
Community
upidown
1.07%
7.79%
7.53%
84.68%
leftright
Table 7: Action Statistics for G2K network (A.e 0.02, Figure 9c)
Directions
Frey. 50.09%
Failure
Success
Continue
Initial Network
upldown leftright
0.21%
44%
Return 0.99
49.91%.
0.41%
40.51%
59.07%
Negative Saturation
Community 0
up/down
4.78 %
2.56%
7.06 %
90.38%
Return = 0.40
loftfright
95.22%
1.36%
0.63%
98. 08.00%
Community
upkdown
72.00%
1.20%
0.30%
98. 98.50%
Return
left/right
28.00%
1.01% %
0.01%
98.98 %
Negation
Community 0
up/down
78.39%
1.0%%
0.57%
98.36%
Return 0.57
letthight
21.61%
1.04%
4.47%
94.50%
Community I
upidown
82.10%
1.46%%
1.31%
97.23%
leftright
26


Reviews:
[{"review_id": "sYkjczhDvo", "reviewer": "Reviewer_s33i", "summary": "This paper proposes a method to learn a functionally modular and interpretable model in an RL policy network. It combines a few ideas:\n\n- Spatially embedded neurons with a distance-weighted loss to encourage locality\n- Neuron relocalization (Algorithm 1)\n- Partitioning the model into different modules using variants of the Louvain algorithm that take into account functional connectivity (equation 5)\n\nThe paper tests these ideas on a few GridWorld RL environments, and find that the discovered modules are interpretable; when the modules are intervened upon, they display effects which are consistent with their modular nature.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review. We appreciate the positive comments on the quality of our execution and are grateful for the constructive feedback, which we respond to below.\n\n**Contributions.** We recognise that our training approach builds on Liu et al (2023). Specifically, we do this by adapting distance-weighted sparsity to RL policies rather than tasks with explicit mathematical structure, and by proposing extensions including log-based sparsity to improve performance. We further develop a novel clustering approach and propose modularity metrics which enable automated detection and characterisation of functional neural modules. We believe this is a crucial step to enabling scalable module interpretability.\n\n**Interpretability Insights and Scalability.** We appreciate your understanding of the difficulties of finding objective insights in interpretability research, a challenge which we approached by quantifying the impact of modules through ablations. We agree that demonstration in more complex environments is an important direction for future work. To move towards this, we have now implemented the framework on a non grid-world environment (Pong), as proposed by reviewer qymG, and will formalise and incorporate these results in the final paper. We believe that our rigorous evaluation approach, particularly the ablation studies and hyperparameter analyses, will provide a solid methodological foundation for further scalability.\n\n**Specific Corrections.** Thank you for drawing the typos to our attention - we have fixed these. The sentence about negative saturation was intended to explain that when modules are negatively saturated, the modified policy behaviour shows evidence of the modules' axis-specific navigation functions. We have clarified this sentence. We also acknowledge that the claim about consciousness is unnecessary to motivate our work and have removed it.", "claims_and_evidence": "The paper claims to:\n\n- extend local neural networks methods to RL\n- create an extended Louvain algorithm to detect communities in the neural networks\n- demonstrate that interventions on the network parameters that are derived from the Louvain algorithm are consistent with their purported roles\n\nThese are fairly thin claims–it seems to me like a straightforward application of prior methods on a slightly different problem, and strictly speaking their extensions are not necessary to make the method work specifically in the RL setting, they're more like enhancements of the methods in general–but they are well supported.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate. These are still very much toy problems to demonstrate the method. If the ultimate goal is to apply this to non-trivial tasks, as the introductory framing in terms of AI ethics implies, one would want to extend this to larger and more complex environments.\n\nAs is often the case in interpretability research, the claims of finding insights are highly subjective. Their interventions in 4.5 partially address this issue, but it doesn't convince me that this will scale to non-trivial problems.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are sound and appropriate.", "relation_to_broader_scientific_literature": "This is a straightforward extension of Liu et al. (2023) to RL with a few bells and whistles to find good networks via the Louvain algorithm.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper feels like a rather straightforward extension of prior work. It is well executed, however. My preference is to evaluate the content on execution rather than the more subjective issue of originality.\n\nI found the Background and Related works section to be particularly well-executed and far-reaching.", "other_comments_or_suggestions": "There's a few typos:\n\n- \"Comission\"\n- \"cna\"\n- \"combine\" rather than combined on line 399\n\nThere's a sentence that doesn't make sense: \"negative saturation of modules evidences their axes specific navigation function\"\n\nI don;t like the claim that conscious decision making relies on modular processing–it seems unnecessary to drag consciousness into this.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxSfbb0MoO", "reviewer": "Reviewer_jddf", "summary": "This paper presents a pipeline for inducing, detecting, and characterizing neural modules within reinforcement learning (RL) policy networks to enhance interpretability. By penalizing non-local connectivity and encouraging sparsity and locality in network weights, the fully connected networks used in the study exhibit functional modules. To automatically detect these modules, they extend the classical Louvain community detection algorithm by incorporating a “correlation alignment” metric that accounts for the unique architectural constraints of neural networks. The method is validated in both 2D and 3D Minigrid environments, where distinct navigational modules emerge that correspond to specific movement axes. Furthermore, the paper demonstrates that targeted interventions—such as disabling or perturbing specific modules—can empirically confirm their functional roles. Overall, the work offers a framework for decomposing and understanding complex RL decision-making processes through functional modularity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed response and consideration of our work. We appreciate your recognition of the compelling nature of our proposed approach, and respond to the points raised as follows:\n\n**Scalability.** We agree that application to more complex tasks will be valuable for further evidencing scalability. Our automated module detection techniques, in particular, provide a robust foundation for this future work. As proposed by reviewer qymG, we have now implemented the framework on a non grid-world environment (Pong) and will formalise and incorporate these results in the final paper.\n\n**Interpretability vs Performance Trade-off.** We acknowledge this trade-off, which, as you note, is a tension observed both in our work and the broader interpretability field. One advantage of our approach is the ability to moderate this balance using the regularisation factor λcc. This differs from other white-box approaches like decision trees and offers a valuable means of tailoring the interpretability-performance trade-off for different use cases.\n\n**Extension to Alternative Architectures.** Thank you for raising the potential for extension to CNNs or transformers. We agree this is a valuable future research direction and share some preliminary thoughts about extending our work in this direction:\n\nConsidering CNNs, the sparsity and distance metrics are not obviously applicable to standard kernel computations, but there is potential to consider distance and sparsity metrics in branched architectures (such as the Inception models). Conceptually, the notion of functional modularity seems more applicable to decision-making than image-processing tasks. It may thus be more interesting to interpret modules within fully connected layers downstream of convolutional layers, and this may also improve the interpretability of the intermediate latent space. \n\nConsidering transformers, our modularity pipeline could be applied (with adaptation) by taking attention heads as network nodes or, as with CNNs, by interpreting MLP layers only. Alternatively, we could frame parameters as nodes and relocate vectors within attention head matrices and positions within the residual stream.", "claims_and_evidence": "The paper’s claims are generally well-supported. It convincingly shows that spatial regularization and neuron relocation lead to the emergence of functionally coherent modules and that its extended Louvain method reliably detects these modules. Interventions validate that these modules serve distinct functions, and the trade-off between improved interpretability (and sparsity) and a modest performance drop is clearly demonstrated. However, while the results in simple 2D and 3D Minigrid tasks are promising, the **scalability claim** is supported only in these limited settings on fully connected networks, suggesting that further evidence on more complex tasks is needed to fully substantiate scalability.", "methods_and_evaluation_criteria": "Methods and evaluation criteria is the main strength of this paper, from inducing, to evaluation, and finally to knock out studies, all seem fine to me.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Not very toughly, I am not much familiar with Minigrid tasks or the Louvain algorithm.", "relation_to_broader_scientific_literature": "The paper tries to bridge RL and interpretability literatures which make it well suited for broader scientific literature.", "essential_references_not_discussed": "Non detected", "other_strengths_and_weaknesses": "The paper’s approach of imposing spatial correlations is conceptually compelling and aligns well with neuroscience theories explaining feature maps in the visual cortex (as cited). This integration of spatial regularization into reinforcement learning offers enhanced interpretability and a clear mechanism for module emergence. However, a notable drawback is that this spatial constraint appears to cap model performance, as evidenced by previous studies and the modest accuracy drop reported here. Consequently, while the model yields solutions that are more interpretable, it may preclude discovering the high-performing strategies that less constrained, more powerful networks can achieve. This trade-off highlights a broader tension in mechanistic interpretability research—balancing the need for clear, interpretable mechanisms against the pursuit of state-of-the-art performance on complex tasks.", "other_comments_or_suggestions": "None", "questions_for_authors": "To what extent the pipeline depends on the MLP architecture of the models tested? Can one apply the same pipeline for CNN or transformers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Km4lROUfGS", "reviewer": "Reviewer_qymG", "summary": "The authors identify that most (post hoc) interpretability methods focus on explaining models' units (e.g. neurons), which does not scale. They propose to have interpretability at the level of *functional modularity*. They just try to identify neural modules, which are groups of neurons that are functionally related. \nThey use a \"connection cost\" lost to induce greater sparsity (than classic L1-loss) in the neural network, and perform neuron relocation.\n\nThey show that their method is able to induce interpretable modules  in a navigation task, and that these modules are more interpretable than the original network.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and thoughtful review. We are grateful for your recognition of the potential of our work and the constructive feedback which we have applied as follows. We have also updated the figures and draft PDF on the project page to reflect the changes: https://sites.google.com/view/mod-xrl/home \n\n**Additional Experiments.** We agree that a non-maze application would strengthen our results and appreciate the Pong suggestion - thank you! We have adapted the Gymnax version to return a symbolic observation, and have implemented the distance-weighted sparsity training. Initial results show this learns a single sparse module that uses only a subset of the inputs and actions. The agent focuses on the opponent even at high sparsity, and we are excited to conduct ablations to see if this offers insights into the Pong misalignment problem. We have added these early results to the project page, and will include complete findings in the camera-ready paper.\n\n**Structure and Clarity.** We appreciate these helpful suggestions. We agree the claim regarding levels of abstraction is unecessary and have instead expanded the argument for the utility of interpretability. We have added task scores to Figure 2 and legends to Figure 9 and have extended all captions such that the results shown are now broadly understandable in isolation from the text. We have added a set of research questions to guide the Experiments section, reduced the Background to contain only necessary information and extended the removed information into a Related Work section. Thank you for the related papers, particularly Delfosse et al. (2024) and Kohler et al. (2024), which provide valuable examples for the utility of RL interpretability. We have included these in the Introduction and the other papers in Related Works. We have retained half a page for the Pong results, but may have to move a portion of these or the Related Works to the Appendices to meet the 9 page limit.\n\n**Performance Metric.** Since the reward is 1 on task completion and 0 otherwise, the return in Figure 8 is equivalent to both the success rate and average reward. We have clarified this in the text and caption. \n\n**Neuron Position Optimisation Intuition.** We appreciate your interest and describe our intuition below. We have added this to the Methods and App. E.2. We consider the distance weighting as encouraging computation to distribute across few weights and neurons, as each additional weight used is 'more expensive' than using the single shortest weight. Since we schedule λcc, sparsity is introduced when the network already implements relevant computations. Position optimisation thus allows existing important weights to move and become 'less expensive', while less important ones are more heavily penalised by the CC loss. We expect this same intuition holds without scheduling, as initial weights will bias learning towards specific local optima in the parameter space. Regarding the **frequency of position optimisation**: applying it every T steps improves module isolation and ARI (as reported in App. E.2). These metrics are artefacts of the learnt network connectivity, so we would not obtain the same benefits from relocating neurons after training.\n\n**Computational Cost.** The complexity of the connection cost calculation is linear with depth and quadratic with width. Relocation is linear with layers and cubic with width, but this can be mitigated by reducing the number of swaps considered or increasing T. We trained a set of networks with increasing widths (32 to 512) and depths (2 to 50). With increasing width, the time increase due to the CC loss remains a relatively constant percentage of train time (15-19%), whereas the relocation percentage increases from 1% to 7% , which supports the theoretical complexity. We present network partitioning complexity results in App. D2. \n\n**Log-loss Novelty.** We are unaware of prior work using a logarithmic sparsity loss, but can't definitively state that it has not been applied in contexts we have not identified.\n\n**Exponential Log-loss.** We initially ran a small set of experiments with the exponential formulation and did not observe any differences in results. Since we trained networks with different numbers of weights, in hyper-parameter tuning and due to varying input and output dimensions, the non-exponential formulation was more desirable, as it is additive with number of weights and made selecting appropriate λcc ranges straightforward.\n\n**CNN Application.** For conciseness, we point you to our response to reviewer jddf. \n\n**Module Interpretability.** We did not come across any non-interpretable modules, but agree this is an interesting point and that high level modules in complex networks may not be immediately interpretable using ablations. We are excited about the potential to address this by applying our modularity technique in a hierarchical manner, which may enable the extraction of interpretable submodules.", "claims_and_evidence": "The main claims are clearly outlined in introduction.\nThe biggest claim is that local and sparse Neural Network are more interpretable than dense ones, which is commonly agreed upon in the litterature. No evidence (requiring human evaluation) is provided to support this claim, but I don't think that it is necessary. They also further explain: \" High isolation implies minimal inter-module connectivity, resulting in stricter decomposability and enabling more independent module analysis.\", which is a good argument for the modules' interpretability.\n\nIn introduction, some minor claims are not supported by citations:\n\"When considering its scaling to complex domains, RL interpretability must be considered at a level of abstraction which\nbalances tractability with fidelity to the underlying model.\" (l 43-45). Such claims could be removed. \nIn the introduction, a lot of litterature argue against the use of black-box neural networks, and favor the use of instrinsically interpretable models (e.g. decision trees, supported by Bastani). I would suggest to also spend the intro motivating the necessity of increasing the interpretability of neural networks (which is the main focus of the paper).\n\nThe results detailled in Functional Interpretability should somewhat be already highlighted in the captions of the figures. You can write Forward/Backward module in 9a.", "methods_and_evaluation_criteria": "The proposed method:\n* brings local and sparse networks to RL.\n* proposes to use Logarithmic Sparsity Loss to increase even further sparsity in the network.\n* extends the Louvain algorithm to detect modules in the network.\n\n\nThe authors evaluate their method on a navigation task, but only report interpretability, no performance metric is reported. Before checking for the level of interpretability of agents, we first need to check to which extent they learned to solve the task. I have not been able to find completion rate, or any other any performance metric in the paper.\nIn *Limitations and Future Work*, the authors mention that \"*λcc controls this balance*\", but I do not see any results on how the performance of the agents is affected by the value of λcc.", "theoretical_claims": "Not Applicable.", "experimental_designs_or_analyses": "There are 2 major points of improvement here. \nThe first one is about clarity. The experimental section is quite hard to follow. A very detailled analysis has been done, but it is hard to follow. I would suggest to have a more structured approach to the experimental section.\nSpecifically, I would suggest to have list of scientific questions at the beginning of the experimental evaluation section (often denoted Q1, Q2, ... etc), that are each answered in different paragraphs. For example\nQ1/ Can more interpretable modules reach similar performances as non constrained baselines?\nQ2/ Does our algorithm induce more interpretable modules than the original network? (maybe name your agents that include every interpretability inducing techniques) \nQ3/ How does the regularisation parameter affect the performance and interpretability of the modules?\nQ4/ other ablation studies.\n\nThe second is to also evaluate their method on a non maze environment. \nThe paper does not limit the methods' interpretability to mazes, maybe the evaluation should go beyond navigation tasks. (Maybe MinAtari (JAX implemented) or OCAtari ?). \nI think that training one network on the OCAtari version of Pong might to similarly interpretable networks. \nThe input space is 6 dimensional (x and y coordinates for each relevant objects).\nYou could identify if e.g. the constant x positions of the Player and the enemy only have nul weights (as they are constant and thus do not lead to any information gain). You might be even able to get an insight to the Pong misalignment problem with your technique (by analyzing that the module focusing on the enemy's vertical position is having high weights).\n\nNow, I know how work demanding it is to include yet another evaluation to the paper, but I think that it would be a great addition to the paper. \n\nI was hesitating between weak reject and weak accept, but if all my concerns are adressed (or for the additional experiments, that could take time, if the authors start them and will include it in the camera ready version), I am willing to increase my score to strong accept if my concerns (also the ones bellow) are addressed in the rebuttal, as I think that the paper is very strong and has a lot of potential.\n\nA more minor point: What do the communities stand for in e.g. Figures 5 and 6? What is the meaning of the colors? I would suggest to add a legend to the figures. Can you interpret the communities? (e.g. the red community is responsible for the vertical agent's movement)? This should go into the captions of the figures. \n\nFor the experimental results, I would suggest structuring the figures like this:\nThe first sentence should highlight the main message of the Figure/Table (e.g. \"*Fine-tuned Internal Louvain* allow for more isolated (thus interpretable) modules than the other approaches.\").\nThe next sentences then explain what is depicted in the Table/Figure. E.g. \"Communities are detected by the Louvain algorithm, and the colors represent the different communities. The *Internal* variant leads to denser communities, *Fine-tuned* leads to less communities, which is a sign of more interpretable modules. The top row depicts ... while the bottom row depicts ... .\"\nFinally, details and references to e.g. appendix can be provided if necessary. E.g. Results on more environmnets are provided in appendix X.\n\nThis would greatly improve the readability of the paper. I personally tend to read the abstract and the figures first, and then the rest of the paper. If the figures are well structured, I can get a good understanding of the paper without reading the whole paper (and thus decide if I potentially dive into it). I know that this is also the case for many other readers.", "relation_to_broader_scientific_literature": "This is the 3rd major opportunity for improvement.\nAs said, I would rewrite the experimental section to have a more structured approach to the evaluation. \nThus, you could add a related work section that discuss interpretability and sparsity in RL (could be merged with the dicussion). I tend to place background before the method (as the background is necessary to understand the method), and the related work (i.e. other approaches to the same problems: interpretability and sparsity) after the evaluation of your method, to avoid having the readers biased towards thinking of other potential approaches while presenting yours.\n\nYou are missing many of the latest published Related Work on Interpretable RL. I hereafter provide a list of papers that should be included in your related work section:\n\n* Delfosse et al. \"Interpretable concept bottlenecks to align reinforcement learning agents.\" NeurIPS (2024).\n\n* Luo et al. \"End-to-End Neuro-Symbolic Visual Reinforcement Learning with Language Explanations.\" ICML (2024).\n\n* Kohler et al. \"Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning.\" RLC workshop (2024).\n\n* Delfosse et al. \"Interpretable and explainable logical policies via neurally guided symbolic abstraction.\" NeurIPS (2024).\n\n* Marton, et al. \"SYMPOL: Symbolic Tree-Based On-Policy Reinforcement Learning.\" ICLR (2025).\n\n* Shindo, et al. \"BlendRL: A Framework for Merging Symbolic and Neural Policy Learning.\" ICLR (2025).", "essential_references_not_discussed": "While they are less essential, there are many published work on interpretable RL that should be discussed in RW.", "other_strengths_and_weaknesses": "I mostly checked appendix A, which gives a nice intuition on the log-based sparsity loss, and appendix D for the runtime results, but globally looked up everything.", "other_comments_or_suggestions": "The paper could benefit from a more detailed discussion on the benefit and the intuition behind the Neuron Position Optimization.\n\nI would bring algorithm 2 before the start of the experimental section and e.g. adjust the vertical space around the equations to have section 4 start on page 4. (This is very final formatting, but it would make this great paper even more appreciable). But the paper might heavily change, so this might not be accurate.", "questions_for_authors": "* You apply Neuron Position Optimization every T steps. Why do you need to apply it every T steps? Why not only at the end?\n* How does the empirical computational cost grow with the size of the problem? Is it feasible to apply it to larger scale problems?\n* Are you the first to propose a logarithmic sparsity loss? (This is quite an simple amazing idea, but I am not sure if it is novel).\n* Can you provide more details on why the log-based sparsity loss is preferable to the exponential-log one? I have read appendix A, but \n* Can you add scores in Figure 2 for the different methods on each task? Maybe next to $\\lambda_{cc}$ ?\n* What part of your algorithm is transferable to CNNs? (Neuron relocation? Louvain Algorithm?)\n* Are there some modules that are not easily interpretable? Would one need to apply a post-hoc interpretability method to understand them? This would not be a reason to reject, but would be great to understand the limits of your method.\n\n\nIntro could be shortened, directly go to the point (particularly if you create a *Related Work* section).\nl 39: \"to improving\" -> \"to improve\"\nl 10: \"which directly implicate on\" -> \"which directly impact\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sU0WoHpiu1", "reviewer": "Reviewer_W7cH", "summary": "This paper addresses the challenge of interpretability in reinforcement learning (RL) models by proposing a method based on functional modules, aiming to overcome the scalability limitations of traditional neuron-level interpretability approaches. The authors introduce spatially aware regularization and neuron relocation techniques to promote weight sparsity and locality, thereby inducing the formation of functional modules within RL policy networks. Additionally, they extend the Louvain algorithm by incorporating a novel 'correlation alignment' metric to detect these modules effectively. Experiments conducted in 2D and 3D Minigrid environments demonstrate the emergence of distinct modules corresponding to different directional navigation tasks, with targeted interventions on network parameters validating the functional roles of these modules.​", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed consideration of our work. We appreciate the acknowledgment of our robust experimentation and the insightful questions raised. \n\n**Figures and Captions.** We appreciate the advice on improving clarity. We have updated the network plots to include legends and have extended the captions to include sufficient detail for the core results to be understood from the figures and captions in isolation.\n\n**User Studies.** We agree that user studies can be valuable for assessing the utility of interpretability approaches. Although we agree with reviewer qymG that human evaluation is not necessary to support the claims in our work, future work leveraging our approach in specific applications would certainly be enhanced by context-specific user studies.\n\n**Function Validation: Complex Scenarios and Alternative Methods.** Our function validation approaches are not limited by the size or internal complexity of the modules. Our results instead show that the efficacy of ablation techniques in isolating module functionality are impacted by the level of connectivity between modules, which we regulate with the λcc parameter. We agree that activation-based intervention approaches may offer a valuable alternative, which we briefly discussion in the discussion section. Beyond this, we are excited about extending our approach to detect hierarchical modularity within complex tasks in future work.\n\n**Validation in Complex Scenarios.** We appreciate that Minigrid presents a relatively simple task set and that validation in more complex and real-world scenarios is a valuable direction for future work. To advance in this direction, we have extended our work to a non grid-world task, Pong, as proposed by reviewer qymG, and will incorporate the results into the camera ready paper.\n\n**Performance-Modularity Trade-Off.** We investigated a number of approaches to mitigating this, leading to the implementation of the λcc scheduling and the fine-tuning of the pruned modular networks (discussed in more detail in Appendix C), and the use of the log-based sparsity loss (discussed in Appendix A). We believe there may be further mitigation opportunities that merit testing in future work, for example by penalising inter-module connections and not intra-module connections in the later stages of training.", "claims_and_evidence": "The paper's primary claims are well-supported by empirical evidence. Notably, the authors demonstrate that increasing the strength of spatial regularization leads to more pronounced modular structures within the network. However, the functional validation of these modules relies on direct interventions in network weights, and the applicability of this approach in more complex or real-world scenarios remains to be fully established.", "methods_and_evaluation_criteria": "The proposed methods—including distance-based weight regularization, neuron relocation, the extended Louvain algorithm, and the correlation alignment metric—are clearly articulated and appropriate for the problem context. While the Minigrid environment serves as a suitable benchmark, its relative simplicity suggests that further validation in more complex tasks is necessary to assess the generalizability of the methods.​", "theoretical_claims": "The paper does not present formal theoretical proofs; thus, there are no theoretical claims to evaluate in this context.", "experimental_designs_or_analyses": "The experimental design is robust, featuring ablation studies that elucidate the contributions of individual components, such as regularization and neuron relocation. However, the functional analysis methods—specifically, negative saturation and negation—are innovative but lack comparisons with existing interpretability techniques, potentially limiting the assessment of their effectiveness.​", "relation_to_broader_scientific_literature": "The paper situates its contributions within the broader context of neural network interpretability research, particularly concerning modularity and hierarchical interpretability methods. It effectively references recent advancements in module detection and spatial regularization, highlighting its alignment with current research trends.", "essential_references_not_discussed": "The paper's citations are comprehensive, covering essential works related to RL interpretability and neural modularity.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Innovatively proposes a functionally interpretable framework for RL models based on neural modularity.​\n\n2. Provides detailed technical implementations and rigorous experimental designs, enhancing reproducibility.​\n\n3. Introduces novel metrics, such as correlation alignment, contributing valuable tools to the interpretability research community.​\n\nWeaknesses:\n\n1. The simplicity of the experimental environments may not fully capture the challenges present in more complex real-world tasks.​\n\n2. The reliance on direct weight interventions for functional validation may limit applicability across diverse neural architectures.", "other_comments_or_suggestions": "1. While the figures are clear, enhancing the legends and descriptions could improve reader comprehension, particularly regarding the directional aspects of network structures.​\n\n2. Conducting user studies to assess human understanding of the proposed modular interpretability approach could provide valuable insights into its practical utility.​", "questions_for_authors": "1. Applicability in Complex Scenarios: Can the negative saturation and parameter negation methods for module function validation be effectively applied to more complex, non-linear tasks? If so, what modifications would be necessary?​\n\n2. Alternative Validation Methods: Have you explored activation-based approaches as alternatives to weight modification for module function validation? If not, could such methods offer advantages in certain contexts?​\n\n3. Performance-Modularity Trade-off: Given the potential performance trade-offs associated with spatial regularization (λcc), have you investigated alternative strategies to mitigate performance loss while maintaining modularity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a mechanism to produce interpretable RL policies via network locality and sparsity, and an extension of a pre-existing Louvian algorithm. Although there are some concerns regarding the limited type of environments and incremental nature of the work, all reviewers support acceptance, and I am in agreement with them.

I would urge the authors to make sure they address the reviewer feedback for the final version, in particular with regards to the extra Pong experiments, as well as the clarity of the empirical presentation.

Nit: The title should probably be "Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning" (changing "Functionally" to "Functional").
Decision: Accept (poster)