What should a neuron aim for? Designing local objective functions based on information theory
Authors: Andreas Christian Schneider
Valentin Neuhaus
David Alexander Ehrlich
Abdullah Makkeh
Alexander S Ecker
Viola Priesemann
Michael Wibral
Venue: ICLR 2025 Oral (2025)
Keywords: local learning
interpretability
neuro-inspired
information theory
partial information decomposition
Primary area: interpretability and explainable AI
TL;DR: This paper proposes using Partial Information Decomposition as a local objective for neurons to improve neuron-level interpretability.

Abstract: In modern deep neural networks, the learning dynamics of individual neurons are often obscure, as the networks are trained via global optimization. Conversely, biological systems build on self-organized, local learning, achieving robustness and efficiency with limited global information. Here, we show how self-organization between individual artificial neurons can be achieved by designing abstract bio-inspired local learning goals. These goals are parameterized using a recent extension of information theory, Partial Information Decomposition (PID), which decomposes the information that a set of information sources holds about an outcome into unique, redundant and synergistic contributions. Our framework enables neurons to locally shape the integration of information from various input classes, i.e., feedforward, feedback, and lateral, by selecting which of the three inputs should contribute uniquely, redundantly or synergistically to the output. This selection is expressed as a weighted sum of PID terms, which, for a given problem, can be directly derived from intuitive reasoning or via numerical optimization, offering a window into understanding task-relevant local information processing. Achieving neuron-level interpretability while enabling strong performance using local learning, our work advances a principled information-theoretic foundation for local learning strategies.

Published as a conference paper at ICLR 2025
WHAT SHOULD A NEURON AIM FOR? DESIGNING LOCAL OBJECTIVE FUNCTIONS BASED ON INFORMATION THEORY
Andreas C. Schneideer1: Valentin Neuhaus David A. Ehrlich Abdullah Makkeh
Alexander S. Ecker4.2 Viola Priesemann Michael Wibral
Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen Max Planck Institute for Dynamics and Self-Organizatioon Göttingen
Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen "Institute Computer Science and Campus Institute Data Science, University Göttingen Jandreas. schneider, valentin. inlehauuu neuhaus, viola.priesaee wwwepppieeanneeeee @ds wmp..pg........ de (davidal exander. .errr.errliee ehrlich, michael. wibra @un -eanii--ee weweeuunntttttneee ingeggigettttnnnnnenenen de
ABSTRACT
In modern deep neural networks, the learning dynamics of individual neurons are often obscure, the networks trained via global optimization. Conversely, biological systems build on self-organized, local learning, achieving robustness and efficiency with limited global information. Here, show how self-organization between individual artificial neurons can be achieved by designing abstract bio- inspired local learning goals. These goals are parameterized using recent ex- tension of information theory, Partial Information Decomposition (PID), which decomposes the information that a of information sources holds about an out- come into unique, redundant and synergistic contributions. Our framework enables neurons to locally shape the integration of information from various input classes, i.e., feedforward, feedback, and lateral, by selecting which of the three inputs should contribute uniquely, redundantly or synergistically to the output. This se- lection is expressed can be directly derived from intuitive reasoning or via numerical optimization, offering a window into understanding task-relevant local information processing. Achieving neuron-level interpretability while enabling strong performance using local learning, our work advances a principled information-theoree foundation for local learning strategies.
weighted of PID terms, which, for given problem,
INTRODUCTION
Most artificial neural networks (ANNs) optimize single global objective function through algorithms like backpropagation, orchestrating parameters across all computational elements the network a unified computational structure. While this global objective approach has proven effective across a large variety of tasks, the role of the individual neuron often remains clusive, hindering the understanding of how local computation contributes global task performance.
In contrast, biological neural networks exhibit a markedly different approach to learning, relying strongly on self oraaaiiganizttion between neurons. Further, biological neurons must individually learn according to information that is locally available, because of being limited by physical constraints regarding locality and communication bandwidth. Interestingly, these limitations come with the advantage that computation of neuron can also be interpreted locally, at least in principle. Combining this local interpretability with the observation that biological neural networks achieve high levels of efficiency and robustness even in the absence of centrally coordinated objective raises the question: Could ANNs benefit from similar local learning mechanisms to enhance the local interpretability of their computations, while maintaining performance?
"Equally contributing first authors; 'Equally contributing last authors

---

Published as a conference paper at ICLR 2025
Neuroscience models have indeed shown that local learning rules (Hebb 1949 Dan and Poo 2004 Song et al. (2000) can solve a variety of tasks, such as sequence learning. unsupervised clustering, and classifcation tasks (Hebb 1949| Földiak 1990; Cramer et al. 2020). Often formulated on the basis of temporal coincidence (spike-timing, these very mechanistic rules do not offer direct insights into the actual information processing at each neuron. To facilitate such functional insight, more recently information theory has been used to formulate more abstract, but interpretable frameworks for local learning rules (e.g.. 2000 Wibral et al. 2017).
2000
1948) allows one to abstractly prescribe how much of the information from different imputs should be conveyed to the output of neuron (Kay) 22000) Moreover, it provides a general and flexible mathematical framework, which captures the information processing at an abstract and interpretable level free of details of the exact implementation. As such, information theory has been employed to define general optimization goals, including global objectives like cross-entropy loss (Goodfellow et al., 2016) and more localized ones such as local greedy infomax (Löwe et al.. 2019), and early attempts at passing only information that is coherent between multiple inputs (Kay and Phillips 201T).
Information theory (Shannon
Nevertheless, classical information theory falls short of describing how multiple sources work together to produce an output. In fact, it completely lacks the capacity to describe how the information from different sources integrated in redundant, synergistic unique ways in the output. These different ways, or information atoms, however, contribute very differently to a neuron's 's function: Redundant information quantifies only coherent information from multiple sources, for example to have neuron forward only information that is in agreement between sensory input and internal context information; while unique information reflects what can be obtained from single source but not from others, for example to make neurons not encode the same information that their peers already encode. Lastly, synergy deseribes the synthesis of new information from taking multiple sources into account simultaneously, for instance when comparing two signals and computing a difference or eтror signal.
This complexity of operations on input information from multiple elasses of sources can be quantified by using the recent framework of "Partial Information Decomposition (PID)" (Williams and Beer 2010; Gutknecht et al.] 202T, Makkeh et al., 202T). Given multiple inputs to an output, it can dissect the mutual information into different information atoms, which enumerate the different ways in which the sources can interact to produce the target, as explained above. Through this decomposition, one can paint comprehensive, yet interpretable picture of how the different input variables contribute to the local computation of an individual neuron.
PID has already been utilized to analyze the information representation and flows in DNNs (Ehrlich) et al. 2023; Tax et al. 2017). We show how this approach can be inverted by using PID to directly formulate goal functions the individual neuron level. These "infomorphic neurons" then have the ability to optimize for encoding specific parts of the information they receive from their inputs, allowing for an application to tasks from supervised, unsupervised and memory learning, as demon- strated explicitly in Makkeh et al. (2025). However, neurons with only two different inputs struggle to achieve good performance on classification tasks. seems that in general, three distinct signals are necessary for local neuron learning: A receptive signal providing the input, a relevance signal helping to filter this input and a distribution signal enabling seff-organization between neurons. In classical neural networks, these different classes of information are provided implicitly via the global gradient signals of backpropagation.
In this work, we show how ANNs based on infomorphic neurons with three input classes can be used to solve supervised classification tasks with a performance comparable to backpropagation, despite relying only on locally available information. We demonstrate how the parameters for the objective functions can either be determined from intuitive reasoning or be optimized by hyperparameter optimization techniques. The latter approach then sheds light onto the local information processing needed to solve a complex task. Through this approach, we demonstrate that local learning rules rooted in information theory can provide understanding of local ANN learning while maintaining their performance.
The main contributions of this work follows: Firstly, we use information theory, and in particular PID, to formulate interpretable, per-neuron goal functions with three input classes, significantly advancing the capabilities of the approach originally presented in Makkeh al. (2025). Secondly, we systematically optimize the goal function parameters for these "infomorphic" neurons, provide
2

---

Published as a conference paper at ICLR 2025
A apical dendrites
context/ feedback
B
X
contextual input
I(Y:C)
((YF.CC)
syn
unq.
(binary) output Y
axon (output)
WF
((YF))
H(Y)
feed-
X
forward
feedforward input
Fng
Hres
basal dendrites
11 red
D
H(Y)
((Y:F:C)
[(YFFIC) + I(YC)FF) Hras
E
G Geurron
goal
neuron
Classical
Inform.
Decomp.
H(Y)
ung.
Il
ayn
Hras
Partial Inform.
Decomp.
(PID)
output entropy
unique
unique
synergistic
active signal
between and
entropy
gradient exchange
Figure 1: Infomorphic neurons abstract, information-theorret neurons inspired by the structure of pyramidal neurons (Wibral et al., 2017). They are trained by adjusting their synaptic weights according to a PID-based goal Junction.A,B Inspired by the distinction be- tween apical and basal dendrites in cortical pyramidal neurons, infomorphic neurons are defined as computational units with separate feedforward (F) and contextual (C) input classes. C. Partial information decomposition (PID) allows one to dissect the total entropy of the neuron into explain- able components. D. PID enables one to distinguish how much information comes uniquely from either the feedforward F (Luugq.F) or contextual C input (Llunq,C) c) and how much they contribute redundantly (el, or synergistically (Ilyy)). Classic information theory (top) cannot disentangle these information atoms: The classic entities cover several of the atoms, so that effectively one can only measure redundant minus synergistic information. E. Formulating goal functions Gueuron in terms of -toom neuronn enables one to formulate how strongly redundant, unique, synergistic information should contribute to a neuron's output. Figure adapted from Makkeh et al. (2025).
intuitive interpretation, and thereby obtain insights into the local computational goals of typical classification tasks. Thirdly, for classification tasks we show that PID-based learning can achieve performance comparable backpropagation despite being limited local information only, while being interpretable on a per-neuron basis.
Next, we give a succinct account of infomorphic neurons with two input classes following Makkeh et al. (2025) before generalizing the framework to the more capable version with three input classes Section 3
BIVARIATE INFOMORPHIC NEURONS
Neurons can be viewed as information processors that receive a number of input signals and process them to produce their own output signal. From an iioommtioon-theortti perspective, the output Y of neuron can be considered random variable, and the total output information can be quantified using the Shannon entropy H(Y). From an information channel perspective, this output information consists of two parts: The mutual information I(Y: X) coming from the inputs X and the residual entropy (Y (X) arising from stochastic processes within the neuron.
Mutual information can be used to quantify the amount of information that is carried by different input classes about the neuron's 's output by considering the aggregated feedforward (F), contextual (C) and lateral inputs (L) as sources X individually. Nevertheless, classical information theory cannot quantify how much of the information is provided redundantly, uniquely or synergistically by different inputs, making it impossible to describe how exactly the information from two source
3

---

Published as a conference paper at ICLR 2025
variables contribute to creating the output. Using PID, the total mutual information I(Y F, F)C) between the output of the neuron Y and two aggregated inputs, e.g., F and C, can be dissected into four PID atoms (see Fig. [):: The unique information of the feedforward connection about the output Y. Fu P. is the information which can only be obtained from the feedforward and not the context input, with the unique information C of the context being defined analogously. The redundant information Ledd reflects the information which can equivalently obtained from either feedforward or contextual inputs about Y. while finally the synergistic information llayn can only be obtained from both inputs considered jointly. All classical mutual information terms between the target and subsets of source variables can be constructed from these PID atoms through the consistency equations (Williams and Beer 2010)
I(Y: : F) Iered Hung.F + -uung. + Hayns
I(Y: : F) = Ieedd + Hng.. and
I(Y: : C) Irred u.g..C
However, these atoms are underdetermined as there are four unknown atoms with only three consis- tency equations providing constraints. For this reason, an additional quantity needs to be defined, which is usually a measure for redundancy (Williams and Beer (010)). Such a definition for redun- dancy, based on the concept of shared exclusions probability space, given by the shared-exclusion PID, 1 (Makkeh et al. 2021) (see Appendix (A.2). Importantly, the resulting PID is differentiable with respect to the underlying probability mass function, which allows for a specification and subsequent optimization of learning goals based on PID.
Depending on the task the network is set to solve, different PID atoms become relevant to the information processing. In this work, focus the application to supervised learning tasks, in which the ground-truth label is provided as the context C during training. Here, the intuitive goal for the neuron is to foster redundancy between the feedforward and context inputs in its output, to capture only the feedforward signal that agrees with the label. Likewise, if the goal is unsupervised encoding of the input, lateral connections between neurons can be used the context signal and the goal might become maximizing the unique information of the feedforward input about the output, i.c., to capture only the information which not already encoded in other neurons (Makkeh et al., 2025).
PID can not only be used to describe the local information processing, but also to optimize it through the maximization of PID based objective function. Generally, such an objective function can be expressed as a linear combination of PID) atoms as
= Trea led + Tunq, Flng. + TTng,, cllunn.c + Jayn + Tres = >" III
(2)
where the residual entropy H HY FF, C), is included in the vector II for brevity of notation. As the 1 measure is differentiable, the goal function can be optimized by adjusting the weights of the incoming connections of the inputs F and C' using gradient ascent- with the analytic formulation of the gradients given in Makkeh et al. (2025), or using a suitable autograd algorithm.
While such infomorphic neurons can be used for supervised learning tasks when each neuron is provided part of the label to focus on (Makkeh et al. 2025). two input classes are insufficient for fully self-organizing the necessary local information processing. This is because if each neuron is tasked with encoding information that is redundant between the whole label and the feedforward signal with no regard to the other neurons, they will inevitably start encoding much of the same information, leading to poor performance. To avoid this, the neurons need to be made aware of what the other neurons encode, similar to the unsupervised example. This can be achieved by incorporating third input class for lateral connections between neurons of the same layer, as presented next.
3 TRIVARLATE INFOMORPHIC NEURONS
Including a third source variable results in a so-called trivariate PID which is more complex than the bivariate case, with atoms in general now describing redundancies between synergies. For example, the atom [[PPHC) denotes information that can be obtained redundantly from the feedforward and context signal but is unavailable from the lateral inputs, making it 'uniquely redundant' to and C. Likewise, the atom [I (C.r)) denotes information that can be obtained from F alone but also synergistically from the pair C, L (but not from C or alone). Overall, upon the introduction of a third source, the four PID atoms of the bivariate setup turn into total of 18 trivariate PIT)
4

---

Published as a conference paper at ICLR 2025
A
context/
B
C
I(Y:
feedback
(FO)FLLCC)
(FNCHL
context
(OLI
W/
feedforward
IJ
FOWEL
feed- forward
output
FY)
H(Y)
ICHL
lateral
M
lateral
Hres
Yed
D
V
ICY
encode information that is redundant between feedforward and context
'AND'
Yunq.F =1
encode information that is unique with respect laterally connected neurons
encode information redundant between feedforward and context, and unique with respect to laterally connected neurons
MIX
Figure 2: Adding lateral connections third input class enables neurons to self-organize to encode relevant and unique information. A,B. Infomorphic neuron with three inputs namely feed-forward (F),, contextual (C) and lateral (L) inputs. C. With three input classes, the number of PID-atoms increases to 18, represented by different colors, plus the residual entropy Hyes in the outer circle. Classical information -teeorrtittttoortttccc quantities such as the entropy HYY and mutual information with individual sources are depicted by ovals, indicating how they can be built from PID atoms. D. Three input classes allow for the optimization of more complex goals based on different terms (compared to only five in Fig. C). This trivariate PID) allows to combine two bivariate objective functions: a supervised learning task, the intuitive goal is to maximize information in the neuron's output that is redundant between the feedforward input F' and label C, while simultaneously ensuring the neuron's 's output stays unique with respect to lateral neurons L. While bivariate goal functions only allow for optimizing one of these objectives at time, both objectives be combined to the goal of maximizing the single atom II (C) in the trivariate case.
atoms (Gutknecht et al. 1 shown in Fig. 2 C. These PID atoms IL can in general be addressed by their antichains a, i.c. sets of sets of input variables, with the inner sets describing synergies and the outer redundancies between them (Gutknecht et al. (2221).
Including the residual entropy Hres = (Y|)F,CC C, L), the decomposition vector and goal parameter vector y thus have 19 elements each. The general goal function of neuron is again given by = 7 II and can be set and optimized the same way as the bivariate case:
G "UPhHHu +... (FFc))l( + Tree Y II.
Utilizing the expressiveness and interpretability of trivariate PID, heuristic goal functions can be derived for different classes of tasks from intuitive reasoning. For the supervised learning task to be solved here, the introduction of the lateral connections third input allows synthesize an intuitive goal function from the two bivariate goals introduced before (see Fig. D). As outlined in the bivariate case above, canonical goals might either be foster redundancy between the feedforward and context signals with no regards for what lateral neurons do, or, alternatively. to optimize for uniqueness of the output with respect to the lateral inputs, which does not differentiate between task relevance of information. With trivariate infomorphic neurons, however, these goals can simultaneously be achieved by optimizing for the atom [I(F) (C)+ which represents the redundancy of the feedforward and context signal which is at the same time unique with respect the lateral inputs. In simple terms, this enables the neuron to find coherent information between label and input data, that is not yet encoded by other neurons.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
We now explain how the general concept of trivariate infomorphic neurons can be practically utilized to construct networks solving a supervised classification task, i.e., the MNIST supervised handwritten digit classification task. To this end, we demonstrate how an infomorphic network with one hidden layer can be set up using either heuristic or optimized goal function parameters.
Neuron structure and activation function To construct infomorphic networks, number of practical decisions need to be made for how the output signal is constructed from the inputs in the forward pass. As first step, the aggregated to the single numbers& F. C. and L by a weighted sum. Subsequently, these aggregated inputs are passed into activation function, which can be chosen arbitrarily, but needs fit the requirements of the specific application. For the supervised classification task at hand, the function A(P, L) = F[(1 0j - 02) + 0i (B3 FC) + a2 (32 has been chosen, where o refers to the sigmoid function and 0 = ag = 0.1 and 31 = B2 = 2 are parameters that shape the influence of the input compartments on the activation function. This activation function builds on concepts of Kay (1944) and makes a clear distinction between the driving feedforward input and the modulatory context and lateral inputs, ensuring that the network performs similarly during training, when the context inputs is provided, and for evaluation, where it is withheld (i.e., by setting C' = 0). To show the independence of the infomorphic approach from particular activation function, the results additionally compared to a simple linear activation function A(F, C, L) F + 0.1L + 0.1C. The output of the activation function is finally mapped to the unit interval by a sigmoid function, whose output is used as a probability to stochastically output "1" (firing) or "-1" (non-firing). For the bivariate output layer, the sigmoid is not sampled from but just interpreted as a firing probability, with the neuron with the highest value being used as the network prediction. The lateral connections between neurons add recurrence to the network, requiring for the same input to be shown multiple times ensure convergence of the activations. In practice, however, we have seen that presenting the same input twice is sufficient to achieve proper network function (see Appendix A.9). The inference procedure for infomorphic networks is summarized in pseudocode in Appendix A.I
signals from the different input classes are
Discretization and local gradients To optimize a PID-based goal function during training. the first step is to estimate the PID atoms. Since the original T measure only works with discrete variables, the aggregated inputs from a batch of samples are first discretized to 20 levels each (see Appendix II. Subsequently, the empirical probability mass function is constructed from the batch, from which the PID atoms are computed by means of a plug-in estimation. Finally, the current value of the goal function can determined as a linear combination of PID atoms (Algorithm 2 and Algorithm 3 in Appendix I). For the local gradient computation, the gradient of the individual neuron's objective function is backpropagated via the output produce the neuron's weight updates.
The parameters y for a PID goal function can be obtained in two distinct ways-eithe either heuristically from intuitive notions about the nature of the computations necessary to solve the problem, as explained before, or optimized using a suitable hyperparameter optimization procedure. In this work, we contrast both approaches. For the hyperparameter optimization, we compared two established techniques: The Tree-structured Parzen Estimator (TPE, BBegsttr et al. 2011). that splits the sample points into two groups with high and low performance, fits model to each group and draws the next sample points according to the ratio of probabilities, and the Covariance Matrix Adaptation Evolution Strategy (CMA-ES, [aansnn (HAA-E,, 2016). which generates new evaluation points by sampling from a multivariate Gaussian distribution that was fitted to the best samples from previous iterations.
Choosing goal parameters
To solve the MNIST task (LeCun et al., 1998). we devised network architecture consisting of an input layer, a hidden layer made up from Nuad trivariate infomorphic neurons and an output layer consisting of 10 bivariate infomorphic neurons. The trivariate neurons in the hidden layer each receive the whole image as their feedforward signal, a context signal that guides their learning, and the output of other neurons of the same layer through lateral connections. The bivariate neurons in the output layer receive all outputs of the hidden layer and additionally exactly one element of the one-hot label as context, which is only provided during training, and are trained to maximize redundancy between their input and the information about the one label they observe. We explored three slightly different approaches to how the hidden layer can be set up using trivariate infomorphic neurons, illustrated in Fig. A. In setup 1, the fully connected one-hot label as
Network Setup

---

Published as a conference paper at ICLR 2025
A
setup
setup 2:
setup
types of inputs:
label & dense lateral
label & sparse lateral label
feedback & sparse lateral
label
label
driving
modulating
Nead
connection matrix:
input (image)
all-to-all
random sparse
one-to-one
differences
10
between setups
lateral*
lateral": no self- connection
B
te error for Nha= 100
bivariate
trivariate
C
0.20
test error
0.15
10
0.10
0.05
lateral heuristic
lateral optimized
context heuristic"
context optimized
setup heurisfic
sotup optimized
setup gotimized'
random projection
backpropagation
10
100
1000
number of neurons in hidden layer Nya
random projection
label dense lateral (setup 1)
label & sparse lateral (setup 2)
type of setup and goal
feedback sparse lateral (setup 3)
backpropagation
direct classification without hidden layer
Figure 3: Infomorphic networks with three input classes approach the performance a similar network trained with backpropagation and outperform networks with two input classes the MNIST handwritten digit classification task. A. In the hidden layer of the network, the infomorphic neurons receive feedforward and lateral connections as well either the ground-truth label or feedback from the output layer, outlined in three setups. B. Networks with trivariate infomorphic neurons outperform models with bivariate neurons with both, heuristic and optimized goals for hidden layer size of 100 neurons. Only the setup with feedback signal instead of the label performs worse for this layer size but outperforms the bivariate models for larger hidden layers as shown in Fig. 8 C. Infomorphic networks achieve similar performance the same network trained with backpropagation. For larger layers, using a sparse connectivity significantly improves performance (setup 2). The lines indicate mean values, with the intervals depicting the maximum and minimum of 10 runs. The goal function parameters have been optimized for networks with a hidden layers size of 100 neurons, indicated by the dashed line in C.
C' and the output of all neurons of the same layer as all-to-all connected lateral input L. In setup 2, sparse connectivity additionally to sparse L, the label is replaced by fully connected feedback from the output layer as context C, indicating a path towards how multiple hidden layers may be stacked the future. For comparison, trained benchmark network with the same connectivity setup I using standard backpropagation and cross-entropy loss. Additionally, we trained the output layer of a fixed random hidden layer setup using step-function activation.
maximum of 100 connections used instead all-to-all for L. setup
RESULTS
The performance of all major setups for Naia = 100 is shown in Fig. BB. The three trivariate infomorphic setups use the same set of optimized goal parameters (see below) for all hidden layer sizes and significantly outperform the random baseline well the networks with bivariate setup for the hidden layer. As shown in Appendix Fig. C, Setup matches the performance of backpropagation for up 100 neurons, and reaches its maximum test accuracy for 500 neurons before
Performance

---

Published as a conference paper at ICLR 2025
A
goal parameters
B
importance of goal parameters
heuristic
optimization for MNIST, 100 neurons (best)
(FONFL)
(F)C)
(FXC]L]
[FKL]
median test accuracy 92.8%
median test accuracy 94.9%
difference in validation accuracy by setting individual goal parameters to
goal parameter value
0
-1.12%
0.00%
1.12%
C 100
simplified goal
D
training validation test
optimized for CIFAR10, for 100 neurons
parameter importance
final error
(FXC)
10
median test accuracy 94.4%
(FHCHL)
original
iFicta
FOR TURPOR
(FC(FL)
difference in validation accuracy
(C
IPL
goal parameter value
successive ablation of parameters
-5.4%
0.0%
5.4%
Figure 4: Given set of optimized goal parameters, an ablation study allows for the identifica-
tion and interpretation of the most critical neural subgoals for given task. A. The heuristically defined goal function (see Fig. D) shows 92.8% test accuracy on MNIST for N = 100; opti- mizing the goal function using hyperparameter optimization increases the test accuracy 94.9%. The optimized goal parameters include the intuitive ((HHC)) but also additional PID atoms to be maximized or minimized the same time. B. For identifying the most important goal parameters, we performed an ablation study (individually setting parameters to 0) and measured the change validation accuracy compared to a network trained with the full goal. C. A successive ablation of parameters in order from lowest to highest individual effect identifies four parameters as being crucial for network performance (see Table 3 for their definition). The lines indicate mean values, with the intervals depicting the maximum and minimum of 10 runs. D. To test whether more complex image classification tasks require different goal parameters, we perform a separate hyperparameter optimization for setup 1, 100 neurons in CIFAR-10 and reach median test accuracy of 42.5% (compared to 42.2% using backprop and 41.1% 1% using the goal function optimized for MNIST).
its performance starts decreasing with larger layer sizes. This decrease in accuracy can be attributed to lack of convergence of the neurons (Appendix Fig. Fig.5). [). likely arising through the interaction of too many neurons. To alleviate the convergence problem, we performed additional experiments (setup 2) with the number of lateral connections reduced maximally 100, which leads to better convergence and contributes to the continuous increase performance for larger hidden layers, reaching a median test accuracy of 97.5% for 2000 neurons, slightly below the 98.0% reached via backpropagation. Finally, experiments with setup 3 provide evidence that the direct label input to the hidden layer can be replaced with feedback from the next layer, while still enabling solid performance especially for large layers. Additionally, we tested the alternate linear activation function as outlined in 4 Here, experiments using the optimized goal function parameters resulted in accuracies between 94.9 % and 95.1 % for 100 neurons, which is par with the original activation function.
Goal parameters The optimized goal function used for training the three main setups at all sizes were obtained by maximizing the validation set accuracy using setup with Naad = 100. We performed total of six hyperparameter optimizations of 1000 trials, three using TPE and three using CMA-ES samplers, respectively (see Appendix Fig, (). The best results were achieved using the parameters shown in Fig. 4 The optimized objective functions outperform the intuitive goal function
8

---

Published as a conference paper at ICLR 2025
by including additional PID atoms besides the intuitively derived II (F) (C)- To compare more complex image classification task, a single CMA-ES optimization run has been performed for the CIFAR- 10 task (Krizhevsky 2012). with the optimal parameters presented in Fig. D.
Parameter importance An analysis of the optimized hyperparameter values (Fig. 4 B) shows that only relatively few parameters are of high importance for performance. By setting goal parameters to zero individually (Figure 4.B) we find that there are four goal parameters that are critically non-zerr for high performance on MNIST. This finding is confirmed by cumulatively setting goal function parameters to zero in the order of the individual drop in performance, which is illustrated in Fig. - Interestingly, we find that CIFAR-10 two of the four critical goal parameters for MNIST seem especially important. As a complementary measure of parameter importance, the results of mean decrease in impurity (Breiman 2001) can be found in Appendix A.6 The results of more detailed analysis of parameter importance are presented in Appendix Fig. 9
Interpretation For the MNIST task, neurons aim for (i) encoding information that is redundant between feedforward and context inputs and thus task relevant, but not already encoded in other neurons (strongly positive 7(HHC)) (ii) avoid encoding information that is not in the context, thus not task-relevant, and already encoded by other neurons (strongly negative 7(FLL)) and (iii) allow redundancies of task relevant information between neurons (slightly positive (C)) Additionally, neurons avoid synergies that require F and either C or for their recovery (XFFC)(F)) (see Appendix Table 3 for an overview of the parameter values and interpretations). For the CIFAR-
10 task (see Fig. 4 DD) the two most important parameters (FHC)L) and T(FHC) point to a prioritization of redundancy over uniqueness between neurons.
6 RELATED WORKS
We are not aware of any other attempts to use PID directly constructive tool for designing ANNs with local learning objectives, aside from the bivariate implementation of infomorphic net- works (Makkeh al. 2025). The most directly related work using other local cnormatiionthhoretti learning objectives is likely the work on coherent infomax by Kay, Phillips and colleagues (Kay and Phillips 1997). Their idea of defining neural goal function that ensures only information coherent between the inputs being passed on in the output of a neuron inspired our idea of using PID based redundancy as a neural goal function. However, as analyzed in Wibral al. (017)) the lack of a PID) at the time of invention of coherent infomax reduced the expressiveness of their goal function, and also prevented an easily interpretable formulation for more than two input classes. Our results suggest that three input classes are crucial to unlock the potential of information theoretic learning. in the sense that a neuron needs to know at least the data to transform (F), the relevance of various aspects of these data (C), and also what is not yet encoded well by other neurons (via L).
Aside from PID proper, there are two prominent principles based on information theory that has been employed to train deep neural networks: information maximization (InfoMax, Linsker 1988) and information bottleneck (IB. Tishby et al. 2000). The main idea of InfoMax is to learn representations that encode as much information possible about the task, while IB additionally penalizes informa- tion that is irrelevant to the task YYu et al. 2020). Various frameworks have employed InfoMax and IB where unlike in our framework, information theory employed at the level of group of neurons (layer). Examples include Deep InfoMax where mutual information of high-level representations and the input is maximized (Hjelm al. 2019), InfoNCE that maximizes the mutual information of future observations and the current latent representation (Oord et al. 2018), and Variational autoencoders that employ an IB layer (Alemi et al. 2017).
Additionally, local learning rules based on concepts other than information theory have been developed and show potential for scaling to larger, more capable networks (Lillicrap et al. 2020 Richards et al.)
2019: Deon and Kim 2023] Bredenberg et al. 2024). This includes learning rules based on concepts from contrastive learning (lling et al. 2022 1: Ahamed et al., 2023), predictive coding (Mikulasch
ct al. 20233 Sacramento et al. [0018] Millidge et al. (2022). and many others (Launay et al. |020
Hinton 2022 Høier et 2023] Lee et al. 2015] Nøkland and Eidnes 2019 Lässig et al. 2023). Most of these approaches are confined to specific learning paradigms and implementations, while our framework describes local learning goals in an abstract manner, making it applicable to various learning paradigms, datasets and implementations, while still being interpretable.

---

Published as a conference paper at ICLR 2025
In terms of abstract per-neuron objectives, recently complementary work has been published, building on concepts from control theory (Moore et al.) 2024) and reinforcement learning (Ott) (020). These
efforts are in line with our work, and we believe similar creative, yet principled approaches could provide a basis for deepening the understanding of local information processing.
Beyond the formulation of information theoretic learning rules, there is literature using PID to analyze deep neural network function, such as Tax et al. (2017); EErlich et al. (2023); (Tokui and Sato (202T): Liang et al. (2023) 2024); MMhaamadi et al. (2023). These studies necessarily differ from the perspective on neural function taken here, as they analyze the function of neurons in feedforward networks trained with backprop, where- different to the infomorphic approach- the learning signals are not directly available as inputs to the neural activation function.
7 DISCUSSION
In this work we demonstrate how neurons can self-organize to solve classification tasks using local PID-based goal functions. Unlike mechanistic local learning rules or purely global objectives. our framework expresses learning in human interpretable terms. These local objectives are expressed by selecting which output information should be uniquely, redundantly or synergistically determined by the various classes of local inputs to neuron. We show how to derive a supervised learning objective based on intuitive heuristic reasoning, and upon some optimization of this simple rule demonstrate performance on MNIST that is comparable to the same ANN trained with backpropagation.
By showing which PID atoms need to be optimized to solve a certain goal and assessing the importance of each goal function parameter, infomorphic networks provide novel insight into the local information processing required at a local level to solve a global goal. Note that this interpretability differs from other notions of interpretability: While other attempts interpretability focus on how specific tasks are solved, infomorphic neurons show on an abstract level what kind of local information processing is necessary solve certain class of problems, e.g., classification. These insights may in the future be used to discern which neurons fail to contribute to the overall task, with the localizability of the 1s redundancy measure revealing for which labels problems occur.
Limitations and Outlook We show that the infomorphic local optimization approach reaches a performance comparable to backpropagation in setup with one fully connected hidden layer. In subsequent work, we will investigate the local information processing and connectivity required for deeper networks. Preliminary experiments show that deeper networks train out of the box using the same goal function parameters and reach the same or slightly higher accuracy compared to the single hidden layer models (see subsection A. More work is needed to determine how this accuracy can be improved, for instance by choosing different goal function parameters for the different layers.
The infomorphic neurons presented in this work use discrete version of PID, which limits the amount of information that individual neurons can convey. While this is similar to the binary output states of "spike" and "no spike" in biological neurons, PID can also be applied to continuous variables and thus could also enable learning in continuous neurons (Ehrlich et al. 2024).
For the study of biological neural networks, infomorphic networks could potentially serve powerful constructive modeling approach. While information-theerrtic gradients are almost certainly too complex to be directly implemented in biological substrate, biological neurons may still strive for similar objectives through simpler approximations and mechanistic local learning rules. Having a principled and general way to formulate, identify and test local information processing objectives could provide a new perspective on local learning in general. Additionally, subsequent work could aim to establish a formal bridge between local goals and known local learning rules, reconciling well-established Hebbian-like rules with novel principled approaches.
In conclusion, the formulation of local objective functions the language of PID directly enables an interpretation of the information processing of neurons. Our work represents an important step towards a principled theory of local learning founded in information theory.
Code Availability The framework and the code to reproduce the results of this work are available Networks
under http://ggibbbcoooiiccccaaacaoomp
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Marcel Graetz, Jonas Dehning, Mark Blümel, Fabian Mikulasch and Lucas Rudelt for their input and fruitful discussions about this topic. We would also like to thank Johannes Zierenberg. Sebastian Mohr, Darius Burr and the rest of the Priesemann and the Wibral Group for their valuable comments and feedback on this work. A.S., V.N., A.E. and V.P. were funded via the MBExC by the Deutsche Forsccuuggeeemennschaft (DFG, German Research Foundation) under Germany's Excellence Strateg--EXC 2067/1-39072994)) A.S., V.N., A.F., V.P., and M.W., were supported and funded by the DFG GRK2906 - project number 502807174, V.N. was partly supported by the Else Kröner Fresenius Foundation via the Else Kröner Fresenius Center for Optogenetic Therapies. D.E. and M. W. were supported by a funding from the Ministry for Science and Education of Lower Saxony and the Volkswagen Foundation through the "Niedersächsisches Vorab" under the program "Big project "Deep learning techniques for association studies of
Data in den Lebenswissenschaftr" transcriptome and systems dynamics tissue morphogenesis" A.M. and M.W. are employed the Campus Institute for Dynamics of Biological Networks (CIDBN) funded by the Volkswagenstiftung, A.E., V.P. and M.W. received funding from the DFG via the SFB 1528 "Cognition of Interaction" project-ID 454648639,. A.E. was supported from the European Research Council (ERC) under the European Union's Horizon Europe research and innovation programme (Grant agreement No. 101041669), M.W. was supported by the flagship science initiative of the European Commission's s Future and Emerging Technologies program under the Human Brain project, SEP--SP..-SSAN. T3.6.1.
REFERENCES
Donald Hebb. The organization of behavior., 1949.
Yang Dan and Mu-ming Poo. Spike timing- plasticity of neural circuits. Neuron, 44(1): 23-30, 2004.
Sen Song. Kenneth Miller, and Larry Abbott. Competitive hebbian learning through spike-timing dependent synaptic plasticity. Nature neuroscience, 3(9):919-926) 2000.
Peter Földiak. Forming sparse representations by local anti-hebbian learning. Biological cybernetics, 64(2):165-170, 1990.
Benjamin Cramer, David Stöckel, Markus Kreft, Michael Wibral, Johannes Schemmel, Karlheinz Meier, and Viola Priesemann. Control of criticality and computation in spiking neuromorphic networks with plasticity. Nature communications, 11(1):2853, 2020.
Jim Kay. Neural networks for unsupervised learning based on information theory, pages 25-63.
Oxford University Press, Inc., United States, 2000.
Michael Wibral, Viola Priesemann, Jim W Kay, Joseph T Lizier, and William A Phillips. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain and cognition, 1 112:25-38, 2017.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3) 1948.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Sindy Löwe, Peter O' O'Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient-isolated learning of representations, Advances in neural information processing systems, 32, 2019.
Jim W Kay and WA Phillips. Coherent infomax computational goal for neural systems. Bulletin of mathematical biology, 73(2):344-377 2011.
Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint /004::/004.0 155, 2010.
Aaron Gutknecht, Michael Wibral, and Abdullah Makkeh. Bits and pieces: Understanding information decomposition from whote relationships and formal logic. Proceedings of the Royal Society A, 477(225)):2020111 2021.
11

---

Published as a conference paper at ICLR 2025
Abdullah Makkeh, Aaron Gutknecht, and Michael Wibral. Introducing differentiable measure of pointwise shared information. Physical Review E. 103(3):03214, 2021.
David Alexander Ehrlich, Andreas Christian Schneider, Viola Priesemann, Michael Wibral, and Abdullah Makkeh. A measure of the complexity of neural representations based on partial information decomposition. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
URL https:://ppeneeviii http://iwiweeet//tteeeeeeeeoeeee htt.://oooummmnndTS333 httpp//eeeeeieeee.eeeoeooooooooa
Tycho Tax, Pedro AM Mediano, and Murray Shanahan. The partial information decomposition of generative neural network models. Entropy, 19(9):474, 2017.
Abdullah Makkeh, Marcel Gractz, Andreas C. Schneider, David A. Ehrlich, Viola Priesemann, and Michael Wibral. A general framework for interpretable neural learning based on local cnformatiionthhoreefe goal functions. Proceedings of the National Academy of Sciences, 122(10): e2408125122, 2025. doi: 10. (0.1073/pmas.2408 2408 125122. htL/httss:/ //wp::///wwwww/ www....................... org/doi/
Jim Kay. inoomatioontheorette neural networks for unsupervised learning: mathematical and statistical considerations. Technical Report 15733877499478662888, Scottish Agricultural Statistics Service, 1994.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In J. Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 24, Curran Asso- ciates, Inc., 2011. URL http:ttss///rrrceeiinngsssss wwwrips.rrccc.. wttp:ppp.ccppppppeeeeep saaigerrfflesseeeeeeeee wwles//pperrrrr//
Nikolaus Hansen. The ema evolution strategy: tutorial. arXiv preprint arXiv: /60400077 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):277-22324 1998.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Leo Breiman. Random forests. Machine learming, 45:5-32, 2001.
Jim Kay and William A Phillips. Activation functions, computational goals, and learning rules for local processors with contextual guidance. Neural Computation, 9(4)8995-110 1997.
Ralph Linsker. An application of the principle of maximum information preservation to linear systems. Advances in neural information processing systems, 1988.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004405, 2000.
Shujian Yu, Kristoffer Wickstrøm, Robert Jenssen, and Jose C Principe. Understanding convolutional neural networks with information theory: An initial exploration. IEEE transactions on neural networks and learning systems, 32(():4355444 442, 2020.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations, 2019. URL https
Karan Grewal, Phil Bachman, Adam
//openreview net
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1007.0077.00000 03748, 2018.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In International Conference on Learning Representations, 2017. URL https //openreview.oo www.eeewwwe/ffoommmeee foreee/fouummmie Hite:mmmiedd--yeeeeeeeee
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 21(6)333--346, 2020.
12

---

Published as a conference paper at ICLR 2025
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):161-1777 2019.
Ikhwan Jeon and Taegon Kim. Distinctive properties of biological neural networks and recent advances in bottom-up approaches toward better biologically plausible neural network. Frontiers in Computational Neuroscience, 17, 2023.
Colin Bredenberg, Ezekiel Williams, Cristina Savin, Blake Richards, and Guillaume Lajoie. Formal- izing locality for normative synaptic plasticity models. Advances in Neural Information Processing Systems, 36, 2024.
Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. Advances in Neural Information Processing Systems, 34:30365-30379, 2021.
Md Atik Ahamed, Jin Chen, and Ahdullah-Al-Zuuaee Imran. Forward-forward contrastive learning. arXiv preprint arXiv:2305.029277 2023.
Fabian Mikulasch, Lucas Rudelt, Michael Wibral, and Viola Priesemann. Where is the error? hierarchical predictive coding through dendritic error computation. Trends in Neurosciences, 46 (1):45-59, 2023.
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. Advances in neural information processing systems, 31, 2018.
Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Predictive coding approximates backprop along arbitrary computation graphs. Neural Computation, 34(6):1329-1366 2022.
Julien Launay, lacopo Poli, François Boniface, and Florent Krzakala. Direct feedback alignment scales to modern deep learning tasks and architectures. Advances in neural information processing systems, 33:9346-955 93660, 2020.
Geoffrey Hinton. The forward-forwate algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13335, 2022.
Rasmus Høier, D Staudt, and Christopher Zach. Dual propagation: Accelerating contrastive hebbian learning with dyadic neurons. In International Conference on Machine Learning, pages 13141- 13156. PMLR, 2023.
Dong-Hyur Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part 15, pages 498-515. Springer, 2015.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals, In International conference machine learning, pages 4839 4850. PMLR, 2019.
Francesco Lässig, Pau Vilimelis Aceituno, Martino Sorbaro, and Benjamin Grewe. Bio-inspired, task-free continual learning through activity regularization. Biological Cybernetics, 117(4):345- 361. 2023.
Jason J Moore, Alexander Genkin, Magnus Tournoy, Joshua L Pughe- Saghe-Sanford, Rob R de Ruyter van Steveninck, and Dmitri Chklovskii. The neuron as a direct data-driven controller. Proceedings of the National Academy of Sciences, 121 18933121, 2024.
Jordan Ott. Giving up control: Neurons as reinforcement learning agents, 2020. URL https
7/ www.ivorrgabba/aa
11642
Seiya Tokui and Issei Sato. Disentanglement analysis with partial information decomposition. arXiv preprint arXiv:2 108. /3753, 2021.
13

---

Published as a conference paper at ICLR 2025
Paul Pu Liang. Zihao Deng, Martin Ma, James Zou, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Factorized contrastive learning: Going beyond multi-view redundancy. arXiv preprint 1.Xiv:2306005268, 2023.
Paul Pu Liang. Chun Kai Ling, Yun Cheng, Alexander Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippe Morency, and Russ Salakhutdinov. Multimodal learning without labeled mul- timodal data: Guarantees and upplications. In The Twelfth International Conference on Learning Representations, 2024. URL
net 2rr
Salman Mohamadi, Gianfranco Doretto, and Donald A Adjeroh. More synergy, less redundancy: Exploiting joint mutual information for self-supervised learning. In 2023 IEEE International Conference Image Processing (ICIP), pages 3990-194, IEEE, 2023.
David Ehrlich, Kyle Schick-Polande Abdullah Makkeh, Felix Lanfermann, Patricia Wollstadt, and Michael Wibral. Partial information decomposition for continuous variables based on shared exclusions: Analytical formulation and estimation. Physical Review 110, 07 2024. doi:
10.1 10.1103/PhysRevE.. 11.00hsss....111.11111.1. 1000114115
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553)):36 444, 2015.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. PSEUDOCODE FOR TRAINING ALGORITHM (SETUP 1)
In this section, we provide pseudocode detailing the procedure to train infomorphic networks with dense lateral connections and label context (i.e., Setup 1 in Figure 3). The code is started from the main function TrainModel, which invokes TrainneuuroTTivariite and TrainneuuronBivariate to update hidden layer and output layer weights, respectively. These functions in turn make use of the auxiliary functions Computelsx Redundancies, which computes generalized redundancies according to analytical definition by Makkeh et al. (2021). and ComputePlDAtomm which performs Moebius inversion to compute the PID atoms from the generalized redundancies (Williams and Beer (010). Note that since the linear operation performed by ComputePIDAtoms is fixed, it can be efficiently implemented as a multiplication with a precomputed matrix.
Note that, originally proposed by Kay (1994) and utilized by Makkeh et al. 2025). ). gradients for the incoming weights are only computed with respect to the conditional probability p(ylf, c, (or p(yl,, c)) of the output Y, since the empirical probability
functions p(ʃ, e, in Algorithm and p(J.c) c) in Algorithm Jre not differentiable. While we currently have no strong priori argument why the empirical probability mass functions can be assumed constant, the learning of PID over training and the good task performance when using these gradients in experiments provides an ex post justification for this procedure.
Function 1: TrainModel
Input: data, num__epmpppppeeee _pochss
Output:
trained model
INITIALIZE model;
foreach epoch in range(num _emeese_pp_h_e_eeeee do
foreach (batch_ pamplessmmles, (aah__s in data do
INITIALIZE hidden_ hidden_state, stteee,,,,,,,, output_ state to zero;
hidden_stanl state <- Siwrarddhhdddenlleenff--b samples, labblss
1=hidden state);
hidden_sttan state forward hetpward_hddee... hww__hhddddyyyyee hiidden_layrrrf-bbtchhsaaa
sebatch_labell.
[hiiden_ssaate)
output_state- +- forward_output ouupppu_ayerrf-hiiddeensttteeee c=batch_labell
foreach neuron in hidden_layed do
TrainNeuron hrtivariaeel-siiddensstttteeeeee
f-batch_ samplees__ labels,,
Pllss_outputs(ottie neurons)):
foreach neuron in output_layer do
TruinNeuronBivaaatt(y---uu state[neuron) f=hidden state, c=batch_
12 return trained model
15

---

Published as a conference paper at ICLR 2025
Function 2: TrainNeuuroTTivariite
Input: y. f, c,
BIN continuous values f, c, 20 equally sized bins;
COUNT occurrences of tuples (f. c. in batch;
COMPUTE empirical probability masses p(f, D);
EVALUATE conditional probabilities p(ylf, c, 1) from the neurons;
CONSTRUCT full joint probability mass function p(y, f.f.)))))))))).))) = (ff.c..loo.f.c.l
isx_ _'redundanccies- <
f, c, I));
pid atoms + ComputteiDAtoommsix redundancics);
goal scalar scalar_product(gen product(goal params,
paramas pid_ _dtomss
PERFORM autograd to maximize goal;
UPDATE neuron weights;
Function 3: TrainneuuroBBivariate
Input: y. f,
BIN continuous values f, to 20 equally sized bins;
COUNT occurrences of tuples (f, c) in batch;
COMPUTE empirical probability masses p(f, c);
EVALUATE conditional probabilities p(ylf, c) from the neurons;
CONSTRUCT full joint probability mass function f.y, =(f.c)p(ulf.e
isx_ redundancies < Computelsxxeedunanciee///
f, c));
pid atoms + ComputePIDAtomsiiix reddddanceddd goal < scalar Bacollr_rrouut(ggee (gaal _paamm, _dd_atoms;;
PERFORM autograd to maximize goal;
UPDATE neuron weights;
Function 4: ComputeesxxRedundaanee
Input:
Joint probability mass function p(y, f, c, 1) foreach antichain a do
COMPUTE conditional probability mass functions P(Y y/ Vaca Niea S = si);
COMPUTE marginal probability mass function P(YY = y);
P(Y=v) Vacu Nica Si=ai),
(Y S.) Ey.fet PP(Y- y, F S.C = c, log,
P(Y
return T TPx(Y: Sa) for all antichains
Function 5: ComputePIDAtoms
Input: 1 (Y: SS,) for all antichains a
COMPUTE pid atoms 1Ig by inverting the linear system of equations 1 Sa) = =E I1g
where 3 x <> Va E a 3b E 3 C b;
return 11 for all antichains 3
16

---

Published as a conference paper at ICLR 2025
A.2 DEFINITION OF CHARRD-EXCLUSSE REDUNDANCY
In this section, briefly motivate and explain the shared-exclusion redundancy measure In intro- duced by Makkeh et al. (2021). The mutual information [ S2) between a target variable T and two source variables S1 and S2 be interpreted in Bayesian sense as average measure for how the prior belief of the target event T = t, needs to be updated light of the event of observing both source events S1 81 and S2 82 simultaneously:
P(T = 1|Si = 81 A S2 = 82)
I(T: S2) =
P(T t. S1 = S1 = s2) log2
P(T = t)
1,81,82
Makkeh et al. (2021) build on this logic and define redundancy as average measure for how the beliefs about the target event T need to updated if instead it is only known that 81 or S2 = have occurred:
P(T
||Si IP(T = 1)
82 82)
(T : S1, = Σ
= , 82) log2
1,81,82
For more than two source variables 8; (where an index enumerating the set of source variables), the term to condition on becomes disjunction between conjunctions of the form V Vaea Aiga S 8; for the redundancy associated with the antichain a. The atoms can then be computed from these redundancies via Moebius inversion, as laid out in detail in Gutknecht et al. (2021).
The definition is symmetric with respect to permutation of the sources, fulfills a target chain rule and differentiable with respect to the underlying probability distribution (Makkeh et al. 2021 which makes a suitable definition for optimizing objective functions.
A.3 COMPUTE RESOURCES
For the figures in this paper. we performed a total of seven hyperparameter optimizations: Three TPE optimizations and three CMA-ES optimizations of MNIST objective function parameter and single CMA-ES optimization for the CIFAR-10 10 task. Each of these optimizations took I 12 hours compute on HPC cluster node with 32 epu cores and two NVIDIA A100 GPUs with 40 GB of VRAM each.
Including evaluation and earlier computations not shown in the results, we estimate to have utilized a total of 200 hours of compute time on a compute node equivalent to the one described earlier.
A.4 IMPLEMENTATION DETAILS AND MODEL PARAMETERS
We implemented infomorphic networks a flexible and efficient python package using py- torch (Paszke et al. 2019) for automatic differentiation of the local goal functions. Furthermore, the optuna (Akiba al. (019) package was used to compute the TPE and CMA-ES hyperparameter optimizations as well as the computation of the parameter importance via the mean decrease in impurity.
In this paper, we use the MNIST (LeCun et al. 2015) and CIFAR- 10 (Krizhessyy 2012) datasets. The MNIST dataset consists 70,000 grayscale images handwritten digits, each sized 28x28 pixels. The dataset is split into training set of 60,000 samples and test set of 10,000 samples. The CIFAR-10 dataset consists of 60,000 images with three color channels and a resolution of 32x32 pixels. Of the 60,000 images, 50,000 are in the training set and 10,000 are in the test set. For each of our training runs with either dataset, 20% of the training set samples are withheld randomly to be used for validation.
The framework and the code to reproduce the results this work are available under htt.t/bbbb.oomm/iesemaannnnn Group/Infomorphic Neetooos
The parameters used to train the models shown in Fig, Blaee listed Table [ The backpropagation model was trained to optimize the cross-entropy loss between the prediction and the true label.
17

---

Published as a conference paper at ICLR 2025
Table 1: Model and training parameters for the models shown in Fig. 3
*The effect of different batch
sizes is analyzed in Fig. T0 **ehe effect of the number of bins is analyzed in Fig. 11
Parameter
Backprop Model
Hidden Layer
Output Layer
NEpochs
100
100
100
NBtch
1024
1024*
1024*
Optimizer
Adam
Adam
Adam
Learning rate '7
0.001
0.002
0.003
Weight decay
0.0
0.00035
0.00015
Number of bins per dim.
20
20**
Binning ranges
(-20,20)
adaptive
Objective function
cross-entropy
trivariate
7 (-0,
0.1, 10010000001110000110000000000011
A.5 MODEL. DYNAMICS DURING TRAINING
As measure for the dynamics of the neurons during training, we introduce the self-cosine distance of the feedforward receptive field as
D = w
w
w
F
where we corresponds to feedforward weights of single neuron of the hidden layer. The median value of _ for the hidden neurons of a trivariate model are shown in Fig. While the self-cosine distance consistently increases with layer size the dense setups, does not increase the sparse setup after layer size of 100 neurons, which is also the number of lateral connections in all of the larger sparse layers.
Dense
Sparse
Hidden Layer Size
Self Cosine Distance
10-1
10
20
50
10-3
100
200
500
1000
1
10
100
1
10
100
2000
Training time in epochs Training time in epochs
Figure 5: The median self-cosine distance of trivariate neurons for different layer sizes during the course of training in a dense (left) and a sparse (right) lateral connected setup.
A.6 OPTIMIZED OBJECTIVE FUNCTIONS FOR THE TRIVARIATE NEURONS
To support the trivariate objective function that was obtained using hyperparameter optimization, we repeated the optimizations three times for both, the TPE sampler and the CMA-ES sampler, with parameter values being limited to the interval 7/4 [-1, 1]. The resulting objective functions are illustrated in Fig. 6 While the objective functions that were optimized with the TPE sampler included all y parameters, the optimization with the CMA-ES sampler was only performed for the parameters that describe PID-atoms while the parameter for the residual entropy because we observed strong correlation between the parameter for the unique information of the feedforward input and the residual entropy which both had be small. In the figure, the respective
0. This done
18

---

Published as a conference paper at ICLR 2025
values are set O. In the figure, also illustrate the mean decrease impurity score which is measure for the importance a parameter.
TPE Sampler
CMA-ES Sampler
Parameter Value
Parameter Importance
Parameter Value
Parameter Importance
94.7 %
%
93.9 %
94.4 %
94.7 %
parameter value
mean decrease impurity
parameter value
decrease impurity
max
The three objective functions shown in the first column were obtained by utilizing the TPE sampler while the objective functions in the third column were obtained with the CMA-ES sampler. The performance illustrated below each of the objective functions corresponds the maximum validation accuracy during optimization. The column next to the objective functions shows the corresponding mean decrease impurity score of the corresponding goal parameter measure for the importance of the parameters. While the objective functions obtained with CMA-ES have higher maximum validation accuracy than the goals obtained with the TPE, we observed that the first objective function obtained with the TPE sampler (top-left) outperformed the others in terms of median validation accuracy for larger layer sizes, which why we used this objective function for the main results.
Figure 6:
A.7 TRIVARIATE CONSISTENCY EQUATIONS
For arbitrary numbers of sources, the PID consistency equations can be written as
T(Y a) Σna TT.
19

---

Published as a conference paper at ICLR 2025
In particular, the seven consistency equations for three input variables F, C and L can be written as
1(Y :) (r) + H[HoC + [rx4
+ [(HHCL)+++
+ 4(yyHy
:(:0) = [cc) + [FXC + [[H)) + [CHHFL) + [
I(Y:) L) [1)) +
(Y:F.O) = [() + + + + + + +((H))) + {H(y)
+ +[[[cyF + +
+
+ [[CHHFL) + [[CHCL)
+ (
I(Y P,L) [1 + [ + [FHL) + (CHH [FHCL
+ +[[cHE2y++ +luurry + [FCHFLyCLY + [[FCEEL [[LLHCL (6)
+ +[(())())0)) (I(r))
I(Y: 44cc + + + 4[HL) + [[GH [[HHLL
+ +[[())FF)) + +[(]]FF + ((FO((P))CC)) + [[Fo)(o + ((Fyyo
+ + +4cc
IY: 14, + Mcc +14 + H[ryC + +44EH + 44CHz)
+ [FH(L)) + [[C)FL)) + [FL) + [FFO(FL)(CL) + [[FO)CL))
+ [[FLHCL) + H[FCHrL) + (4rHCHH) + +l[rcy + +llp + [(FL)) + [[FL)))
A.8 THE ATOMS OF THE PARTIAL INFORMATION DECOMPOSITION FOR THREE SOURCE VARIABLES AND THEIR INTERPRETATION
To improve the clarity and intuition for the atoms of the partial information decomposition with three source variables, Fig. shows larger version of decomposition diagram already shown in Fig. C. As guide for the reader, the meaning of the different information atoms are listed in Table 2 Additionally, TableeJexplin lists the goal parameter values found via optimization together with their interpretation.
I(Y C)
(FXCHL)
(FC)FLJCLL)
(CL)
|FCHCL
[C)
(FL)
(CHFL)
:
FFIC)
H(Y)
CHL
[FRFFL]
(FL
(L)FFO)
(FO)
(L]
(FLICL)
[FCL)
Hres
:
Figure 7: Larger version of decomposition diagram already shown in Fig.
20

---

0.98 Published as a conference paper at ICLR 2025
Table A list of all atoms that are part of the Partial Information Decomposition with three source variables and their meaning.
Atom (ILtiichaa)))
Meaning
(())))))
information that is redundant in all of the three sources
[[))
redundant information between feedforward and context input redundant information between feedforward and lateral input redundant information between context and lateral input
[[F)4)
[[H)L)
[(F))L))
information provided by both, the feedforward input and the synergy between the context and the lateral input
information provided by both, the context input and the synergy between the feedforward and the lateral input information provided by both, the lateral input and the synergy between the feedforward and the context input
(FF)
unique information provided by the feedforward input unique information provided by the context input
unique information provided by the lateral input
(FFC((F))(C))
information redundantly provided by each of the pairwise synergies
information provided by both, the synergy between the feedforward and the context input and the synergy between the feedforward and the lateral input
[[[CHCL)
information provided by both, the synergy between the feedforward and the context input and the synergy between the context and the lateral input
[[LLCL)
information provided by both, the synergy between the feedforward and the lateral input and the synergy between the context and the lateral input
Urfy (rr
synergy between the feedforward input and the context input synergy between the feedforward input and the lateral input
[(CL)
synergy between the context input and the lateral input
11FFCL)
information encoded by all input sources synergistically
Table The four most important non-zero goal parameters found through optimization on MNIST and ablation lead to interpretable requirements on each neuron's 's local information processing. Note that the optimization was performed with parameter values being limited to the interval 7: E [-1, 1].
parameter
definition
value
interpretation of local information processing goal
((FHC)
redundancy between and
maximize information that is provided both by feedforward AND context but not by other neurons-ann and thus relevant for the task
((F(L)
redundancy between F and L
-0.99
minimize information that is redundant with other neurons, but NOT provided by the context- and thus not relevant for the task
((HCCH)
redundancy between F. C' and L
0.33
moderately maximize information that is redun- dant with other neurons AND the context-thus relevant, but already encoded
([FC(FL)
redundancy between synergies requiring F'
-0.97
minimize synergistic information that requires F and either C or L to be recovered
21

---

Published as a conference paper at ICLR 2025
Heuristic Models
Baseline Models
Optimized Models
Lateral Heuristic
Random Projection
Lateral Optimized
Context Heuristic
Backpropagation
Context Optimized
Trivariate Heuristic
Readout Only no hidden layer
Trivariate Optimized
Trivariate Optimized Sparse
Trivariate with Feedback
Comparison of Heuristic Models
Comparison of Optimized Models
10a
10a
Training Error
10-1
10
10-2
10
10
100
1000
10
100
1000
10a
10a
Test Error
10
10
10
100
1000
10
100
1000
Number of hidden neurons
Number of hidden neurons
Figure 8: An overview over the training and test performances of the heuristic (left) and optimized (right) models with different model structures, neuron models and objective functions well as the performances of some baseline models. In the bivariate models, only one of the non driving inputs (context or lateral) and the neurons optimized the heuristic objective functions illustrated in Fig. D. The corresponding optimized objectives were optimized with the CMA-ES sampler similar to the trivariate objective function.
A.9 RECURRENCE OF THE HIDDEN LAYER
To perform an information decomposition which includes the information of the lateral activity. we include lateral connections within the hidden layer as illustrated in Fig. A. As described in Appendix I we only perform two iterations within the hidden layer one iteration to compute the activity of the neurons within the layer and second iteration to include this activity in the final output the neurons. Therefore, the lateral activity contributes to the neurons output. Due to the stochasticity of the neurons, the lateral input will never fully converge. The continuous output probabilities converge already after the second single iteration with only minor changes the output probabilities of the order of 10-4, 4 This is due the small influence of the lateral input on the output which is limited to 0.1 (2FF.. i.e., a maximum change of +0011 in the activation.
A. DEEP INFOMORPHIC NETWORKS
major factor in the success in many neural network architectures is their layered structure, allowing the network to process the relevant information in successive steps to make them more accessible to the final output layer. For this reason, scaling infomorphic networks to multiple hidden layers the canonical next step for making them applicable to more complex tasks.
In preliminary experiments we found that networks with more hidden layers work out-of-the-box (without any further optimization of the goal other hyperparameters) and perform equally well or slightly better than comparable shallower networks. We provide two examples: First, networks with layer sizes 800-200-11 10 and 600-200-200-11 (each layer receiving the label context) reach a mean final test accuracy of 97.2 % and 97.1 % on MNIST. This matches the performance of architecture 1000-10 10 (setup 2) with slightly smaller model complexity (same number of neurons,
22

---

Published as a conference paper at ICLR 2025
0.930
0.935
0.940
0.945
0.950
0.955
(FXC)L)
Modification
0
(F)C)
Reduced by 10%
Increased by 10%
Reduced by
(FXL)
increased by 0.1
(CXL)
(FXCL)
(C)FLL
(L)FC)
(F)
Mocilied Gamma
(L)
(FC)FL)CL)
(FCXFL)
(FC)CL)
(FU)CL)
(FC)
(FL)
(CL)
(FCL)
H_res
0.930
0.935
0,940
0.945
0.950
0.955
Validation Accuracy
Figure 9: To estimate the importance of the different parameters of the trivariate objective function, we modified individual parameters of the trivariate objective function. The performance change induced by setting the parameters to was already shown Fig. 4 and was used for the results shown in Fig. C. Additionally, we changed the individual goal parameter relative to the parameter strength (=: 10)), but also an absolute manner (#:0.1) to get a better intuition about the importance of the fine tuning of the parameters.
fewer parameters). Second, an architecture of 1600-400-10 (each layer receiving label and feedback from the output layer context) reaches mean final accuracy 97.7 %, outperforming the performance of 2000-10 (setup 2) with 97.5 %. These results provide evidence that stacking multiple layers works in principle.
In subsequent work, we will optimize the goal function parameters for feedback setup to evaluate the efficacy of this approach and uncover differences to the setup with the label context signal. Furthermore, will conduct investigation into how the optimal goal parameters differ between individual hidden layers of deeper network.
As shown in figure D now created the goal function for the trivariate neurons by maximizing the atom that corresponds to the overlap the two bivariate atoms. This atom corresponds to the redundant information between the feedforward and the contextual input, that is uniquely provided by the two sources and not by the lateral input, resulting in the heuristic goal function of the form Guiv Tl FHC)- It is important note, that even if this goal looks identical to the bivariate goal that
23

---

Published as a conference paper at ICLR 2025
maximizes the redundant information between F and C, it is different as the meaning of the atoms depends on the number of sources despite the identical notation.
A.11
BATCH SIZE AND NUMBER OF BINS
Final Validation Error in de
5.4
10
Batch size
Validation Error
1024
2048
5.0
4096
4.8
10-1
4.6
4.4
1024
2048
4096
10
100
1000
Batch size
Epoch
Figure 10: To evaluate whether batch size of 1024 is sufficient for estimating the probability distributions required information decomposition, we conducted additional experiments with the trivariate model. These experiments utilized the optimized goal function, and hidden layer size of Niid = 100, while varying the batch size. Each configuration was run in different initializations and trained for 1000 epochs to ensure convergence, demonstrated in the right figure. The left figure reveals that performance remains consistent across different batch sizes. However, models with smaller batch sizes converge faster, reducing the computational cost of optimization. Consequently, batch size of 1024 appears adequate for the requirements of this study.
10"
Final Validation Error in
Number of Bins
7
Validation Error
4
6
16
10-1
32
2
16 Number of Bins
4
32
10
100
1000
Epoch
Figure 11: Building on our analysis of the dependence of training on batch size, we conducted similar experiments, this time varying the number of bins used estimate probability distributions. Theoretically, using too few bins results in greater information loss, as more distinct samples are grouped into the same bin. Conversely, too many bins make it challenging to reliably estimate information-theone thhorrttcc quantities. As shown in the figure, a high number of bins accelerates con- vergence, while too few bins introduce noise into the training process. Consistent with theoretical expectations, the optimal number of bins falls within a balanced range.
24


Reviews:
[{"review_id": "EWApOHqsda", "reviewer": "Reviewer_SbrR", "summary": "The authors claim that modern deep neural networks focus on global optimization and the learning objective for individual neurons are obscure, whereas real-life biological neurons grow with local learning objectives with limited global information. Based on this intuition, the authors propose a biologically inspired neural network that enhances interpretability of the learning dynamics of individual neurons. The authors leverage Partial Information Decomposition (PID) in information theory to formulate objective functions on the individual neuron level, and these neurons are termed “infomorphic neurons”. This novel formulation is able to achieve comparable performance to backpropagation on MNIST and CIFAR10 datasets, showing preliminary signs of utility. Besides, the proposed framework allows for interpretability analysis on the partial information components which is quite unique and interesting.", "questions": "1.\tPlease refer to Weakness 2. Would the authors consider the “less existing” representation of the quantitative results or provide some other alternative that are similarly straightforward?\n2.\tI still have a hard time understanding how the partial information components are realized in actual implementation. It would be good for the authors to provide a brief explanation on how that is done or point to the relevant text in the paper.\n3.\tIt is a bit uncommon to have “Related Works” and “Limitations and Outlook” as part of the Discussion section. Is that intentional or is it just a typo?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThe topic of biologically inspired neural networks has attracted great attention in the recent years. Proposing alternative solutions to the long-dominating backpropagation method is also interesting and profoundly influential. I encourage the authors to work along these lines.\n2.\tOverall this is a paper very rich in content.\n3.\tThe visualizations in Figure 1 and 2 are very helpful for understanding the partial information components in partial information decomposition.\n4.\tThe performance on MNIST and CIFAR10 are quite promising. From what I read, the proposed method achieved beyond 98% accuracy on MNIST and 94.4% accuracy on CIFAR10.\n5.\tThe parameter importance assessment is thoughtful and informative.", "weaknesses": "1.\tWhile I believe we should not expect super comprehensive experiments from a paper that initially introduce a novel concept or paradigm, it might be even improve the soundness of this paper if the authors can also include the performance on slightly more challenging datasets. If standard large datasets such as ImageNet are too computationally expensive, the authors could consider variants that are decently big and challenging, such as TinyImageNet.\n2.\tFor demonstration of experimental results, while I like the richness of Figure 3B and I do notice Figure 8 in the appendix, I would personally argue it would be more straightforward to include a “less exiting” bar and whiskers plot of backprop as well as different variants of bivariate and trivariate models. I suspect the authors left the performance of the bivariate model to the appendix to avoid overcrowding the results, but this omission from the main text might trigger confusion from the audience. Again, a good old bar and whisker might be a valid solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8i7xkCAP6K", "reviewer": "Reviewer_Mc5F", "summary": "Based on the concept of partial information decomposition in the information theory, this work formulates a learning framework that allows the implementation of per-neuron goal functions and neuronwise interpretability. In an ANN setup for classification tasks, the goal function is optimized via parameters of the ‘informorphic’ neurons, with neuron-level interpretation, good performance (comparable to the same ANN trained with bp) and insights into the local computational goals. Examples with bivariate and trivariate informorphic neurons are demonstrated where the three-input classes unlock the potential of information theoretic learning, validating the abovementioned advantages in comparison with classical information composition.", "questions": "How do we understand the performance presented in Fig. 3B by considering that at Nhid = 100, there is a convergence for all results? Why a sparse lateral outperform a dense lateral under large Nhid conditions? What are the computational and memory costs?\nWhat does the extraction of high-effect parameters mean, and would this be useful to construct networks with higher performance, or understand the nature of the problems under training?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The use of partial information decomposition in a learning framework with bivariate and trivariate implementations enables interpretable information processing at the per-neuron level, which is also new. The discussion based on a comparison between heuristic and optimization approaches demonstrates the potential of the interpretability of local learning framework in a task-relevant context and without the loss of performance compared to ANN trained with bp.", "weaknesses": "The title ‘What should a neuron aim for’ is broader than the scope explored in the current work, where a single-layer bivariate and trivariate local learning framework is studied. There is still a large gap between the interpretability of the model here and that of the neuron. The advantage of interpretability from an application perspective is not demonstrated or discussed, which is expected to be very limited by the single-layer simplicity here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "22DMsRVp8Z", "reviewer": "Reviewer_dmEh", "summary": "The paper presents a local learning framework for neural networks by introducing a local training objective inspired by partial information decomposition (PID) in information theory.\nIt trains each neuron individually by a local goal function that is a linear combination of PID atoms, which decomposes the information that three input signals \n(feedforward, context, and lateral) provide about the neuron's output. The weights of the combination can be chosen heuristically or optimized on validation data. The weights represent \nhow information in the inputs contribute to the neuron's output and are human-interpretable. Experiments on a single-hidden-layer network show that the proposed learning method \ncan achieve similar performance to conventional training by cross-entropy loss, yet providing interpretable learning process of individual neurons.", "questions": "Questions related to numerical computation of PID atoms:\n- To compute the PID atoms during training, the authors empirically evaluated the joint probability mass function of the aggregated inputs and output of each neuron. Since the input is high-dimensional, is a large batch size \nneeded for such numerical estimation to be accurate and stable? For example, the input dimension is at least 28^2 = 784 for MNIST, and each dimension is discretized to 20 levels. Is the batch size of 1024 sufficient?\n\n- The authors mentioned that the shared-exclusion redundancy and thus the PID atoms are differentiable with respect to the probability distribution. However, in my understanding, the empirical probability mass function \nof a discrete random variable is not differentiable w.r.t. its samples. Then the goal function will not be differentiable w.r.t. a neuron's weights, which is needed for training. Could the authors clarify this point?\nIn particular, how is the empirical probability mass function differentiated w.r.t. the output (and subsequently the weights) of the neuron?\n\nThere seems to be a gap between the claimed \"neuron-level interpretability\" and the proposed learning framework. \nUsually, interpretability means understanding how a trained network makes predictions and what the network has learned.\nHowever, the method proposed in this paper only provides interpretation for the learning objective, rather than insight of what the model and each neuron has learned. \nFurthermore, the interpretability is still global, not neuron-level, as the goal parameters are shared across all neurons, i.e., the interpretation for all neurons are the same. \nFinally, the cross-entropy loss and the backpropagation process are quite human interpretable, in my opinion.\nCould the authors comment on this?\n\n\"For the supervised classification task at hand, the function ... has been chosen ... This ensures that the network performs similarly during training, when context and lateral inputs are provided, and for evaluation, \nwhere the context signal is withheld.\"\nIt seems that the given activation function is for training, since it uses the context (label) as input. Can the authors define clearly what the activation function during testing is?\n\n\"One promising path towards constructing deeper networks is using stacked hidden layers that receive feedback from the next layer, similar to setup 3.\"\nIt is a bit unconvincing to replace label feedback by feedback connection from the next layer. In the latter, the neuron's learning goal is to capture the part of the feedforward signal that agrees with the next layer's output, which is less intuitive than \ncapturing the part that agrees with the label.\n\nFigure 8, comparision with bivariate model: Since the goal is to compare trivariate with bivariate, it might be better to put trivariate and bivariate on the same axis. If it'd be too crowded, \nmaybe group the results by \"Heuristic\"/\"Optimized\".\n\nIn \"Goal parameters\" paragraph in Experiments, it might be better to state that \"heuristic goal function\" is $\\Pi_{ \\{ F\\} \\{C\\}}$. Although it can be inferred from Fig. 4, it's better to define it in the text as well.\n\nIn Fig. 4, is the difference in validation accuracy defined as \"after setting to 0 - before\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using information-theoretic loss functions to train individual neurons is interesting, with nice connection to biological neural networks and neuroscience. \n\nThe novelty of adding a third input class representing lateral connections between neurons is intuitive and interesting and demonstrates better empirical performance than the bivariate model.\n\nThe overall method and motivation are clearly presented.\n\nThe experiments are well designed. Nice ablation study on the goal parameters. Good interpretation of \"important\" PID atoms identified by goal parameter optimization.", "weaknesses": "Some parts of the methodology are not clear. Please see questions below.\n\nThe PID-based goal functions and infomorphic neurons were originally proposed by Makkeh et al. 2023, and the contribution of this work is to introduce lateral connections as a third input class.\nHowever, in the Introduction, the authors claim the PID goal function to be one of the main contributions of this paper. The authors should explain more clearly the difference between this work and Makkeh et al. 2023, \nand define their contributions more accurately.\n\nExperiments were performed on neural nets with single hidden layer, which limits the scope of the paper. It is unclear whether the observations and insights can generalize to deeper neural networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g9wLOef7rU", "reviewer": "Reviewer_wfkU", "summary": "This paper introduces trivariate infomorphic neuron model to develop an interpretable and biologically-inspired local objective function for training artificial neural networks. This abstract neuron model is based on Partial Information Decomposition (PID) framework, which decomposes information into unique, redundant, and synergistic components (atoms). Local objective (goal) function is constructed as a linear combination of these PID information atoms. Neural network models with one hidden layer and various configurations (e.g. sparse lateral connectivity, different context signal) are trained with the proposed loss function on MNIST and CIFAR10 classification tasks, demonstrating promising results.", "questions": "* Is the activation function in line 283 commonly used? How the $\\alpha_i$ and $\\beta_i$ (for $i = 1, 2$) values are determined for this activation function?\n\n* Regarding lines 293-297, which mention that lateral connections introduce recurrence, how did you determine that presenting the same input twice is sufficient? For more complex datasets, more iterations may be needed. I am curious if the output converges after two presentations.\n\n* Is there any particular heuristic or rationale for using 20 equally sized bins in the algorithms in Appendix A.1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The problem is well-stated in the introduction.\n\n* Section 3 and Section 4 effectively introduce the PID framework and explain how it is used to develop local objective functions for training neural networks.\n\n* The shared code is well-structured, which aids the reproducibility of the work.\n\n* MNIST results seem promising.", "weaknesses": "The paper has a few areas that could benefit from further clarification or expansion. Please see the following points and the Questions section for more details.\n\n*  If I did not overlook it, the consistency equations for the trivariate infomorphic neuron model are not included in the paper.\n\n* The discussion of weight updates is limited. It is mentioned that either autograd or the analytical formulation in (Makkeh et al. 2023) can be used. However, the analytical approach in (Makkeh et al. 2023) applies to  bivariate informorphic neuron model. Although the weight update derivation is potentially a similar approach, the absence of an explicit derivation raises a concern. Without this, I am uncertain if the resulting learning is biologically plausible.\n\n* It is mentioned that (line 308) the $\\gamma$ parameters can be derived from intuitive notions. However, unless I overlooked something, the explanation for determining these parameters is not provided in detail beyond a brief note in the caption of Figure 2.D.\n\n* I am concerned that plug-in estimation of the information atoms based on empirical probability mass functions may face challenges in high-dimensional settings due to the difficulty of density estimation. While MNIST may be manageable despite being high-dimensional, this could be an issue in more complex settings.\n\n* Lines 511-515 mention that deeper networks were trained and achieved comparable or better accuracy results; however, these results are not shown. The paper only includes experiments with a single hidden layer.\n\n* The algorithm in Function 2 (Appendix A.1) requires quantities labeled 'isx_redundancies' and 'pid_atoms' (lines 732 and 733), but it is not specified how these quantities are computed.\n\n* According to Appendix A.3, the proposed method requires a lot of computational power even for MNIST. This raises concerns about its scalability to more complex datasets.\n\n* I think the abbreviation IM in Table 1 likely stands for \"infomorphic networks,\" but this is not clarified in the paper.\n\n**Minor Comments:**\n\n* In line 907, the word 'function' is repeated.\n\n* I think Figure 9 is neither referenced in the text nor is it fully interpreted beyond the caption. Given its complexity, more discussion could make it easier to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)