A Measure of the Complexity of Neural Representations based on Partial Information Decomposition
Authors: David Alexander Ehrlich
Andreas Christian Schneider
Viola Priesemann
Michael Wibral
Abdullah Makkeh
Venue: Accepted by TMLR ()

Abstract: In neural networks, task-relevant information is represented jointly by groups of neurons. However, the specific way in which this mutual information about the classification label is distributed among the individual neurons is not well understood: While parts of it may only be obtainable from specific single neurons, other parts are carried redundantly or synergistically by multiple neurons. We show how Partial Information Decomposition (PID), a recent extension of information theory, can disentangle these different contributions. From this, we introduce the measure of ``Representational Complexity'', which quantifies the difficulty of accessing information spread across multiple neurons. We show how this complexity is directly computable for smaller layers. For larger layers, we propose subsampling and coarse-graining procedures and prove corresponding bounds on the latter. Empirically, for quantized deep neural networks solving the MNIST and CIFAR10 tasks, we observe that representational complexity decreases both through successive hidden layers and over training, and compare the results to related measures. Overall, we propose representational complexity as a principled and interpretable summary statistic for analyzing the structure and evolution of neural representations and complex systems in general.

Published in Transactions on Machine Learning Research (05/2023)
A Measure of the Complexity of Neural Representations based on Partial Information Decomposition
David A. Ehrlich 12,+
davidalerander. http:/deceaddererriceeeeeeeeeeee de undrens.schmeiderreee map..
Andreas C. Schneider
Viola Priesemann*.4
Michael Wibral
Abdullah Makkeh
Goettingen Campus Institute for Dynammics of Biological Networks, University of Goetlingen, 37077 Goettingen, Germany
Man Planck Institute for Dynamics and Self- Deff-Orgenization, 37077 Goettingen, Germany
Institute for the Dynamics of Complen Systems, University of Goettingen, 37077 Goettingen, Germany Campus Instātute Data Science, University of Goettingen, 37077 Goettingen, Germany
Reviewed on OpenReview: https: // openreveew..ee.... net/ forum? id- R8TU3pf22F
Abstract
In neural networks, lask-relevant information is represented jointly by groups of neurons. However, the specific way in which this mutual information about the classification label is distributed among the individual neurons is not well understood: While parts of it may only be obtainable from specific single neurons, other parts are carried redundantly or synergistically by multiple neurons. We show how Partial Information Decomposition (PID), a recent extension of information theory, can disentangle these different contributions. From this, we introduce the measure of "Representational Complexity", which quantifies the difficulty of accessing information spread across multiple neurons. We show how this complexity is directly computable for smaller layers. For larger layers, propose subsampling and coarse-graining procedures and prove corresponding bounds on the latter. Empirically, for quantized deep neural networks solving the MNIST and CIFAR10 tasks, we observe that representational complexity decreases both through successive hidden layers and over training, and compare the results to related measures. Overall, we propose representational complexity as a principled and interpretable summary statistic for analyzing the structure and evolution of neural representations and complex systems in general.
*These authors contributed equally this work.

---

Published in Transactions on Machine Learning Research (05/2023)
I(T: S1, S2, Ss)
(123)
T(T: S1)
{12)
(13)
(12)
I[T: :Sa)
(T ..2)
(1)
(1H23)
(1)33)
(1)(2)
(12233)
127
(2)3))
(2)
12)/22)
*
I(T S2, Ss)
: Ss)
I(T S1, S2)
2
* (1,2331,,)
M 3
Figure 1: Mereological diagram illustrating how the PID atoms IT(T: S.,) constitute all classical mutual information quantities involving three source variables, introduced by WWilliams & Beer (2010). Classical mutual information terms are represented by black ovals with general redundancies given by their overlaps. The PID atoms correspond to the contiguous areas of the overlaps and are labeled by their corresponding antichains a. The green shading indicates the degree of synergy m of the respective atoms.
1 Introduction
Despite their tremendous success, the inner workings of artificial neural networks remain mostly elusive to this date (LeCun et al. 2015 Alom et al. 2019 Samek et al. 2017). It is known that, since all of the information a network can use to solve a task must already be contained in the model's inputs, the purpose of hidden internal components remains to distill the pertinent features and make them available to later computational steps. However, the way in which the features are represented among multiple neurons and how this represcntation changes throughout the training phase is not well understood (Li et al. 2015 Bengio et al. 2013). This lack of understanding continues to pose a major obstacle to the adoption of machine learning techniques in critical settings, such as medical image analysis self-driving cars, but also hinders
principled development of better machine learning algorithms (Sanck et al. 2017 Amodci al. 2016
Eykholt et al. 2018).
In a classification problem, the difficulty of extracting information about the target label from internal representations in hidden layers is a key factor for network performance and interpretability alike. As first step towards these objectives, one needs to define and quantify this notion of difficulty, which originates from complex higher-order statistical interrelations between neurons. These relations if present make it necessary to consider large groups of neurons at time to be able discriminate between certain represented target values. The number of neurons minimally needed to obtain a piece of target-discimmnatory information is conceptually related to the classical question of how many neurons are active in the representation of an input sample, and may thus be viewed as an information-thccorties equivalent of sparsity mcasure. We propose to quantify this 'information sparsity" in a principled way by answering an inluitive question: On average, how many neurons do I need to observe simultaneously in order to gain access to a piece of information? Note that while we limit ourselves to classification problems in this paper, these questions pertain to vast class of problems not limited to machine learning systems wherever information about some target variable is distributed amongst set of equivalent information bearing units.
2

---

Published in Transactions on Machine Learning Research (05/2023)
Note that while multiple notions of information exist in literature (e.g. semantic information that has meaning for biological or technical system) in this paper the term information is used in a narrow sense referring to Shannon information only. Originally developed for the analysis of communication channels, Shannon's Information Theory (Shannon 1948) is a framework capturing general, possibly non-linear correlations between random variables. As such, it lends itself to the analysis of neural network as a high mutual information between the classification label and latent representations can be proved to be necessary requirement for good classification accuracy via Fano's inequality in conjunction with the data processing inequality Geiger 2021 Appendix IV). The choice of Shannon Information over other non-semantic measures such as Renyi Entropy Rényi 1961) or Tsallis Entropy (Tsallis) 1988) has been made since the cross entropy loss function that is ubiquitous in classification networks uses Shannon information terms (Goodfellow et al.) 2016], making it plausible that tracking information theoretic quantilies can provide an insight into the learning dynamics.
From the perspective of information theory (Shannon 1948). p, neural networks are just information channels that forward the lask-relevant part of the input to the output layer while shedding lask-irrelevant parts (Tishby & ZZalavsky 2015 This said, the idea of using Shannon information to quantify the information in the activations of Deep Neural Network (DNN) hidden layers has been met with fieree debate after methodological flaws surfaced [Saxe et al. 2019: Goldfeld et al. 2019) in the influential early works by Tishby et al. (Tishby& & Zaslavsky 2015 Shwartz-Ziv & Tishby 2017). We address these issues in this work and show how information theory can be applied to quantized networks in a theoretically sound way (Section 2
Yet, even with such methodological problems out of the way, the question posed above on how features are represented cannot be answered using Shannon mutual information alone. This is due to the inability of pairwise information measures to disentangle redundant and synergistic contributions between multiple neurons (Williams & Beer 2010). leading to inevitable double- or under-countnng of information contributions. Non-overlapping information contributions can be obtained by decomposing the mutual information I(T: S) between a target random variable T and several source variables S = (Si...., S,) into its information atoms I[T : S.) such that
I(T S)
S)= = EnE U(( Sa),
(1)
which has been made possible by the introduction of recent extension to information theory known Partial Information Decomposition (PID) (Williams & Beer 2010). These atoms are the smallest set of quantities from which the mutual information between I' and all possible subsets of S can be combined (Figure 1 (Gutknecht et al. 2021). Note that here, each information atom is referred to by its corresponding 'antichain' a, which will be explained in Section 3.1
From the PID atoms, we can answer the question of how many neurons are needed on average to access a piece of information in principled way by introducing the summary statistic term 'Representational Complexity A representational complexity of C' C= 1 means that all information can be obtained from single neurons, while a representational complexity C that is equal to the number of neurons in a layer means that no information can be obtained about the target unless one has access to all neurons. By tracking the representational complexity C' of the hidden layers over training we shed light upon a key aspect of how the internal representations of the target variable evolve during the training phase. This understanding may subscquently be employed to improve network designs by providing tool to compare the dynamics different network architectures.
The main contributions of this work are (i) describing principled approach for applying PID to analyze the representations in DNNs and related systems, (ii) the introduction of representational complexity as a measure of information sparsity, (iii) discussing subsampling and coarse-graining procedures for the estimation of representational complexity and proofs of bounds on the latter, (iv) empirical results in quantized DNNs showing a decrease in representational complexity over training and through successive hidden layers, (v) comparisons of the empirical results to two related approaches and (vi) empirical results showing a difference wetween representational complexity computed on the train or test dataset.
3

---

Published in Transactions on Machine Learning Research (05/2023)
2 Related works
Tishby & Zaslavsky (015) were among the first to attempt to analyze deep neural networks from an information-theoretie perspective. Information theory, which was developed for the analysis of noisy informa- tion channels [Shannon 1948), allows to quantify information in an observer-independend way and helps to form a clear criterion of relevancy of information in hidden representations by distinguishing between the mutual information of layer activations with the ground truth classification labels and with the input variables. Viewing the networks as a sequence of such channels, Shwartz-Ziv & Tishby (2017) computed mutual information from binned activations of the individual hidden layers, and plotted the resulting trajectory in what they termed the information plane. This information plane a two-dimensional space with the mutual information between hidden layer activations and input, and between hidden layer activations and ground-truth label as its two dimensions.
However, their claim to estimate actual mutual information quantities of the network variables with their binning approach was later shown to be unfounded (axx et al. 2019 Goldfeld et al. 2019). In fact, the true mutual information between the nerkork's inputs or label and hidden layer either infinite, in the case of continuous features, or constant and equal to the finite entropy of the discrete inputs or labels (Saxe et al., 2019; Goldfeld et al.] 2019 Geiger 2021]. This is due to the fact that the network itself defines deterministic and almost always injective mapping from inputs to hidden layer activation values. For this reason, the results shown by Shwartz-Ziv & Tishby (2017) do not constitute estimates of actual information--heerrtie quantities and may at best be reinterpreted as measures for geometric clustering Gollfeld et al. 2019 Geiger 2021
Building on this realization, Goldfeld et al. (2019) showed that a meaningful information-theoeeee analysis can be performed by disrupting the injectivity and limiting the channel capacity of the network forward function. While they achieve this by explicitly adding Gaussian noise to each activation value, we achieve the same goal by training and analyzing quantized very-low precision activation values in this work. This approach is in line with recent trends towards low (Gupta et al.) 2015) and very-low precision (Hubara et al. 2017) computing for reasons of efficiency and scalability. The crucial difference to Shwwatzz Ziv & Tishby (2017) 's binning approach lies in the fact that the quantization we employ is intrinsic to the network itself, making mutual information quantities well-defined and meaningful as well as ensuring the data processing inequality holds for the Markov chain of successive hidden layers.
The picture of artificial neural networks as mere information channels needs to be refined following the insight that network layers need not only to pass all relevant information but also to transform it in such a way that it becomes accessible for subsequent processing, To understand this representation of information within each layer, one needs to go beyond analyzing hidden layers as a whole and look at the structure of information representation across the neurons within those layers instead, which can be done in a principled way by employing partial information decomposition.
Previous works on PTD in artificial neural networks have used PTD to motivate and interpret classical mutual information quantities (Wibral et al.) |017) and used it to analyze filters in convolutional neural networks YYu et al. 2020). Furthermore, Tax al. (2017) used PID to analyze pairs of neurons in generative neural networks and find that the networks move Lowards more unique representations of the larget in later stages of training. We expand upon this approach by being the first to analyze all neurons of a layer as individual PID sources, which allows also to uncover higher-order interactions between them.
Closely connected to our approach, Reing et al. (021) derived scalable measures for quantifying higher-order interactions between ncurons using Shannon information. By focusing more on scalability, however, the authors trade in some of the interprelability and expressivity of an approach based on PID. Furthermore, Reing et al. (2021) cover mostly the analysis of representations without reference to target variable, while we focus on analyzing only task-relevant information contributions. An empirical comparison to this approach can be found in Section 43
In the context of representation learning, ietommttion-theoeete approaches focus mostly on quantifying the degree of entanglement of latent dimensions in Variational Autocncoders, often with respoct to the information about some underlying generative factor. This is done by measuring the total correlation (Kim & Muih 2018 a summary statistic or the difference between two classical mutual information quantities (Chen al.
4

---

Published in Transactions on Machine Learning Research (05/2023)
2018
Tokui & Sato 2021 These works are related to ours, but analyze only specific layer, do not use the label information target and stay within the realm of classical Shannon information framework.
Other works on representations in deep neural networks have attempted to define notions of usable information based on the idea of restricting the ability of an observer to perfectly decode the presented information (uu et al. 2019 Kleinman et al. 2020). A related approach the probing of representations using simple linear readouts, for example Alain & Bengio (2017), who find empirical evidence hinting less complex representations in deeper layers. Additionally, several other summary statistics of representations have been
suggested based on dimensionality analysis (Ansuini et al. 2019 Fukunaga & Olsen 1971 or canonical
correlation analysis (Morcos et al. 2018 Kornblith et al. 2019) We compare an analysis based on local
intrinsic dimension to our measure in Section
Several other complexity measures were compared by Jiang et al. 2019)) who analyzed them in terms of their ability to predict the networks' capacitios for out-of-samplc generalization. This idea is founded on the intuitive notion that less complex representations should be more robust. to minor changes in the inputs. As our paper is focused interpretability of the novel measure of representational complexity, potential ties to generalization ability have not been studied as of yet.
3 Deriving an interpretable measure for the complexity of a representation from partial information decomposition
3.1 Background: Partial information decomposition
The mutual information T(T S) = [[T Si...., Sn)) that several source random variables $ S= (SI...,, S.) carry about a target T can be distributed amongst these sources in very different ways. While some pieces of information are unique to certain sources, others encoded redundontly by different sources, while yet others are only accessible symergislioally from several sources considered jointy. With three or more sources, even more complex contributions, in general describing redundancies between synergies, emerge. For example, some information about T' might be accessible either from source S3 alone, or redundantly to this from a synergistic combination of sources S and S2, but from nowhere else; this information would constitute one of the information atoms
As mentioned before, these information atoms can be combined to form all classical mutual information quantities I(T Sa) between T and subsets of sources S. {SS,i E a} with indices a C {l....,n) n). Conversely, the information atoms can be uniquely identified by which classical mutual information quantities they contribute to. Mathematically, this notion is captured by their corresponding parthood distribution Φ P([1,..., n)) - 1) a binary function defined on the powerset P of source indices that is equal to "1" for exactly those sets a of source indices for which the atom II(T: : Sτ) is part of the mutual information I(T Sa) (Gutknocht et al., 2021 Scction 2) (for detailed explanation, see Appendix A.1.1p Thus, the mutual information of any set of sources Sa and T can be written
(a)
I[T :.). {1)
Σ
(2)
Instead of labelling the atoms by their parthood distribution + the atoms can equivalently be referenced as I[T :S.) using certain sets of sets of source indioes P(P((1..... mn]), which can be mapped one-to-one to parthood distributions (Gutknecht al.} 2021)) and are referred (2010), who first introduced PID using these antichains (for detailed explanation, The antichains make apparent the connection between atoms and their meaning as redundancies between synergies: For example, the atom from before capturing the information that can be obtained either from S₃ alone or synergistically from S and S2 logether is referred to by the antichain {(1,2), (3))-
anlichains by Williams & Beer
Appendix [A.12.p.
The number of atoms scales super-exponentially with the number of sources n, increasing from 7579 for n sources over 7.8 million for n (Williams & Beer 2010 Wiedemann 1991). Note, however, that this increase in the number of atoms should not be mistaken for a shortcoming of PID but rather
5

---

Published in Transactions on Machine Learning Research (05/2023)
acknovledgment of the vast number of configurations information can be encoded in among multiple variables.
On the other hand, there are only 2 - I mutual information quantities that provide constraints through Equation 2 One way to resolve this underdeterminedness is through the introduction of a measure for redundancy [T(T: S.) = [T( (Sa....., Sa,,) between collections of sources indexed by a, S {I,.... n). Noting that mutual information can be interpreted as "self-redundancy" such that [(T Saa) I-(T:) (Sa)), these redundancies can be constructed from the information atoms in an analogous and consistent way to Equation as
IT S.) = Em (I(T:S))
(3)
33a
where 3 refers to the partial order of antichains on the redundancy lattice (for a detailed explanation, see Appendix [A.1.3) (Crampton & Loizou 2000 2001 Williams & Beer 2010). Since now the number of
defining equations is equal to the number of atoms, these can be computed by inverting Equation
(3) which
is known as a Inverssinn (Rota 1964 Williams & Beer 2010).
Over recent years, a number of different redundancy measures have been suggested which fulfill multitude of different additional desiderata (Lizier et al., 2018), e.g., from decision theory (Bertschinger et al., 2014), game thcory (Inco] 2017) Kelly gambling (Finn & Lizier, 2018). In this paper, utilize the 1 mcasure introduced by Makkeh et al. (002)), where '8x" stands for shared exclusions of probability mass. In essence, the measure defines redundancy by the regions of probability space which are jointly excluded by observing the realizations of multiple collections of random variables. As this redundancy measure draws only on notions from probability theory it is the most canonical choice for our purpose of analyzing neural networks. The concept of representational complexity that we introduce here, however, can readily be generalized to any other redundancy- based multivariate PID.
3.2 Representational complexity
In order to gain insight into the representation of task-relevant information in multiple equivalent source variables, we introduce in the following a principled way to evaluate the question: On average, how much of system needs to be observed simultaneously to access particular piece of information? We propose that this difficulty of retrieving label information may be quantified by considering the minimum number of sources that are needed jointly in order to reveal a piece of information. As first step towards this goal, one needs to dissect the total mutual information into pieces which individually have a clear notion of how many neurons are needed to retrieve the information.
Note, however, that such a dissection cannot be achieved with mutual information due to its inability to disentangle synergistic and redundant contributions: As an example, take two source variables S and 52 which carry information about a target T. Since the mutual information term I(T : S;) captures all information that the single source S, carries about T, one might think that the total information that be obtained from single sources is given by the sum I(T S1) + I(T S2). However, apart from their individual unique contributions, S₁ and S2 might carry some identical pieces of information about T, and this redundancy is double-counted in the above sum. Morcover, the two sources might carry some information about T in a way that il is only accessible from both sources taken logether, e.g., if one source represents a cipher text with the second providing the decryption key. The joint mutual information T(T S1, S2) thus consists of four atoms: The two unique atoms of the respective sources, their redundancy and their synergy, while the individual mutual information terms I(T S)) consist of the unique information of the respective source and the redundancy between both sources. Thus, if we subtract the sum of the individual mutual information terms from the joint term we obtain
I(:: S1, 82) 1(T S1) - 1[T : S2) = synergy - redundancy,
with no way to quantify synergy or redundancy individually. Since the redundancy can be obtained from any single variable while the synergy necessarily requires access to both variables jointly, it is impossible to
6

---

Published in Transactions on Machine Learning Research (05/2023)
dissect the joint mutual information into pieces with well-defined minimal mumber of neurons needed to retrieve the information within the framework of classical Shannon information alone.
The PID atoms, on the other hand, do have a well-defined minimal number of neurons necessary to retrieve the information, which is captured by what we term the "Degree of Synergy" m given by
m(a) == min |a): aEa
(4)
which is illustrated for all trivariate PID atoms in Figure 1 For instance, the PID atom with the antichain ((1)4)233)) 3)) has degree of synergy of m = 1 since the information can be obtained by observing the single sources S1 or S4, while the atom with the antichain {f1,2)33 4, 5}} has degree of synergy of m = 2 since the smallest set of sources the information can be retrieved from is the pair consisting of S and S2-
Finally, the average degree of synergy, weighted by the relative information contributions of the respective atoms, defines the "Representational Complexity" C' as
1 C: I(T 5) (.)
(5)
A representational complexity of C'= 1 means that all information can be obtained from single sources, while C' is equal to the number of sources if the information is spread purely synergistically between all of them. Bare in mind that representational complexity is defined as the average over PID atoms and does thus not reflect the "worst case": Even if C' is smaller than its theoretical maximum one might need more than C sources identify specific target label.
5 is to first sum up all PID atoms of fixed degree of synergy and then taking their weighted average. These sums of PID atoms are conceptually related to the Backbone Atoms introduced by Rosas et al. (2020). Rosas measure, however, does not constitute PID in the strict sense, as mutual information terms between the target and subsets of sources cannot be constructed from the atoms. This makes the backbone atoms less suitable for quantifying how many neurons are needed to access a piece of information.
An alternate way to view the average in Equation
The concept of representational complexity is linked to the idea of sparse coding in neuroscience (Olshausen & Field 2004), in that both approaches aim to capture the relevancy of higher-order relations between neurons. While sparsity measures quantify the spread of activily tied to individual realizations across the neuron population, e.g., by measuring the average momentary number of active neurons, representational complexity measures the spread of information, i.e., how distributed the ability to distinguish between realizations of the target variable (e.g. the label in classification tasks) is. Note further that representational complexity computed on the mutual information with the network's target variable, thus relating to only task-relevant components of the activation patterns, while sparsity measures are generally not selective about the task-relevancy of the activations.
From its definition as an expectation value of the degree of synergy weighted by the fractional information contribution of the respective PID atoms, it is evident that C is well-defined whenever I(T: S) * O, otherwisc C is not only mathematically undefined by also conceptually vacuous. Another aspect to pay altention to is that when both the target and the sources are continuous variables, C can be undefined when I(T: S) is infinite. Thus, we can determine that C is well-defined for the quantized activation values with finite alphabets considered in this paper.
From the coormattion-therette structure of C, it is easy to see that certain desirable properties of mutual information and PID are inherited by C.
Symmetry Representational complexity is invariant under the reordering of the source variables.
Invariance under isomorphisms Representational complexity is invariant under isomorphic mapping of the target or the sources, for example, invertible relabelling of classification labels.
There are other desirable properties that are related to the choice of the PID measure. The first is the continuity a function of the joint. probability distríbution S), This is an important, prerequisite to
7

---

Published in Transactions on Machine Learning Research (05/2023)
A label representation
B
E
01234500 5
label representation
S
1
0
2
1
3
1
1
So S1 S2 S3
So
S1
S2
S3
So
C=1
C
1.21
C 1.89
C label representation
D
F
label representation
S2
0
1
1
1
2
0
1
2
0
1
S1
15
1
15
0
label
So
0
So
S1 S2 S3
So
S
S2
S
1
C C11.67
C=1
C 2.83
Figure Demonstration of representational complexity with simple examples. Subfigures A-D show different label encoding schemes in binary neurons S,,..., (circles). Subfigure E shows an encoding of two labels (purple and yellow) in two eight-level neurons, while F shows the three-dimensional extension. The corresponding representational complexities C' are denoted in the bottom left corner.
robust estimation of the PID atoms, as small deviations in the empirical probabilities don't incur drastic changes the represcntational complexity. Another less important property diffcrentiability as function of the joint probability distribution p(T,S). The fax measure utilized in this paper has both properties (Makkeh) et al. 2021
Finally there are two intuitive bounds for the representational complexity. The representational complexity is lower than or equal to the maximum number of neurons and higher than or equal to one. The first bound is attained if one needs the whole layer to extract any the task-relevant information, while in the latter case, all information can be extracted from single neurons.
Example applications of representational complexity To become familiar with the intuitive inter- pretation of reprosentational complexity, demonstrate its with simple information encodings in small toy examples. Consider categorical random variable with finite number of distinct classes, which are labelled by integers and occur with the same probability. How does representational complexity differ between different representations of this information in multiple binary neurons?
First, imagine four distinct values are represented sparsely across four neurons (Figure A). As for all such one-hot encodings, the representational complexity of this encoding is equal to one (for formal proof, refer to Appendix A.2.2). This is intuitively clear: For each realization, you only need access to the one neuron that is equal to to fully determine the correct label.
Using the same number of neurons, we can also encode the same information in a more complex way. For instance, take pairs of neurons that redundantly represent digits of a binary representation of the label index (Figure B). In this case, there is no longer a single neuron that contains the full information about the target, which reflected by an increase of the representational complexity to C' 1.21.
Using all of the coding capacity of the four neurons from the example before by encoding 16 distinct states in binary code (Figure CC) the representational complexity increases further: C reaches value of 1.67
8

---

Published in Transactions on Machine Learning Research (05/2023)
more information has to be encoded synergistically between the four neurons. The reason why C' is not equal to 4, despite the fact that the correct, label can only be isolated with access to all four neurons, is that parts of the information, e.g,, the parity of the label number, can be extracted from fewer than all sources, reflecting the average nature of C. Conversely, if we now again expand the 16 realizations to 16 neurons in a one-hot encoding (Figure D), we revert back to a representational complexity of just C = 1. From this, we gain the intuitive realization that the closer one gets to the channel capacity, the more one is forced to encode some of the information in more complex, higher-synergy terms.
However, even small amounts of information can be encoded in a highly synergistic manner. Consider the more realistic case of two discrete artificial neurons with cight activation levels cach, with the target value being the exclusive disjunction (XOR) of the thresholded neurons' activations (Figure E). Despite encoding just a single bit of information, the representational complexity assumes a value of C= 1.89. This value is close to the theoretical maximum value of C n = 2, with the small difference being due to the nature of the In redundancy mcasure. Extending this task to the parity of three thresholded neurons (Figure F), the representational complexity attains value of C = 2.83, similarly close to its maximum of C = 3.
4 Application to deep neural networks
To exemplify the utility of our measure, show how it can be applied to analyze the hidden layer representations of deep neural network classifiers solving the well-cstablished MNIST (LeCun et al. 1998) handwrilten digit recognition and CIFARIO (Krizhevsky) (209)) image classification lasks. For the former, we employ a small fully-connected feed-fore forwafo network of which we analyze the last four layers before the output layer while for the latter we employ more sophisticated convolutional network architecture and analyze the final small fully- calyycooneeted layers.
The MNIST network consists of seven fully-connected layers, starting with vector of the 784 grayscale pixel values of the image input, tapering down to only five neurons in layers L3, L4 and Ls and culminating in a ten neuron one-hot output vector, in which each neuron represents one of the ten possible digits (Figure 3 A The structure has been chosen such that the three successive five-neuron layers can be analyzed in full, as it is practically infeasible at present to compute the full PID for more than five sources because of the fast-growing number of PID atoms.
In order to limit the channel capacity to be able to observe non-trivial information dynamics Goldfeld et al., 2019), all layers are discretized to cight or four quantization levels (3 or 2 bits) per neuron during both training and analysis (Appendix A.3). The networks are trained using stochastic gradient descent for 10 training epochs and reach, with eight quantization levels, an average accuracy of 99.9(1) % on the train and 14. % on the test set for 20 runs with unique random weight initializations. Details about quantization schemes and forward-stochastic backprop algorithm used in training can be found in Appendix A.3
The larger CIFARIO network comprises three convolutional layers with max pooling and ReLU activation functions. The outputs of the last convolutional layer are flattened and fed into fully-connected foed-forward section of the network employing tanh activation functions (Appendix A.3). Due to the conceptual difficulties with quintizing functions with semi-infinite ranges we applied the quantization with 8 levels only to the last layers with tanh activation function, which are also the ones analyzed later. For 10 random weight initializations, the CIFAR10 network achieves an average accuracy of 99.90(4) % on the train and 68.9(7) % on the test set after 10 epochs of stochastic gradient descent training.
Except for when explicitly specified otherwise, all analyses have been done on the train dataset. The rationale for this is twofold: Firstly, since the train set samples drive the learning process, analysis the train dataset samples allows more insights into what information-theoret representation the backpropagation algorithm implicitly pursues. Secondly, the train datasct - being defined as the set of samples the network is trained on is trivially finite in size and can be analyzed in full. The (est dataset, on the other hand, may be interpreted as merely a finite-sized sample (e.g. 60.000 handwritten digits for MNIST) of a potentially infinite generative process (c.g. the process of handwriting and formatting digits)) which mcans that potential biases of the sampling and estimation need to be kept in mind. A comparison between results computed on the train and test datasets can be found in Section 4.5
9

---

Published in Transactions on Machine Learning Research (05/2023)
A
B
Repr. Compl. C
train
Acc.
test
0
01
10
10
L3
L
L5
1H/
Lp
L1
1/2
L3
L4
Ls
L6
0
1
102
104
784
50
10
5
5
10
Training Epoch
Figure 3: Representational complexity of hidden layers with respect to the label decreases over training and throughout successive hidden layers. A The MNIST classifier network consists of seven fully-connected layers with tanh activation functions for all but the last layer, which is equipped a softmax activation function. B The representational complexity of the three small hidden layers La,, T-4 and Tog computed from their PID atoms the train data set. The solid lines show the average of 20 randomly initialized runs, the shaded areas contain 95% of the data points. The inset shows train and lest set accuracy.
A
B
train
Repr. Compl. C
Acc.
train
3
Acc.
Repr. Compl.
test
test
3
01
102
104
01
102
10a
2
L3
L
2
La
L11
Lg
1 1/
1
1///
0
102
104
102
104
Training Epoch
Training Epoch
Figure The observed decrease in representational complexity appears to be a robust trend reproducible on different output encodings and on larger convolutional networks solving CI- FAR10. A The MNIST network with binary output encoding shows a brief increase in representational complexity after about 10 epochs of training before decreasing. The representational complexity of the binary output layer for 10 equiprobable classes has been numerically determined to be C 1.46 and is indicated by dotted line. B The representational complexity of the small fully-connceted layers of the CIFAR10 network shows a similar small increase in representational complexity in the beginning before converging to a low value close to the theoretical minimum of 1.
4.1 Representational complexity in small layers
For layers with up five variables, the PID can be computed in full. For the MNIST network with eight quantization levels, analyzing the mutual information of the hidden layers L2, Lg and La with the ground- truth label allows to track the representational complexity both over training and through the successive layers (Figure 3 B). We find that both with increasing training epoch and layer index, the representational complexity C decreases.
This decrease representational complexity appears to be a robust trend observable in networks independent of the encoding enforced on the output layer, chosen task and network architecture. While neural network seems to approach the output complexity in the case of one-hot output representation, for which Cone-hot (proven in Appendix A.2.2.. the representational complexity of the hidden layers decreases below that of the
10

---

Published in Transactions on Machine Learning Research (05/2023)
A
B
C
D
0
Repr. Compl. C
3.0
L
08
Repr. Compl. C
3.0
L
2.5
2.5
2.0
2.0
1.5
1.5
Hh
H/
0
1
102
104
0
1
102
10
Training Epoch
Training Epoch
Figure 5: Representational complexity of subsampled and coarse-graine grained neurons decreases over training in the MNIST network with four quantization levels. A Five neurons randomly selected from layer L2 of the MNIST network to form the random variables for the subsampling analysis. B For 26 random choices of source variables each on 20 training runs, the subsampled representationas complexity shows a decrease over the training phase, but with high variance. C Five coarse- saasse-rrained random variables are constructed from pairs of neurons of the second hidden layer. D The coarse coarso-grainct grained represcntational complexity averaged over 20 training runs is seen to decrease with increasing training epoch.
targeted output representation in the case of binary label encoding (Figure A.A). In the binary encoding case, however, an intermediate increase in representational complexity is observable, which might be attributable to early restructuring of the representation.
Likewise, for the five-neuron layers L10 and L1 of the CIFARIO network, one observes a plateau of the representational complexity until about epoch 10 which is followed by a subsequent decrease (Figure B). Again, C. is lower for the layer later in the network.
4.2
Representational complexity in larger layers
Typical production neural networks have layers which are much wider than five neurons (e.g., He et al.) 2016 Krizhevsky et al., 2012). Since the compute required for a full PID for six or more source variables is prohibitive because of the rapidly increasing number of atoms, in order to be able to apply the tool PID in general and representational complexity in particular to wider layers, one needs to devise procedures to reduce the number of random variables analyze. In this section, we present two complementary approaches to make representational complexity applicable to moderately wider layers, namely subsampling and coarse-graining, and show the latter to be the more theoretically sound approach.
Subsampling A straightforward approach for reducing the number of random variables, which has been employed in previous works (e.g., Tax et al. 2017), is to subsample only ñ neurons from layer to use as PID sources. By randomly selecting five neurons from the second layer of the MNIST network with four quantization levels, we again observe a decrease of the representational complexity of the hidden layer with respect to the label over the training phase (Figure 5 A, B), albeit with a larger amount of variability.
However, this approach suffers from fundamental conceptual flaws. By choosing only ñ source variables S. = (S.,....,, and decomposing their mutual information I(T S.)) with the target variable T, only atoms with degree of synergy of less than equal to ñ can be quantified, meaning that any higher interaction than ñ will be overlooked, resulting in potential underestimation of the true representational complexity of all n sources. At the same time, pieces of information that appear to only be obtainable synergistically within one subset S. of sources may very well be redundant with a single source S, & Sa, thus leading to potential overestimation of reprosentational complexity. For these reasons, no bound on the true representational complexity can be established from subsampling and we find subsampling to be an unsuitable approach for overcoming the scaling difficulties of PID.
11

---

Published in Transactions on Machine Learning Research (05/2023)
A
B
//
Avg. IPCA Dim.
train
Acc.
Acc.
3
test
3
CRAing
01
10
104
2
01
102
10
L3
1
LA
2
Ls
H/
0 HH1 0
0
102
104
10
104
Training Epoch
Training Epoch
Figure 6: Comparisons with alternative approaches shows decreasing trend but no ordering through successive hidden layers. (A) The Reing Representational Complerity CReing (Equation (9) decreases only in the first epochs of training and seems fluctuate a relatively high level. (B) The local intrinsic dimension of the hidden layer activations of the MNIST classifier network based the Fukunaga-Olsen algorithm [Fukunaga & Olsen 1971 shows similar trends as representational complexity but increases al the beginning of training. For the intrinsic dimension est imation,, the number of neighbors set to 100, and the alphaFOO parameter was set to 0.05, both being fairly common settings.
(A) and (B) Both quantities have been computed for the same three hidden layers of the network as the results shown in Figure
Coarse-graining A complementary approach reduce the number of variables is combine multiple neu- rons, forming fewer nigheer-imeenionall random variables; this procedure will be referred to coorse-graining. This approach appears more theoretically sound because one can show that the act represenlational com- plexity can be bounded up to factor by the coarse graining representational complexity. In Appendix [A.2.]] we prove that if d neurons S; each are combined to random variables 3,, the true representational complex- ity of the layer is bounded from below by the coarse gaasse-rrined representational complexity computed from 5 Spi...,,S//J) while being simultancously bounded from above by d times this value, i.e.,
C(T :r: 5) - C(T: 8) 5 :dC(:
(6)
In our example network with four quantization levels, the representational complexity computed from randomly assigned neuron pairs in the second hidden layer consisting of ten neurons, exhibits decreasing pattern over training that is highly similar to that of the representational complexity computed with individual neurons as sources in layers La to Ls (Figure C, D).
4.3 Comparison to Neural Information Decomposition by Reing et al.
In their 2021 paper, [Reing et al. introduced an alternate way to decompose the mutual information between a target variable T and multiple source variables S into what they term Directed Local Differences Cr 1[[k), such that
I(T :) Σ Cr(k-11(k), - 1[(k),
(7)
where n is the number of sources. These terms, defined as
Cr(kk - 1|(k)
S(T
Σ H(T)Sa)
(8)
ja|=k-1
are proposed by the authors as "a measure of information order k the source variables about a target variable T (Reing al. 22021] and are conceptually close analogs to the sum of PID atoms of fixed degree of synergy. Nevertheless, due to their definition depending on the conditional entropy of the target
12

---

Published in Transactions on Machine Learning Research (05/2023)
(in particular fractional sums of (T) Sa)), these quantities cannot be constructed from the PID atoms. This means that a theoretical comparison is out of reach for this paper, whereas an empirical comparison between the two approaches might still be enlightening. Comparing the sums of PID atoms of a fixed degree of synergy to the directed differences for the MNIST network revcals that the sums of PID atoms attribute more of the information to only single neurons by the end of training, while the directed differences imply that higher-order interactions stay relevant to a higher degree (Figure gb.
In close analogy Equation 5] one can utilize the directed differences Cr - 1l]b) to introduce a measure of Reing Representational Complezity
" I(T S)
CReing
Cr
k=1
that captures the average order k of information between the target T and multiple sources S.
While we empirically observe this Reing representational complexity to decrease in the earlier epochs of training, it does not decrcase monotonically, stays higher than the PID based representational complexity and does not decrease throughout successive hidden layers (Figure G While there is no priori reason to expect convergence to a particular value of complexity, over training and the ordering of C for successive hidden layers - the latter of which is absent for CReing to be intuitive findings which might reflect the networks tendency to encode the relevant information in more and more simple terms.
find the general trend of decreasing complexity
These results show that representational complexity leads to different values while also arguably being the more well-founded measure compared to the directed differences, because it leverages the more expressive PID framework compared to Shannon mutual information. On the other hand, the approach using directed differences has the advantage of being more easily applicable larger number of source variables.
4.4 Comparison to Local Intrinsic Dimension
Additionally to the comparison with the approach of Reing et al. (2021) b we want to provide independent baseline comparison with an established non-information theoretic measure. For this empirical comparison, we chose the local intrinsic dimensionality of the manifold of discretized layer activation patterns as described by Fukunaga & Olsen (1971). This method estimates the number of dimensions based on the largest singular values of the covariance matrix of the data, and is implemented in the scikit-dimension package (Bac et al. 2021j. The reason we chose this method out of multitude of potential candidates that it tries to estimate the minimum number of parameters that are necessary to locally describe the data, which is similar to our approach. Taking the average over the dimensions estimated for cach sample, however, is quite different from taking the average over PID atoms. Additionally, the intrinsic diinension is less dependent on the orientation of the manifold with respect to the individual neuron dimensions. However, the most important conceptual difference to our approach is that the intrinsic dimensionality is not immediately related to the label since it takes into account only the internal representations of the layers.
For the MNIST dataset, we see empirically that both the local intrinsic dimension and the representational complexity decrease with training, but the intrinsic dimension shows no consistent order throughout successive hidden layers (Figure B) The two measures seem to be related, however the lack of consistent decrease over layers also points at potential benefit of our approach: being target-oriented towards the label allows for focus on the more relevant aspects of the representation in terms of solving the task.
4.5 Representational Complexity for Train and Test datasets
To further illustrate the behaviour and possible usccases of representational complexity, we next compare the representational complexity computed on the train dataset with results computed on the test dataset. The train dataset is the default choice for computing the representational complexity, since it defines the task that the networks actually optimize their representations for (sce also Scction 4 on page D). In Figure [1 we compare the results on the train and test datasets for averages over multiple runs for the MNIST dataset, and we provide additionally the results for individual training runs in Figure T2 Generally, find consistent.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Train Set
Test Set
Repr. Compl. C
Acc.
Acc.
La train
Lg test
0
L4 train
3
01
102
10a
102
L4 test
Lg train
la test
11
Hh
0
10a
10*
10a
10*
Training Epoch
Training Epoch
Figure 7: When trained on an MNIST dataset with shuffled labels, representational complexity shows a different behaviour for train and test datasets. Both figures show the representational complexity for the three hidden layers L3, L4 and L5, using the architecture illustrated in Figure 3A, but trained on a dataset with a random new assignment of labels to MNIST dataset images. Coinciding with the rise in training accuracy, the representational complexity for both train and test datasets reaches a pronounced peak around epoch 200. The representational complexity on the train dataset decreases towards the end of training. However, for the test dataset (right, also with shuffled labels) it remains substantially higher after this peak. For improved visibility of this phenomenon we combined the averages from both figures into single diagram in Figure 10 in Appendix A.4
and robust trend of higher representational complexity for the test dataset compared to that for the train dataset towards later stages of training.
In the extreme case of forcing the network to predict randomly assigned, fixed labels (by memorizing them via overfitting, see Figure 7 this increased representational complexity on the test dataset starts to appear with rising accuracy of the network and the gap between representational complexity on the train and test. dataset is particularly wide.
While conclusive statements cannot be drawn from these empirical findings alone, using representational complexity for investigating the phenomenon of overfitting from novel perspective appears as enticing topic for further rescarch.
5 Discussion
In this work, introduced representational complexity a measure of sparsity of task-relevant information in neural networks. We first explained how principled and meaningful application of information theory in neural networks is possible by using quantized activation values in both training and evaluation. Subsequently, we derived representational complexity from partial information decomposition and show it to be the theoretically sound answer to the question of how many neurons need to be observed simultancously in order access an average piece of information. For larger layers, we presented subsampling and coarse-graining procodures. We discussed issues with the former approach, while deriving bounds from results of the latter approach.
In small quantized deep neural networks, we find representational complexity to decrease both over training and through successive layers. We empirically find the reduction to be robust result which can not only be observed when analyzing small layers neuron-wise, but also when using subsampling or coarse-graining on larger layers. Furthermore, we provide first evidence that the final values the representational complexity converges to do not depend on the representational complexity of the chosen output representation. We hypothesize that the dynamics of representational complexity in early stages of training might indicate an early restructuring of representations for the binary output encoding and the more complex CIFAR10 task.
14

---

Published in Transactions on Machine Learning Research (05/2023)
A comparison to the related information-theorctic tools introduced by Reing et al. (2021 shows that by focusing more on computational tractability on larger networks, Reing et al. measures can only partly reproduce our finding of a decreasing representational complexity throughout successive hidden layers. Similarly, an analysis based on local intrinsic dimensions (Fukunaga & Olsen, 1971 shows that while this dimension does indeed decrease over training, it does in practice not necessarily decrease with successive hidden layers, which is intuitively what we would expect for the complexity of the relevant part of the representation. This points to an important feature of our approach that many other comparable measures lack, namely the possibility to specify target random variable. Thereby one is able to study a specific aspect of the representation, for example the degree of synergy of the task-relevant mutual information with the label. Discovering the precise differences between these related approaches on a theoretical ground, however, remains a promising avenue for future research.
Interestingly, towards the end of training we observe a consistent difference between the representational complexity computed on the train and the test datasct, pointing at a difference in the corresponding representation (see Section 4.5). In the extreme case of network overfitting on randomly shuflled labels for the MNIST dataset (Figure [0) we report substantial gap between the representational complexity on the train and test datasets. We believe that subsequent rescarch studying this behaviour explicitly could result in new insights into the phenomenon of overfitting.
Our results also indicate some problems with previous approaches that apply PID by subsampling only pairs of sources (e.g. [Txx el al. (2017)) Since the representational complexity is high in the early stages of training, high-order interactions between neurons appear to play major role. These, however, go unnoticed in pairwise approaches. Representational complexity thus not only allows for assessing whether and when it is justified to only look at lower-order interactions, but can also be used to quantify how much of the higher-order interactions are disregarded.
Limitations Scaling our approach to larger networks remains challenge. To mitigate this problem, we introduced subsampling and course-graining procedures, which allow the application of representational complexity to moderately larger networks. However, while subsampling suffers from conceptual flaws, the proven attainable bounds on coarse graining become impractically loose for large networks. For typical production networks we suggest two approaches: Firstly, results found on small loy net nerworks may be generalizable to larger networks, e.g., by inductive proofs, and secondly, estimation of representational complexity could become feasible by finding way to compute without computing all PID atoms beforchand. The latter can be achieved by employing a PID based on synergy instead of redundancy, for example by extending on ideas of Rosas et al. (2020).
Furthermore, our analysis methods are currently restricted to intrinsically discrete systems, 1 was originally defined for discrete variables only (Makkeh et al., 2021). However, in the meantime, a continuous generalization ion of Tax has been proven to exist and to be measure-theorrticalll well-defined (Schick-Polan ltt al.) 2021). Once an efficient estimator for this generalized measure is available, this will make it possible to analyze also continuous systems in which the total mutual information is inherently restricted to a finite value, e.g., by some form of noise in the system.
Conclusion and outlook We have here shown how to use the representational complexity to gain insight into the structure of the internal representations as an average across all source and target realizations. Due to the local nature of TAX, this analysis can also be further broken down into the representational complexity for individual target realizations, i.e., class labels. Such an analysis may reveal for example that some classes are linked to representations of high complexity while others are not, or that some classes represented with low complexity earlier than others during training, while yet others fail to reach a low complexity. Complementary, we have shown that a more detailed analysis could also be achieved by. instcad of averaging over the different degrees of synergy, splitting up the total mutual information into "backbone atoms" (Roaas et al. 2020) which combine all atoms of the same degree of synergy m and investigating how they evolve over training individually. In addition, once a proper mcasure for Synergy basod-PID is proposod, one could use that fact that these Backbone atoms, as demonstratod by Rosas et al. (2020), can be computed individually which will allow for computing the representational complexity for significantly larger layers.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Some theoretical properties of representational complexity have yet to be uncovered. For instance, lower bound to C. might be derived from the notion that closer to channel capacity, one is forced to represent some information in higher synergy terms.
Given the PID atoms, a whole suite other interesting and easily interpretable quantities can be devised. One enticing candidate the average multiplicity of information, defined as M = (1/TTT S)) Ea (Taa) lo The quantity reflects the average number of times piece of information is represented redundantly and thus appears to be promising candidate to serve complementary summary statistic representational complexity, which relates to the synergy of the information encoding.
In future research, one may also choose the PID source and target variables differently. While in this work we focused on analyzing how the information about the classification label is encoded in the network, a similar analysis for the information the hidden layers about the whole input samples might reveal differences in the representation of task-rclevant and task-irrelovant information.
More generally, we promote representational complexity as a principled novel tool for general complex systems in which group of equivalent variables jointly holds information about target variable. New possible applications include both other artificial network architectures such as recurrent networks, but also biological or ecological systems. Being derived from first principles in information theory and partial information decomposition, representational complexity provides a clear and intuitive interpretation and the suggested subsampling and coarse-graining procedures make applicable to wide variety of questions.
Code availability
To analyze artificial neural networks using information-theoreticc tools, we developed the uninfo python package containing scripts reproduce the main results of this paper. The code available on GitHub under the URL hctps://ggttauo..c//rrreeemm
Acknowledgements
We would like to thank Alexander Ecker for fruitful discussions and feedback on this paper. We would also like to thank Valentin Neuhaus, Lucas Rudelt, Marcel Graetz and the rest of the Priesemann group for their valuable comments and feedback. MW and AM are employed at the Göttingen Campus Institute for Dynamics of Biological Networks (CIDBN) funded by the Volkswagen Stiftung. DE and MW were supported by funding from the Ministry for Science and Education of Lower Saxony and the Volkswagen Foundation through the *Niedersächhssches Vorab" under the program "Big Data den Lebenswissenschaften project "Deep learning techniques for association studies of transcriptome and systems dynamics in tissue morphogenesis". AS and VP received funding from the Deutsche Forschungsgemeenschuuf (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2067/1 390729940.
16

---

Published in Transactions on Machine Learning Research (05/2023)
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. International Conference on Learning Representalions, 2017.
Md Zahangir Alom, Tarek M Taha, Chris Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Mahmudul Hasan, Brian C Van Essen, Abdul AS Awwal, and Vijayan K Asari. state-of-the-at survey on deep learning theory and architectures. Electronics, 8(3):292, 2019.
Dario Amodci, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safely. Xiv preprint urXi::1606. 06565, 2016.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Jonathan Bac, Evgeny M Mirkes, Alexander N Gorban, Ivan Tyukin, and Andrei Zinovyev. Scikit-dimension: a python package for intrinsic dimension cstimation. Entropy, 23(10):1366 2021.
Heinr Beckurts and R Dedckind. Über Zerlegungen von Zahlen durch ihre grössten gemeinsamen Theiler. Springer, 1897.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pottern analysis and machine intelligence, 35(8))1798-1828, 2013.
Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 1614))2111-2188, 2014.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018
Thomas M Cover and Joy A Thomas. Flements of information theory 2nd edition (wiley series in telecommu- nications and signal processing). Acessado em, 2006.
Jason Crampton and George Loizou. Two partial orders on the set of antichains, 2000.
Jason Crampton and George Loizou. The completion of a poset a lattice of antichains. International Mathematical Journal, ((3):223-234) 2001.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world world attacks deep learning visual classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1625-1634, 2018.
Conor Finn and Joseph T Lizier. Pointwise partial information decomposition using the specificity and ambiguity lattices. Entropy, 20(4):297, 2018
Keinosuke Fukunaga and David Olsen. An algorithm for finding intrinsic dimensionality of data. IEEE Transactions on Computers, 100(2):176-183, 1971.
Bernhard C Geiger. On information plane analyses of neural network classifiers review. IEEE Transactions Neural Networks and Learning Systems, 2021.
Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. International Conference on Machine Learning, 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pp. 1737-1746 PMT.R, 2015.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Aaron J Gutknecht, Michael Wibral, and Abdullah Makkeh. Bits and pieces: Understanding information decompasition from part-whole relationships and formal logic. Proceedings of the Royal Society A, 477 (2251):20210110, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 770-778, 2016.
Itay Hubara, Matthicu Courbariaux, Daniel Soudry, Ran El-Yanis and Yoshua Bengio. Quantized neural
networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869 6898, 2017.
Robin AA Ince. Measuring multivariate redundant information with pointwise common change in surprisal. Entropy, 19(7):318, 2017.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference Learning Representations, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2658. PMLR, 2018.
Michael Kleimman, Alessandro Achille, Daksh Idnani, and Jonathan Kao. Usable information and evolution of optimal representations during training. In International Conference on Learning Representations, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geolfrey Hinton. Similarity of neural worw representations revisited. In International Conference on Machine Learning, pp. 3519-3529, PMLR, 2019.
Alex Krizhevsky. Learning multiple layers features from tiny images, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based lcarning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Yann LeCun, Yoshua Bengio, and GeofTrey Hinton. Deep learning. nalure, (21(3333) 444, 2015.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different. neural networks learn the same representations? In Feature Extraction: Modern Questions and Challenges, pp. 196-212. PMLR, 2015.
Joseph T Lizier, Nils Bertschinger, Jürgen Jost, and Michael Wibral. Information decomposition of target effects from multi-source interactions: Perspectives on previous, current and future work. Entropy, 20(4): 307, 2018
Abdullah Makkeh, Aaron Gutknecht, and Michael Wibral. Introducing differentiable measure of pointwise shared information. Physical Review E, 103(3):032149, 2021.
Ari Morcos, Maithra Raghu, and Sany Bengio. Insights on reprosentational similarity in neural networks with canonical correlation. Advances in Newral Information Processing Systems, 31, 2018
Bruno A Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in newrobiology, 14 (4):481-487, 2004.
Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Discovering higher-order interactions through neural information decomposition. Entropy, 23(1):79, 2021.
Alfréd Rényi. On measures of entropy and information. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, volume 4, pp. 547-562. University of California Press, 1961.
18

---

Published in Transactions on Machine Learning Research (05/2023)
Fernando E Rosas, Pedro AM Mediano, Borzoo Rassouli, and Adam B Barrett. An operational information decompasition via synergistic disclosure. Journal of Physics A: Mathemalical and Theoretical, 53(48): 485001, 2020.
Gian-Carlo Rota. On the foundations of combinatorial theory i. theory of möbius functions. Zeitschrift. für Wahuscheinichhettstheeree und verwandte Gebiete, 2(4):340-368, 1964.
Wojciech Samek, Thomas Wiegand, and Robert Müller. Explainable artificial intelligence: Understand- ing, visualizing and interpreting deep learning models. arXiv preprint arXiv:1 (tl::1708.086 088808 2017.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Kyle Schick-Poland, Abdullah Makkeh, Aaron J Gutknecht, Patricia Wollstadt, Anja Sturm, and Michael Wibral. A partial information decomposition for discrete and continuous variables. arXiv preprint arXiv:2106. 123223 2021.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379 423, 1948.
Ravid Shwartz- Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint Xrx:::1703. 000.00010, 2017.
Tycho Tax, Pedro AM Mediano, and Murray Shanahan. The partial information decomposition of generative neural network models. Entropy, 19(9):474, 2017.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Seiya Tokui and Issei Sato. Disentanglement analysis with partial information decomposition. arXiv preprint arXiv:2108.0 1021100811775.33 13753, 2021.
Constantino Tsallis. Possible generalization of boltzmann-gibls statistics. Journal of statistical physics, 52: 479 487, 1988.
Michael Wibral, Conor Finn, Patricia Wollstadt, Josoph T Lizier, and Viola Pricsemann. Quantifying information modification in developing neural networks via partial information decomposition. Entropy, 19 (9):494 2017.
Doug Wiedemann. A computation of the eighth dedekind number. Order, 8(1):5 6, 1991.
Paul L Williams and Randall Beer. Nonnegative decomposition of multivariate information. ar preprind arXiv: 1rX4: 004.25515 2010.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. International Conference on Learning Representations, 2019.
Chhavi Yaday and Léon Bottou. Cold case: The lost mnist digits. Advances in neural information processing systems, 32, 2019.
Shujian Yu, Kristoffer Wickstrøm, Robert Jenssen, and Jose Principe. Understanding convolutional neural networks with information theory: An initial exploration. IEEE transactions on neural networks and learning systems, 32(1):435 442, 2020.
19

---

Published in Transactions on Machine Learning Research (05/2023)
A Appendix
This appendix starts by providing more background the PID problem in Appendix A.1.1 by identifying the PID aloms by their parthood distributions derived by GGutknecht et al. 20221 from first principles of mereology. Then we explain the equivalence of the parthood distribution and the antichains that was established by Gutknecht ct al. (2021). Finally we describe in Appendix A.1.3 the lattice structure that allows computing the amount of information the PID atoms upon the introduction of a redundancy measure as shown by the seminal work of Williams Beer (010))
In the subsequent two sections of the appendix we provide novel proofs of the bounds course graining (in Appendix A.2.1 and the proof of the representational complexity of the one-hot representation being one for any number of labels (in Appendix A.2.2
Finally, we include a description of the implementation and the quantization schemes in Appendix A.3 and show additional results in Appendix A.4
A.1 Additional background
A.1.1 Parthood distributions characterize the PID atoms
Gutknecht et al. 2021) recently showed that the PID structure can be built independently of the redundancy concept and a specific measure from first principles mereology, i.e,, part-whole relationships. This is because PID seeks to find the parts of information that constitute the whole information I(T .i...., 5,) and are characterized by which collection of sources they are part of. In what follows we give a brief introduction to the PID structure using the mercology approach as has been developed by (Gutknecht et al. 2021 Section 2).
Through the lens of mereology, PID atom is that part of the total information about the target that is provided by specific collections of source variables but not others. Thus, each atom is associated with a certain set collections of sources Sar...., Sa.. E P(S). The corresponding PID atom is then defined as the information contribution that can be obtained if and only any of the collections accessed.
For instance, in the case of two sources, consider all collections of the source variables, i.e., the powerset. P(S) (0, {S1}. {S²). [S1, S2 Now by looking at what collections of sources the PID atoms can be part of, Gutknecht et al. (2027) postulates two straightforward principles: (i) no information atom can be part of the empty set as it represents no source variable and (ii) all the information atoms are part of the full collection (fS,S$2) This leaves four possibilities to combine the collections: Atoms can be either be a part of (a) both { and {S2} or (b) only {Si} or (c) only {S2} or (d) only (S1, (Table I].
Atom
{Si}
(S2)
[S1,52)
a) (.)
b) ((((S:)))
0
0
1
c) ((((5))))
0
1
1
d) ((H(
0
0
1
Table 1: Parthood distributions for two sources. The value "0" stands for not being a part of the collection's information about T, as opposed to "1". "0" and "1" represent principle (i) and (ii), respectively. "0" and "1" represent the four possible choices of part-whole relationships to {S1}} and {S²} realizing the four PID atoms. Each atom is annotated with its combination collections of sources excluding supersets, e.g., the atom corresponding to possibility (a) annotated with the collections ((S1), {S2)) as shorthand of {(S1), (S)), (S1, } and so on.
The first atom in the table is [(([S)). (S2))) that is part of S and S2. This atom represents what is rodundant to S, and S₂ as is part of both variables. The second and third atoms are exclusively parts of either S, and S2, thereby they are unique to either variable. These two atoms represent what unique S and S2 respectively. Finally, the atom S2))) is only part of the set S2) but none of the sources
20

---

Published in Transactions on Machine Learning Research (05/2023)
individually. This constitutes an information contribution that cannot be realised by any of the two variables unless they are considered jointly, capturing the synergistic information.
This PID, indeed, serves its aim to partition [(T je) for any number of sources |J] into the realized atoms via the part-whole relationships. For example, in the case of two sources, we arrive at the following partition (Gutknecht al. 2021 Williams & Beer 2010):
1(T: S1,
-(()))))))())))))-((((((())))))) ((((S) - 1(T: S1) = ((((S)),((_)))
+ ((((S:)))
I(T: : (2) [T((S)), (S:)) ( +[{(S))))
Each alom is represented with a string of zeros and ones referred to as a parthood distribution (Table I Gutknecht et al.] (2021) then uses these parthood distributions to build the structure PID for any number of sources. We have alrcady seen that these parthood distributions follow two principles: (i) none of the PID atoms are part of the (ii) any PID atom part of the joint set of source variables. However, an additional principle arises from the nature of the information contributions and is vital when considering more than two sources: The principle states that whenever atom is part of the information provided by a collection Sa. then it must be part of the information provided by any collection containing Sa.
Hence, each PID atom is associated with a parthood distribution that is formally a boolean function @ ((II....,n)) - {0,1} satisfying three principles: (i) no information is in the emply set, i.e. (00) 0, (ii) all information in the full set, i.c $((1...., collection a is provided by any of its super- colleccoll,s,noss, which equivalent to saying that is monotone. An example of such association is present in Figure
1, and (ii) any information that provided by
Atom
(0)
{1}
{2}
(3)
(1,2
{1,
1,3)
f2,
()
(1,
Boolean function
I(({)}))
0
1
0
0
((10))(2,3)) 3)))
0
1
0
0
1 1
Su(((1))22333)
Figure 8: Parthood distributions are isomorphic to PID atoms In this figure, use the shorthand notation by dropping the S's and resorting only to the indices of source variables. The first atom II((1)) is the unique information of Sj. This information is part of S1 (i.e. {1}) and by principle (iii) any superset of (1). The boolean function fu((y) realized from the parthood distribution maps (1} and all its supersets to whereas the rest are mapped to zero. The second atom [I(((1), (2) the information that is exclusively redundant between S, and the joint source variable of S2 and Sg, but not redundant between S₁ and any strict subsets of {S2. Sz). This can be realized from the parthood distribution as the information is part of (1) and by principle (iii) all its supersets, however, additionally this information is also part of (2, 3). Finally this parthood distribution is packed into the monotone function boolean function (1(((1)((2333)))
Therefore, the number of PID atoms for n sources is equal to the number of monotone boolean functions over P((I...., minus 2 since the two monotone boolean functions +(a) 0 for all and (aa = for all a are excluded by principles (i) and (ii), respectively Gutknecht et al. (2021). This entails that the number of atoms grows super-exponentially numbers Beckurts & Dedekind (1897).
the number of monotone boolean functions, i.c., as the Dedekind
A.1.2 Equivalence between parthood distributions and antichains
In this section, we explain the equivalence between the parthood distribution and antichains that has been demonstrated by Gutknecht et al. (2021] Section 2). An information atom can be equivalently referenced by either its parthood distribution + as I[ Sp)) or its corresponding antichain a as II S.). This equivalence given by the facts that (i) the parthood distributions P(fl.... + (0, 1} are monotonic, iee., they fulfill the relation C b + (a) < ((b), and (ii) antichains succinct. representations of such
21

---

Published in Transactions on Machine Learning Research (05/2023)
monotonic boolean functions. To gain deeper insight into the equivalent descriptions of the atoms, we will thus address these two points one by one in the following.
Gutknecht et al.] introduces the monotonicity of the parthood distribution as a constraint in its defini- tion (Gutknecht et al. 2021 definition 2.1.). In fact, one can see that this monotonicity is in line with an essential property of mutual information: The mutual information I(T : Sa) between the target T and a subset of sources Sa is contained in the mutual information [T': Sb) whenever b is a superset of a. This fact is result of the chain rule of mutual information and the non-negativity of (conditional) mutual information [Cover& & Thomas 2006) I(T:: Sb) I(T:: Sa. Soia) I(T: Sa) +I(T' SbialSa) 2 I(T:: Sa) which implies that the correlation between Sa and T still persists in the correlation of S' and T and in fact only additional correlation can emerge with added sources. Thus when thinking about atoms that decompose the mutual information, it is only natural to evoke the constraint that if an atom II is part of the mutual information SS), i.e., ((a) = 1, it also has to be part of T(T : So), L..., {(b) = whenever a C b, which is exactly the property of monotonicity: a S b = ((a)= I ((b).
So far represent them in a succinct way as antichains. To this end, note that a boolean function is uniquely defined by the set of inputs ([[[]]]= == fa E P(fI..... of monotonicity, this representation can be compressed even further: Any sets in [-((()] which are supersets of other sets in [-[[[1] need not be recorded, as these are forced to map to "1" by monotonicity. Thus, a monotonic boolean function can be represented by a S [[]], constructed by removing all sets in ( which are supersets of others. The resulting set of sets a then contains only sets which are incomparable
that parthood distributions are monotonic boolean functions and we will discuss how to
f(a) =} that are mapped to "1". Given the constraint
given the partial order of set inclusion, which are referred to in the literature "antichains" (Williams &
Beer 2010
Therefore, each PID [(T': S atom identified with a parthood distribution & can be represented by the antichain representation of Φ, which provides an equivalent labelling of the atom as :S).
A.1.3 The lattice structure of PID using redundant information
In this section, we will discuss the well-known redundancy lattice and how this lattice structure of PID (Williams & Beer 2010) arises when using redundant information. First, recall that redundant information ITT : is the information that is part of all Sa for which ((a) = 1. This means that [-(T: : Sa) is part of all the mutual information terms I(T: Sa)) where (a) = 1. This part-whole relationship can be exploited to formulate a partial ordering of the general redundancies (T S.). We say that that a redundancy I[T Sq)) is contained in another redundancy [T S if the redundancy [n(T S..) is necessarily
part of all the mutual information that [n Sy) is part of. This implics that any part of the information contained in (T Sp) is also necessarily contained in In(T: : Sv) which forms the basis of this ordering. This ordering of the redundant information (T S can be expressed in terms of parthood distributions as (Gutknecht et al. 2021 Subsection 2.ii)):
C V <> (o(a) 1 (a) for any S n)......))..
(10)
Using the equivalence between parthood distributions and antichains described in Appendix A.1.2 this partial order can be similarly expressed in terms of antichains as (Williams & Beer 2010
a 1 3 Vb E 3 3a E aa
(11)
This partial ordering defines the algebraic structure of the "Redundancy Lattice". We can now define the PID atoms as the partial differences on this lattice, i.e., the atom I(T : Sa) contains all the information that the redundancy (T :4)) has in excess of the sum of all atoms below & on the lattice. Conversely, this allows the redundancies [n Sτ) to be the sum of all the atoms that are identified by a parthood distribution that is less than or equal & since all of the information in these atoms is contained in I[T Se) via the redundancy ordering. This can be equivalently expressed in terms of antichain (due to the aforementioned
22

---

Published in Transactions on Machine Learning Research (05/2023)
ordering) yielding the system of equations discussed in the main text:
[T Sa)= = Σ I[T Sg))
(12)
This system of equations states that for any node a of the "Redundancy Lattice", there is a redundant information that is the sum of the atom I[[ SS) for any 3 3 a. This system of equations can be inverted to compute the amount of information in each individual atom using the Moebius Inversion due to the distributive structure of the Redundancy lattice (Williams Beer 2010
A.2
Mathematical proofs
A.2.1 Proof of bounds on representational complexity by coarse-graining
To make representational complexity applicable to settings with more source random variables, we propose coarse-graining, i.e, combining source variables to form fewer, but higher dimensional, "super variables", as suitable procedure. As a first step, we clarify how the new coarse granne- variables are constructed from the original variables using a course-grain mapping.
Definition A.1 (Coarse- Grain mapping). For ñ E Nso and n < n, an n-to-n coarse-grain mapping f: (1,1...., n) - f1....,f) n is a surjective function that maps variable indices to fewer coarso-grained variable indices.
We write the pre-image of the coarse coarse-grain mapping for subsets coarse- gratied source indices à C (I,...,ñ as f- = ii E (1.....n) (() E a}.
Furthermore, we write sets of random variables indexed by the set of indices P((I,.... ,..., nI) Sa {S a} and finally sets of sets of random variables indexed by sets of sets of indices P(P((L...,,n))) as S. = (Sa/a E a).
Using this notion of a coarse-grain mapping, we can define what a coarse-grained random variables is.
Definition A.2 Stears-Grrained Random Variable). Given a vector-valued random variable S == (S Sw) and an n-to-ñ coarse-grain mapping f. the coarso-grained vector-valued random variable $ is defined as 5 ( Sa), where the elements S, = S--1((] = (k) = i are themselves vector-valued random variables, called coarse-grained variables, partitioning the n original variables into ñ variables.
Example A.1. Given four source variables S = (S1, S2, Sz, S1), the mapping
1, ie (1,2)
f (1,2,344 + [1,2). it
2, i i [3,4]
defines the two coarse-grained random variables 5, = =(S, and S2 = SS,,S4).
In general, a coarse-grain mapping can produce coarse-grained variables consisting of different numbers of original variables. An important special case, however, is that of the uniform coorse-grain mapping, which always maps source variables to one d-dimensional "super-variable"
Definition A.3 (Uniform coarse-graie grain mapping). A uniform coarse-gran grannn mapping of order d E N such that d/n given by f (I......n) n) + (1....,n/dd) n/d), i [(i - 1)/dj 1, where [+] refers to rounding down to the nearest integer.
Having defined the coarse-grained variables 3, the next question that arises is how the PID atoms of the original variables can be combined to form the coarse-grained PID atoms of S.
Theorem A.1 (Coarso-graining of PID atoms). The PID atoms IT(T S8) of the coorse-grained source variable S are composed of the PID atoms I[ S.) of the original sources S as
IT (T: 56) = Σ [T
(13)
-1
23

---

Published in Transactions on Machine Learning Research (05/2023)
Proof. A PID atom [T: ST) is part of the coarse-grained PID atom II (T S3)) exactly if it contributes the same mutual information terms (T Sa) with the coarso-grained variables Sa. Since the identity
I (T: :T:5a) Sa) I (T Sy-1a)
follows readily from the definition of the coarso-grained variables, we find that if (T So) is part of I [T: Sry-a)))) i.e,, (a)= = 1, it is also part of / (T': Sa) in the coarse-grained picture, and vice-versa. Thus, the parthood relations of (T' S4) with regards to the coarse- coarse-grained grained variables are captured by the parthood distribution & = & f-1 which leads to the conclusion that must be part of the coarse-grained atom II (T: 82). The coarse- grasind atoms must now be the sums of all original atoms which contribute the same coarse-grained mutual information terms; thus the theorem follows.
This coarse-grained PID naturally gives rise to both lower and upper bound the representational complexity of the original variables. As first step, we show how representational complexity can be equally computed from the parthood distribution of an alom.
Lemma A.1 (Computing the degree of synergy from parthood distribution). The degree of synergy of the atom indezed by the parthood distribution & is given by m( = min |a|.
Proof. The parthood distribution corresponding to an antichain a the boolean function that maps all a E a and all supersets thereof to one. This means, in addition to all sets a, that - the fibre of under D contains only sets a' a, for which |a'| >> |a| and which thus have no influence on the minimum cardinality of sets:
m(a) = min (a) = min aa = min |a| =: m()).
(14)
aEa
[(I) #(a)=1
The next step in our quest to prove bounds on the representational complexity is to prove bounds on the degree of synergy m.
Lemma A.2. Let & and & refer to the parthood distributions of an alom and course-grained alom, respectively. I[T So) is part of ITT S4), then the degree of synergy m(() is constrained by the degree of symergy m(0) as m( S m( S (²), where the upper bound holds for a uniform coarse- gurrs-ggriining of order d.
Proof. Lot the atom IIT 54) be part of the coarso-grained atom II( Sz). follows from Equation 13 that $- $ o f-1 and therefore
m( = min | d()==
min
a
S min (()==1 ((a)=1
min
[f(a])
|a = =m()).
$
where the fact that [(aa] < |a| has been used. Note similarly that for uniform coarse-graining of order d, (a)( 2 |al holds, hence finds lower bound to m( as
1 m d
m m(Φ) min [f(a)] ((a)=1
min |a| (()==1
From the bounds on the synergistic degree, the bounds on the representational complexity which is a weighted average of synergistic degrees are straightforward to derive.
24

---

Published in Transactions on Machine Learning Research (05/2023)
Theorem A.2. The representational compleaity of a vector of sources S (Si,..., Sp.) with respect to some larget T is bounded from below by the represenlational complexity of any coorse-graining S
C(T: S
(15)
Proof.
C(T :) 1(T': 5)
Σ
ITTT: (6)m(()
$
1
Σ
Σ
11 [T: S4) (T)
5 m())
(Theorem A.1
I(T 3)
Σ
Σ
II [T: Sq) m())
(Lemma A.2
1(T ()
0:0of-
1
Σ
IT (T: S m())
T(T :S)
+
C(:::
Theorem A.3. The representational complexity of vector of sources S = (Si...., S,) with respect to some target T is bounded from above by d times the representational complexity of a uniform coarse-graining of order d
C(T sCC(::))
(16)
Proof.
C(T:
Σ
I[T m())
1(T 3)
1
Σ
Σ
(T': St) (p))s m(Φ)
(Theorem A.1
I(T: 8)
à
4:bof 1=
Σ
Σ
IT: Sa) m(b)/d
(Lemma A.2
T(T (
o:0of-1
1
IT(T: So) m())
dT(T: :))
C(T :S)
1
d
A.2.2 Proof of representational complexity of one-hot encoding
In DNNs solving a classification task, the output labels are typically encoded a "Onc-Hot Encoding", which there are many neurons classes with only the neuron corresponding to the correct class being while all others are zero. Here prove that all such encodings have representational complexity of C = 1. Definition A.4 (One-hot encoding). The one-hot encoding Y of categorical variable Y with finite ordered alphabet Ay = .aa.y,,..,,,... is the image of the bijective mapping
Ay + Ay C (0, 1) + J = (6y. ) (0,
0),
J+1
25

---

Published in Transactions on Machine Learning Research (05/2023)
where 6y3y is the Kronecker Delta.
In what follows, we recall the concepts of local mutual information i, local I redundancy in and its additive decompasition into local informative redundancy को and local misinformative redundancy क These information functionals are needed in proving that any one-hot encoding has representational complexity equals to one.
Definition A.5 (Local information and their informative and misinformative parts). Let ''' be the target variable and 5 be the set of sources. Then, we have the following:
The mutual information 1(T S) is, in fact, the expected value of the local mutual information i(t : s) as follows:
p( Br,a [i(l 8)]
:TT=S):= = Σ p(t loga (()
1
The local mutual information i(t 8) can take negative values and so it decomposed into non- negative informative i (1: 8) and misinformative it *) parts as follows:
8 (f: = loga (G)
loga (s((s
PP (t)
logz 1
i- (t: 8) loga
(t)ss
Pst |t)
The 1 redundancy (T' S.) is in its turn the expected value of the local redundant information (x s,) as follows:
(T Sa)= Σ P(Ins) (() P (tn aoca Fa) (() [1 (Naca Sa)]
Er, In (t: sa)l
The local 1st redundancy (t 8a) can also take negative values and so it is decomposed into nonnegative informative exxt (t: 8,) and misinformative in (t 8.) parts follows:
1
1
ex+ in (1: 8a) = logg 1- IP (Naca 5a)
log
ps.. (8a)
(t)
in (t: sa) log2 (t) PP (tn
log
aaa 5a)
(S.
Proposition A.1. The misinformative part of the redundancy I=x (T: S.) vanishes if there exists a mapping J:T S from the largel to the sources.
Proof. If there exists a function f : + S, all conditional probabilitics of the form PS.,ITSS t) = Ps..IT(( ((San n San n... n...) JUUUU ... t) are equal to either one or zero. Thus, 1ax (T S.)
- -s, (og, loga (9. t) = 0 for any a.
To uniquely determine the label from a one-hot representation, it is sufficient. to observe the neuron that is equal to one. However, gets the same information by observing all neurons which are equal to zero, since, by exclusion, the last one then has to be one. Let aj := Hi,().... - 1.j + .....n]] be the antichain describing the redundant information between the j-th source and the rest of the sources taken together. Further, let ((Y: Y) describe the mutual information between random variable Y and its one-hot representation, which is trivially equal to its entropy H(Y) due to the construction of Y
Lemma A.3. The size of the local TEX redundancy 8 (1) Tay) is logg pv (v).
26

---

Published in Transactions on Machine Learning Research (05/2023)
Proof. Since the one-hot representation is a bijective function of the variable, Proposition A.1 implics that the misinformative part of the redundancy vanishes. The local T redundancy in question then amounts
to iक (M; Ba,)) = -xct
in (yy : Da,) = - logg Py (y) U (y) n... n 95-1 O 95+1 n... (.)) = -log2 PY Uy) =
- log, Py (v).
The atoms are ordered on a lattice by the partial ordering relation between atoms with antichain a and B being given by (Williams & Beer 2010
a 3 3 + vb 3 3a E a such that a S b.
(17)
The atoms are, then, computed as the Moebius inversion of the corresponding redundancies on the lattice, which is referred to throughout the literature as the Redundancy Lattice.
Lemma A.4. The degree of synergy m E N increases monotonically on the redundancy lattice, i.e., a 8 = m(a) S m(3).
Proof. Let a, E ((P((1....,,))) be two antichains on the n redundancy lattice such that a I B. By definition of the degree of synergy (Equation (4), there exists a set b 3 such that m(3) = [b). For this b, it follows from the definition of the partial order of the antichains that there must then also exist a set a E a for which a C b. Thus, 8 + m(a) S la s |b| = m(3).
Theorem A.4. The representational complexity of a categorical random variable Y and its one-hot repre- sentation is equal to one.
Proof. The local mutual information of the event (T S = 5) amounts to i(y () = log, (pr (v)) and is thus equal to the local redundancy ( :o,,) (Lemma A.3) Because all local atoms are non-negative
(Proposition A.I. D. all local atoms π with antichains / succeeding ay must be zero. Since m(a;) 1 and m(3) = 1 for all B N a (Temma A.4) the representational complexity of the one-hot encoding is C = 1. J
A.3 DNN implementation
A.3.1 MNIST feedforward deep neural network
The MNIST networks analyzed in this paper are fully-connected feed-forward deep neural networks with quantized activation values, but float-precision weights. These networks are trained the 60000 28x28 grayscale pictures of handwritten digits of the training set of the MNIST dataset (LeCun et al. 1998). To get better statistics for the test error as well as better estimates of iiformationnthhorede quantities on the test set, the additional 50000 test samples of the QMNIST dataset have been added to the 10000 regular MNIST test set samples (Yadav & Bottou 2019). The networks use tanh activation functions on the hidden layers, while on the output layer employing a softmax (for one-hot oulput layer) or sigmoid (for binary output. layer).
Three different networks have been trained with 20 different random weight initializations each: Figures 3 B and A A have been computed on networks trained and evaluated with eight quantization levels per neuron but with different output layer representations: While the networks represented in Figure 3 B have a ten-neuron one-hot output layer, the networks whose results are depicted in Figure 4 have only four output neurons of which each represents one bit of the binary representation of the numeric label. While the networks shown in Figures B and D share the same network architecture as the ones in Figure 3 they have been trained and evaluated with only four quantization levels to make the computation of the coarso-grained PID more efficient.
Finally, the MNIST network with randomly assigned labels, as depicted in Figure 7 has the same network structure as the default MNIST network and has been trained for 5 different weight initializations with a different random shuffling of the train and test labels each. There we use eight quantization levels.
For all networks, we used stochastic gradient descent with a batch size of 64, learning rate of 0.01 and Xavier weight initialization. The networks with ono-hot output representation employ a cross-cntropy loss, while for the binary representation, a mean square error loss was chosen. Parameters and accuracies of the networks are summarized in Table 2
27

---

Published in Transactions on Machine Learning Research (05/2023)
Setup
#runs
#quantization levels
Output rep. (#neurons)
train acc.
test acc.
Default
20
one-hot (10)
99.96(2)
95.0(4)
Binary Output
20
binary (4)
(9.59(4)
95.4(2)
Reduced
20
4
one-hot (10)
99.80(8)
94.7(4)
Shuffled Labels
5
8
one-hot (10)
30(3)
10.2(2)
Table 2: Network parameters and final accuracies for the three MNIST fully-connected DNNs referenced in
this paper.
A.3.2 CIFAR10 convolutional neural network
For the more complex CIFAR10 image classification task Krizhevsky 2009) we employed larger herd-forward neural network with convolutional layers. The 50000 32x32x3 pictures fed into three convolutional layers with 32, 64 and 128 filters, respectively and kernel size of 3. The layers employ max pooling and ReLU activation functions. Afterwards, the result is flattened to 2048 numbers and fed into the fully-connected part of the network consisting of four layers with a width of 128, 32, 5, and 5, respectively, with tanh activation functions. These fully-connected layers are quantized using 8 quantization levels. Finally, the results are fed into the 10 neuron one-hot output layer.
Similar the MNIST networks, the CIFAR10 network has been trained using stochastic gradient descent with batch size of 64, learning rate of 0.005, Xavier weight initialization and cross-entropy loss.
A.3.3 Quantization schemes
In order to limit the information capacity of the networks, the activation values have been quantized to very few discrete values. For evaluation, the quantized activations l are computed from the continuous values as
mmin
O
(18)
where L-] denotes rounding to the closest integer, the bin size is given by = (amax - (mimm//mmiinn 1) and mmin and max are the bounds the activation function. This quantization scheme has been chosen as it reproduces the bounds exactly, i..,, & = min/mmxx + miin/maxs and limits the rounding error to e/2.
For the training phase, the activation values of the forward pass are stochastically to make the training more robust. The stochastic scheme builds the deterministic scheme presented before in that it rounds the values to the same value. However, whether values are rounded up down to the nearest rounding point is no longer deterministic but given by probability scaling lincarly with the distance from the next two rounding points, giving
mod 1)
r
cλ + min where
(19)
8-amin
T auin mod 1).
and E [0 is drawn i.i.d. from uniform distribution for each neuron and each evaluation.
28

---

Published in Transactions on Machine Learning Research (05/2023)
A.4
Additional results
Reing Directed Local Differences
Layer La
Layer La
Layer Ls
k 1
Cr - 1((k)
k 2
k 3
k 4
k 5
H/
Sums of PID Atoms of Degree of Synergy m
L = II(T S.))
1
2
3
4
5
HH
Hh
HA
10a
10a
102
104
1
102
10a
Training Epoch
Figure 9: Summing up PID atoms of a fixed degree of synergy shows more information is stored across single neurons than estimated by the directed local differences by Reing et al. (2021). The figure compares the sum of PID atoms of certain degree of synergy (bottom row) with the directed local differences (top row) on three layers of the MNIST classifier network. Note that both decompositions for given layer sum up the total mutual information with the label.
29

---

Published in Transactions on Machine Learning Research (05/2023)
Repr. Compl. C
La train
4
Lg test
L4 train
3
L4 test
2
L5 train
L5 test
1/
102
10
Training Epoch
Figure 10: Detailed comparison between the average representational complexity over runs for the shuffled-label task described in Section 4.5 and Figure 7 Shown are only the averages, but not the confidence intervals for better visibility. is apparent that on this task that can only be solved by memorization of the random labels, the representation of the train dataset much less synergistic than the one of the test datasct.
A
B
train
Acc.
Repr. Compl.
Repr. Compl.
test
0
2
2
01
102
10a
L3
L4
L5
1/1
1/
102
104
0
1
102
10a
Training Epoch
Training Epoch
Figure 11: Representational complexity computed on the train (A) and test (B) datasets follow the same general trend. (A) is copy of Figure 3 and was put here to allow casier comparison between the results on train and test datasets. The setup for this experiment is described in the caption of Figure 3 but also in Table 2 The comparison between train and test datasets can be found in the main text in Section 4.5
30

---

Published in Transactions on Machine Learning Research (05/2023)
Run 0
Run 1
11
3
train
Acc.
Acc.
test
0
0
2
10
10
01
10
10
L3 train
L3 test
H/
L4 train
Run
Run 3
La test
Repr. Compl. C
L5 train
Acc.
Acc.
L5 test
2
10
10
10
14
Run 4
Run 5
3
Aec.
Acc.
0
2
102
10
102
10
HH
102
104
0
102
104
Training Epoch
Figure 12: Individual runs of the default MNIST setup show consistent deviation between the representational complexity computed on the train and test datasets starting at around 100 epochs into training. Shown are the first out of overall 20 training runs with the same settings but different seeds.
31


Reviews:
[{"review_id": "h3LOsa6Cwy", "reviewer": "Reviewer_jPSX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think the paper is well written, and has an audience at TMLR.\n\nI felt as though the paper did a good job motivating the measure of representational capacity and giving enough background that people could follow and have a good sense of what was being measured.\n\nThe paper then reports on the measurements they took, for, admittedly small networks, but they acknowledge this and also use approximation schemes to measure larger networks as well.  All of the empirical results suggest and validate the claim that their measure of representational capacity is decreasing over time.\n\nIn terms of weaknesses.  I suppose one thing I would have liked to have seen is whether it would be possible to break this observation.  For instance, if we expect that the final prediction is classification for which we expect a one-hot like set of activations at the logit layer, I would perhaps naturally assume the earlier layers would also be driven towards lower representational capacity.  It seems like one way you could try to break this would be to insert a fixed random rotation matrix between the readout logits and the resulting class prediction.  I.e. force the final predictions to be distributed.  (I'm assuming here that such a representation would necessarily have a higher representational capacity, but am not certain about that).  \n\nAs it stands the paper introduces a new measure and measures it and sees that it shows the same behavior across several different networks, which I can't really complain about, but also feel a bit as though I'm left not understanding what the implications or inferences I should draw from those observations are.  I don't really have any sense of why this should happen, or under which conditions I shouldn't expect it to happen.  If you could add some experiment, toy or otherwise that demonstrated different dynamics for the representational complexity, it seems like that would add a lot to the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper uses partial information decomposition in order to define and then measure a measure of representational complexity.  Intuitively this measure is meant to capture how many neurons need to be observed in order to extract useful information.  For something like a one hot encoding you'd expect their measure of representational complexity to be 1, and would go up as a representation becomes more distributed.  This measure itself is defined by a weighted sum over all of the terms in a partial information decomposition, something that grows super exponentially.\n\nThe paper then measures their proposed complexity measure on some simple networks and shows that the representational complexity decays over time, this seems to occur in a few different networks.  Given computational challenges, their measure can't be computed exactly on anything more than 5 neurons, but for larger networks they discuss and present some approximations which back up the other findings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please include version of the figures that report the representational complexity on the training set as well as on the test set, I'm curious whether there are any differences there.\n\nPlease also seriously consider including a scenario in which the representational complexity *doesn't* decrease over time, either by inserting some kind of strange fixed bijection at the readout layer or by some other means.  I think that would add a lot to start to formulate hypothesis about why this behavior might be happening or what implications it might have.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w4pSx2TI5A", "reviewer": "Reviewer_MWVV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of manipulating Mutual Information in order to overcome its inherent limitations when applied to neural networks is interesting. However, despite the title, it is hard to find \"rigor\" in this paper or a conclusive statement of any sort (beyond the plots). \n\nThe paper seems based on two assumptions: Mutual Information (MI) measures information and it is limited (as measuring information, it should satisfy a set of properties that are enforced upon it via this decomposition). This is simply $\\textbf{not}$ true: Mutual Information appeared as a solution to a very practical problem in communication. As a convex functional of measures, it seems to satisfy a series of properties that are useful in a variety of settings (properties that follow from the Kullbacak-Leibler divergence of which the Mutual Information is a specific instance). It has never been claimed anywhere that Mutual Information should measure information (even because the matter is more philosophical than mathematical at this point, what is even ``information''?). One is thus not measuring information in the layers but rather how much they depend on the input (seeing MI as a measure of $\\textit{dependence}$, following the axiomatic approach introduced for instance by Rényi in his work published in 09/1959 and titled \"On measures of dependence\"). Also, later on in the paper (Appendix A.1), the word \"predictability\" is associated with Mutual Information which is also false in general. The property being used is essentially just \"Data-Processing Inequality\" which is a consequence of convexity and the amount of stochasticity induced when applying Markov Kernels (intuitively, if I apply more randomness e.g., adding noise, then the variables will depend less on each and Mutual Information decreases). This is true of $\\textbf{any}$ divergence measure which is a convex functional. In order to make such a claim one would have to introduce a notion of \"predictability\" and provide results that show how the MI is involved in such an operational problem. This assumption seems relevant in multiple areas of the paper and in particular Section 3.2. \"We propose that this difficulty of retrieving information may be quantified by considering the minimum number of sources that are needed jointly in order to reveal a piece of information\". This is very informal, attaches meaning to MI, and seems much more related to deterministic mappings (e.g., how many \"neurons\" seen as mappings, do I need to recover the input exactly?) than to MI, which is a quantity measuring dependence in stochastic settings and, by extension, also to deterministic mappings in $\\textbf{discrete}$ settings. \n\nI find most of the contributions under-explored. The Representational Complexity is introduced in (5) but not properly defined. It is unclear what type of object it is, and what properties it satisfies. Is it an information measure itself? What properties does it satisfy? \nIn the \"Example applications of representational complexity\" the intuition is nice, but there is a lack of formality and rigor. What are the details of the computations of $C$ for the various setting? How can a reader (or in this case, a reviewer) check them? \nIt also seems that most of the limitations of using MI still persist. What happens to $C$ when $T$ is a deterministic function of one of the $S$'s? Does the quantity blow up? Is it not defined? What are the measurability requirements necessary to define $C$? When is it well defined? When is it not?\n\nA comparison section is missing: why is this Representation Complexity better than other approaches? Why is Rosas' paper mentioned in the Limitations but not in related work? Why is there no direct comparison with the decomposition provided in that work (which, at least at a glance, looks rigorous)? \n\nIn order to provide a rigorous and theoretically sound decomposition one should at least formally compare with (a few) others and argue why their approach is better/more fitting. Why have other options not been explored? Otherwise, it feels arbitrary. A few examples don't make for a theoretical argument but they can, at best, back a theory up. If the approach is presented as something meant to be used in practice (that would be perfectly fine as long as it is thoroughly justified with toy settings and real-world settings) but then it cannot be employed because of big computational issues then the theory should be much stronger in order to push the community to address said limitations. Otherwise, I see no real contribution in introducing a practical approach that cannot be used in practice. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors leverage a decomposition of Mutual Information quantities (called Partial Information Decomposition) to define an interpretable measure of the complexity of Neural representations. This measure is called \"Representational Complexity\" and is computed in some toy examples as well as empirically in some deep neural network models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest conforming to standard notation and use $I(T\\textbf{;}S)$ rather than $I(T\\textbf{:}S)$.\n\nI believe that the section on PID should be expanded as now is quite short and not sufficiently self-contained, perhaps moving a more thorough exposition to the appendix. The space gained should be employed to expand upon the theoretical analysis of the quantities introduced with a formal treatise of examples. Some examples in which the random variables are continuous should be included. The behavior of $C$ in extreme settings should be explicitly written (as the ones proposed in the \"Strengths and Weaknesses\" part of the review). \n\nMore comparisons should be advanced in order to justify the choice of (5) rather than any other approach which is already present in the literature. \n\nSince the code cannot be shared (although I don't see why one cannot make a novel anonymous GitHub profile just to share the code for the purpose of the review) more details on the implementations for Section 4 should be included. \n\nThe paper should either be presented as a theoretical treatise of some new quantity or as experimental. Wherever the choice falls, relative sections must be expanded. So far it is lacking both in theory and in experiments. Two simple examples are not enough without theory and if more cannot be provided (due to computational constraints, which is an important weakness of the approach, strongly limiting its applicability in concrete settings) then the theory should be stronger. \n\nOften unclear terminology is used to push formal arguments. I would suggest either removing said comments or bringing forward more formal statements. One example is in Section 4.2.Subsampling: the authors claim that subsampling suffers from conceptual flaws but I cannot follow the argument as the terminology used can mean multiple things and does not possess formally agreed-upon definitions (lack of rigor). Similarly, Appendix A.1 is very confusing. The first half of the \"proof\" is unclear: monotonicity is part of the definition of parthood distribution (Definition 1.(iii) of Gutknecht et al, https://arxiv.org/pdf/2008.09535.pdf) why would it need to be proven, in what looks like a tautological argument? The second half needs to be expanded with more formal arguments. Analogously, Appendix A.2 faces similar issues. \nTo understand the quantity in (3) the reader is referred to Appendix A.2 making it look like the topic is self-contained but it actually isn't. I would suggest expanding the Section.\n\nWhenever a reference is given for a formal argument, please refer to the specific result (rather than the entire paper, which can be a significant amount of pages long). For example, shortly before equation (7) Gutknecht et al is referred but it is not easy to find the specific argument. Similarly for Equation (8), which should be justified through Appendix A.1 and (Williams and Beer, 2010): where in that work?\n\nThe subsection on Coarse-graining (proposed as an alternative to Subsampling) also needs to be meaningfully expanded and justified. Especially since it is being used to provide experimental settings to justify the proposal of (5) as a reasonable quantity.\n\nI strongly suggest removing all the sentences in which MI is endowed with a meaning that does not belong to it: e.g., Section 3.2: \"Since the Mutual Information term $I(T:S_i)$ captures all information that the single source $S_i$ carries about $T$ ...\" what does 'capturing information' mean? \n", "broader_impact_concerns": "I do not see any particular ethical implication for this work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xGHxo265fr", "reviewer": "Reviewer_mLwm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The mathematical definition of the proposed representational complexity makes a lot of sense; I personally think this quantity has some potential to be a useful tool for the interpretation of the neural network architecture.\n\n- The quality of writing is satisfactory, and the visual illustrations are very effective.\n\n- The experiments are concerned with the \"quantized\" deep neural networks, without using any binning procedure; this is a good choice because binning obfuscates the message of the empirical result as it does not provide a good-enough approximation of the continuous quantities (as was discovered for information bottleneck principles).\n\nWeaknesses:\n- The paper does not clearly tell us where the proposed quantity is going to be useful for, and only gives some hints about it. The paper argues that understanding the decision of neural networks is important, but how exactly does this quantity help the purpose?\n\n- Perhaps due to the aforementioned lack of clear use cases, the empirical results lack a baseline to compare to. Given some target application, how superior is the proposed measure when compared with other competing quantities, in terms of the interpretability? Why are the empirical results a \"good sign\" regarding the utility of the proposed quantity?\n\n- Most of the formal theorem statements are hidden in the appendices, making the technical contribution of the paper very unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new measure, called \"representational complexity,\" which quantifies the number of intermediate-layer neurons one should inspect to make an inference about some target random variable. The paper then proposes several algorithms for approximating the represntational complexity when the number of neurons is large. Empirically, the paper reports that the quantity tend to reduce as the layer being inspected gets closer to the output, and as the training proceeds.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Please make it clear where the proposed quantity will be useful for. Validate the claim by comparing it to other baseline quantities.\n\n- Please make the technical contributions clear in the main text.\n", "broader_impact_concerns": "I do not think there is any need to include additional broader impact statements.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper defines a novel rich measure of complexity quantifying the representational power of subsets of a neural net nodes towards a target (label) variable. It also provides small scale experiments computing it across layers for very small networks. Approximation schemes are proposed for slightly larger experiments, together with additional numerics. Some comparisons with similar notions found in the literature are discussed.
The paper has generated a great deal of meaningful discussions with the reviewers, which I'd like to thank. In particular, the lack of "rigorous and precise statements" or the fuzzyness of certain notions has been emphasized, leading to improvements. I got the impression that numerous points raised by the reviewers were adressed, at least at some level.
From my own reading, I got on the one hand the same impression as all reviewers, of a paper containing interesting original concepts. Also reviewer mLwm, despite suggesting a reject, pointed "The mathematical definition of the proposed representational complexity makes a lot of sense; I personally think this quantity has some potential to be a useful tool for the interpretation of the neural network architecture." and I agree with the statement. On the other hand, it is also true that after reading some concepts / notions remain hard to really grasp and quite elusive (it took me some time to even understand the pre-requisite of what an atom is). Yet, I ended with the impression that the paper will spark the interest of some readers and may lead to interesting follow up. The writing could be a bit more rigorous to ease the reading, I believe, yet I think this style may fit others. I think that it is important for the authors to publish the final link to the codes to make the notions more "concrete" and, importantly, reproducible, especially for other specialists in the field to compare to their own favorite complexity notion (as the comparison work done here remains partial). A curiosity: I also wonder whether links could be established with refs https://www.mdpi.com/1099-4300/20/10/739, https://arxiv.org/pdf/2008.00520.pdf . Given all this, I believe that the paper is of sufficient interest to be published as such.
Decision: Accept (poster)