Learning diverse causally emergent representations from time series data
Authors: David McSharry
Christos Kaplanis
Fernando E Rosas
Pedro A. M. Mediano
Venue: NeurIPS 2024 poster (2024)
Keywords: emergence
representation learning
Primary area: deep_learning_architectures
TL;DR: learning emergent features using representation learning

Abstract: Cognitive processes usually take place at a macroscopic scale in systems characterised by emergent properties, which make the whole ‘more than the sum of its parts.’ While recent proposals have provided quantitative, information-theoretic metrics to detect emergence in time series data, it is often highly non-trivial to identify the relevant macroscopic variables a priori. In this paper we leverage recent advances in representation learning and differentiable information estimators to put forward a data-driven method to find emergent variables. The proposed method successfully detects emergent variables and recovers the ground-truth emergence values in a synthetic dataset. Furthermore, we show the method can be extended to learn multiple independent features, extracting a diverse set of emergent quantities. We finally show that a modified method scales to real experimental data from primate brain activity, paving the ground for future analyses uncovering the emergent structure of cognitive representations in biological and artificial intelligence systems.

Learning diverse causally emergent representations from time series data
David McSharry" Department of Computing Imperial College London dm22230ic ac. uk
Christos Kaplanis* Google DeepMind kaplanis@google.. com
Fernando E. Rosas Sussex AI, University Sussex f. r.rosasdsussex.c uc
Pedro A.M. Mediano Department of Computing
Imperial College London
p wemeeeinooooo mediano@@mperial. ac. uk
Abstract
Cognitive processes usually take place macroscopic scale in systems charae- terised by emergent properties that make the whole 'more than the sum of its parts.' While recent proposals have provided quantitative, information-theoree to detect emergence time series data, it often highly non-trivial to identify the relevant macroscopic variables a priori. In this paper we leverage recent advances in representation learning and differentiable information estimators to put forward data-driven method find variables with emergent properties. The proposed method successfully detects variables that exhibit emergent behaviour and recovers the ground-trute emergence values synthetic dataset. Furthermore, show the method can be extended to learn multiple independent features, extracting a diverse set of emergent quantities. We finally show that modified method scales to real experimental data from several brain activity datasets, paving the ground for future analyses uncovering the emergent structure of cognitive representations biological and artificial intelligence systems.
metrics
Introduction
Cognitive processes usually take place in systems made multiple interacting parts, e.g. neurons composing the nervous system of organism. Importantly, cognitive processes themselves don't 't seem to take place at a 'microscopic' level of individual units, but at 'macroscopic' levels involving assemblies of several coordinated units [28]. Hence, when trying unveil the inner workings of cognitive system, it is crucial to able to identify relevant macroscopic
natural or artificial - variables that best characterise the corresponding cognitive processes.
The identification of macroscopic variables has traditionally been driven by intuition and expert knowledge. For example, the investigation of collective behaviour in statistical physics is based on macroscopic variables known as 'order parameters, which are typically identified heuristically and then used to describe phase transitions and other phenomena of interest [49]. Unfortunately, identifying relevant macroscopic variables is often more an art than a science, being heavily dependent on prior knowledge and expectations. Having automated procedures to identify relevant macroscopic variables of cognitive systems would open important avenues for investigating the inner workings of different cognitive architectures.
promising approach identify empirically useful macroscopic variables provided by unsuper- vised representation learning [38, 22, 52]. For example, information maximisation has proven to be
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

powerful objective for learning representations within neural networks [23, 38]. In this paper we combine this approach with recent breakthroughs our ability formally characterise emergent phenomena [43, 34], which have proven to be not only theoretically sound but also empirically powerful [30, 40).
Building on this literature, in this paper we leverage recently proposed metrics of emergence to identify representations that display emergent properties. Specifically, we propose an end-to-end differentiable architecture that can learn maximally emergent representations of multivariate time series data. Our results show that causal emergence speeds up learning of more complex features of the data relative to pure mutual information maximisation, and we demonstrate the scalability of our method through an analysis of real-world brain activity data.
Methods
2.1 Quantifying emergence
Consider scientist measuring system of interest composed of parts, and let X denote the state of part i time The information that the joint process carries together from to 1' can quantified by the standard Shannon mutual information / (X;; Xe), where X; (X² _;., X²)) is the joint state of the system at time
How can one characterise an emergent macroscopic variable of such system? Following Ref. [43], one can define emergent variables as satisfying two key criteria:
(i) Supervenience: there exists a function (or course-graining) f such that V, f(x)).
iii) Unique information: Vi holds unique predictive information about the future evolution of the system X₂ that cannot be found in the individual parts x,....,X? X; by themselves.
Critically, the unique information that V2 holds about X² can be rigorously quantified using the framework of Partial Information Decomposition (PID, [54]), and its recent extension to time series data ($ID, [33]). Emergence. therefore, defined the capability of a supervenient variable to provide predictive power that cannot be reduced to underlying microscale phenomena.
Quantifying unique information in high-dimensional systems can be highly non-trivial. Luckily, the &ID formalism allows us to derive simpler measures that provide sufficient criteria for emergence. In particular, it has been shown that the following sufficient condition for causal emergence [43]:
I(Ve: Ve+1)
Σ
(XA: V(+1) > 0.
(1)
Importantly, V is comparatively easy calculate, as it relies only pairwise marginal distributions and on standard Shannon mutual information. These key features allow the framework to be applicable on wide range of scenarios, as illustrated by the applications reviewed in Ref. [34]. Note that here we take t' = t + 1, but in principle any t' > is valid.
The reason why V O is only sufficient, but not necessary, condition for emergence is that in some systems multiple X; can have the same information about V++1, and hence the sum of terms I(X;: Vi+1) may 'double-count' information counting can be alleviated by discounting from Eq. (1) the redundant information between the Xt's about V4+1. Here do this using measure of redundancy known 'Minimum Mutual Information' [4], which yields the following 'adjusted' emergence criterion (Supp. Sec. C):
resulting in negative bias in V This double-
A = I[Vi V(41
Σ Ex(X Vi+1) + (n 1) min ((X V2+1) 0.
(2)
2.2 Model architecture and information estimators
Maximising emergence. Our aim is to establish an automated procedure to identify emergent macroscopic variables V, with respect to microscopic substrate X,. For this, we investigate parametric coarse-grainings V, fo( (X,) that can be optimised to maximise v via differentiable objective function.

---

I(Va Viaa)
heg
Ve
V.
Vici
Jg
Jy
X
Figure I: Architecture for calculating loss terms for learning causally emergent representations. A representation network fa applied to the data X; learns feature Vg. This feature is trained to optimise y made up of predictive and marginal mutual information terms estimated by ge and lig- respectively (left). A further critic k may be added to calculate the mutual information with another emergent feature V.A. to encourage the learning of a diverse set of emergent features (right).
key ingredient to maximising V is employing a suitable estimator of Shannon's 's mutual information. Although many estimators of mutual information exist [27, 29], most not differentiable, and thus not suitable for optimisation with standard representation learning architectures. Fortunately, a number of differentiable estimators of mutual information exist [39, 50].
We use the Smoothed Mutual Information "Lower-bound" Estimator (SMILE, [50]), which is one of family of approaches that formulates mutual information estimation and was specifically designed to address the issue of high variance in existing estimators such as NWJ [37] and MINE [5]. The SMILE mutual information estimator is given by
variational problem,
I(X;Y) Y) Ep(m.)) log p(r,y)
(3)
p(z)p(v)
2 Ep(an) (1, y)1 log Bp())(()) [clip (cootz.e), c- .,),, (X:Y),
where Ip a parameterised function that estimates the log density ratio log (p(z, y) (p(x))())))
clip(u, 4, u) = max (min (v, () and r 2 0 is hyperparameter. As T - co, 1 converges to the MINE estimation [5], but finite prevents the potentially exponential growth of the variance of the estimate with MI, which MINE suffers from [50].
Equipped with this estimator, we can now formulate our representation learning algorithm for causally emergent features. The architecture schematically shown in Fig. 1. Our main method involves three learnable functions:
1. A representation network fo, that learns a supervenient variable V, = (o()).
2. A critic for the macroscopic variable Io. that controls the estimation of [(V;; V4+1).
3. A set of crities for the microscopic variable heg that control the estimation of (X]: V(+1). The number of crities equals the number of constituent atoms of the system.'
We will demonstrate that the representation critic learn emergent features algorithmic dataset by maximising the following metric:
s sp, (g-))
Σ (X)
(5)
Alternatively, one could consider using single critic lig shared across all x We found this option to be computationally cheaper but less effective.
3

---

function, We refer to S as the emergence funciio,, and the first term in the RHS of Eq. (5) as the predictive mutual information [6] since (by the data processing inequality [15]) represents a lower bound on the joint mutual information between the past and future states of the whole system, T(X;; ). As a control condition, we also ran experiments with an objective function consisting of only the predictive information, removing the marginal mutual information terms. After training has converged, we ran 'post-hoc' tests to further verify that the learned feature is emergent by freezing the representation network fe. retraining the critics to accurately estimate / (V;; V(+1) and I [();; V++1). and calculating V/ A. this resulting VA is positive, we conclude the feature is emergent. For convenience, pseudocode for the algorithm used train the model is provided in Supp. Sec. A.
Learning diverse emergent features. well-known fact about complex systems is that they can have more than one emergent property, and/or be described by multiple order parameters [25].
To learn a diverse set of emergent features, we consider a scenario where an emergent feature V.4 has been learned and its corresponding parameters 04 are fixed. Intuitively, our goal is to find new feature that is substantially different from the existing one by incentivising it to be statistically independent from V.A. To implement this, add a penalty term (f (X,) x) fe (X))) to the objective function and include fourth learnable function to our method: a critic ko that estimates this mutual information, also using SMILE.
Improving training stability. An important point to note of the proposed method is that there is an adversarial relationship between the microscopic critics lig+ and the macroscopic critic ge through the representation network fe. reminiscent of the adversarial learning dynamics in GANs [16]. In essence, the former pushes the representation network fe to maximize information about the system's 's state, while the latter pushes it to minimize it. This could lead to potentially unstable learning dynamics and various failure modes- for example, representation network could "trick" the microscopic critics into reducing their MI estimate by exploiting their mistakes, rather than genuinely reducing T(X;; V4+1), resulting in an artificially inflated value of V
Fortunately, unlike in GANs, it is not harmful for the critics to be overpowered compared to the representation learner because the SMILE estimation is bounded from above by the true MI [50]. Following this reasoning, we i) update the critics multiple times for each representation learner update, using a higher learning rate and more parameters; and ii) pre-train the critics before we start training the representation learner, so that they provide more robust MI estimation and better training signal. This results in slower, but more stable, learning dynamies for fe-
Another method to increase training stability is to use a dedicated choice of the architecture of the representation network. fo first linearly projects the input to a digger-ai space, then passes it through an MLP with residual ('skip') connections, and then projects it linearly down to the output vector V,, Because of the skip connections (and because of the auto-correlation of X,), at initialisation the output already contains useful information about the system and yields a high value of the macroscopic information (fo((.)) So xx+1) before 0 has been trained. Therefore, this allows us to reduce the contribution of this term in the objective function, and thus reduce the influence of 9 on fe.
2.3 Datasets
2.3.1
Synthetic datasets
Bit-string dataset. We evaluate our method for learning causally emergent representations by
applying it to sequences of random bit-strings of length n with two constructed temporal correlations:
1. The parity of the first 1 bits is auto-correlated across time, such that
1
IP o X = xx] = Parrity > 2
where @ represents modulo-2 addition.
In GANs this could lead to mode collapse [51].
"Atthough this could in principle lead to learned representations with reduced predictive information, in
practice we observe this leads to increased stability and only small decrements in L(V;; V++1).

---

2. The last (or extra) bit in the bit-string X" is auto-correlater across time, such that
P P(X = X.) Vextra
Since parity is synergistic function of the bits of a bit-string (i.e. cannot be predicted from any of the input bits individually [42], and since the parity predicts some information about the future evolution of the system, Vi = 01=1" X₂ is an emergent feature of the system.
Despite its simplicity, this dataset has two important advantages: there is known emergent feature (the parity), and one can calculate the mutual information and the emergence measure analytically." These properties will allow us to verify that the model has successfully extracted the expected emergent properties and that mutual information is being accurately estimated.
Conway's Game of Life. To further evaluate our method on more complex system, we apply it to simulations from Conway's Game of Life (GOL) - canonical setting for the study of emergence [1]. GOL is a cellular automaton on an N x N grid where each cell's state evolves based on simple, deterministic rules. We initialize a "glider" pattern [19] at random position within the grid, and this glider moves diagonally by cycling through four distinct states until it reaches a boundary and the simulation concludes. The dataset consists of multiple such simulations concatenated to form continuous sequence. To effectively capture the spatial dependencies inherent in the Game of Life, we replace the multilayer perceptron used in previous experiments with a convolutional neural network for the representation learner fo, allowing to exploit the 2D structure of the grid-based simulations.
2.3.2 Real world brain datasets
Primate ECoG dataset. We evaluate our method on a dataset electrocorticography (ECoG) brain activity data from a macaque monkey, originally reported by Chao et al. [10]. The dataset contains long time series of electrical activity measured with 64 electrodes placed across the monkey's 's brain surface. We minimally pre- pe-process the data by applying second-order Butterworth high-pass filter with a Hz cutoff, downsampling the signals to 300 Hz, and finally standardising the data before applying method to learn emergent features from it.
Human MEG dataset. We evaluate our method on a magnetoenceehhalggrppyy (MEG) brain activity dataset from healthy human participants, originally reported in four pharmaco-MEG stud- ies [36]. The dataset includes resting-state recordings collected before the administration of various pharmacological agents. MEG signals were sampled at 600 Hz with 0-300 Hz bandpass filter. For our analysis, we focus on the placebo condition (pre-drug administration) and train our model fe on data from multiple subjects, using each epoch to represent different participant capture features common across individuals. Data was pre-pross pre-processed following standard procedure [9] and standardised before applying method to learn emergent features.
Human fMRI dataset. We also evaluate our method on a functional magnetic resonance imaging (IMRI dataset from 100 unrelated human participants from the "minimally preprocessed" release by the Human Connectome Project [53, 20]. We performed additional preprocessing following Luppi al. [31] and calculated 100 time series capturing the activity of each of the regions in the Schaefer brain atlas [45]. We further standardise the data before applying our method to learn emergent features from it.
3 Results
In this section, we discuss results on learning emergent features on the synthetic and real world datasets introduced above, as well as comparisons to baselines and some ablation studies. All experiments can be run on a single A10G GPU in less than two hours.
*Specifically, the emergence metric is V = 1 Hu (7ppan) and the predictive information is / ((xx;
2 Ha ("/ooo)) H2(Yemm), where Hz(p) represents the entropy of a Bernuuli distribution with parameter
Here "'pritt "lextra 0.99, and show results for other " Textra values in the Appendix.
5

---

Full emergence objective
Prediction-only objective
1.0
1.0
V (estimated)
Emergence
0.5
0.5
Predictive MI
estimates
V (ground truth)
0.0
0.0
10
20
30
0
10
20
30
1.0
1.0
Parity bit
Decoding
0.8
0.8
Extra bit
accuracy
Chance level
0.6
0.6
2
4
8
10
0
6
8
10
Training steps (x 100))
Training steps (x10%)
Figure 2: The proposed architecture recovers ground truth emergent features. Using the emergence objective function (left column), the model finds the correct v value and is able to recover the known emergent feature (parity bit). Using only predictive MI as the objective (right column), the model fails to discover any emergent features.
3.1 Learning emergent features in synthetic datasets
Bit-string dataset. Results show that our proposed architecture can accurately estimate the ground- truth value of V in the synthetic dataset, confirming it is able to learn causally emergent representations (Fig. 2). To interpret the contents of the learned representation, we trained decoders with standard supervised learning to predict both the parity of the first n bits (the parity bit) and the last auto-correlated bit (the extra bit). We found that the parity bit could be decoded with high accuracy but the extra bit could not, contirming that the learned representation indeed corresponded to an emergent feature, For additional verification, we fixed the representation learner's 's weights 0 and re-trained both critics estimate both and V A. which were both positive and confirmed our results.
As expected, when the marginal MI terms are removed from the objective function (Fig. 2, right column), the model is longer able to obtain the correct V value and, interestingly, only the extra bit (but not the parity bit) is encoded in the representation. We hypothesise that, the absence of the regularisation induced by the marginal MI, the system's inductive biases lead it towards learning "low-order" (i.c. non-emergent) representations. Note that, despite having a constraint removed, the model without marginal MI loss unable extract the full predictive information of the system (which equals approximately 1.84 bit), showing that using the full emergence loss could incentivise the system learn features that provide information about the system's dynamics that would otherwise be ignored (we elaborate on this further in Section 3.3). We obtain qualitatively similar results with noisier version of the same data generating process (Supp. Fig. 5).
Conway's Game of Life. By employing convolutional neural network for the representation learner fo. we successfully learned emergent features from this dataset, evidenced by positive V value after freezing the representation network and retraining the critics (Supp. Fig. 8a). To interpret the emergent feature, we attempted to predict the glider's position and state from the representation. While the position prediction performed at chance level, the state prediction achieved an accuracy of 53%, significantly higher than the random baseline of 25% (Supp. Fig. 8b). This suggests that the emergent feature encodes information about the glider's state. When fo is trained with prediction- only objective, the state prediction achieved an accuracy of 35%, suggesting the learned feature encodes less information about the state of the glider when trained this objective.

---

First emergent feature
Second emergent feature
1.0
1.0
Emergence
0.5
0.5
V
estimates
0.0
0
2
4
2
1.0
1.0
Parity bit
Mutual
0.5
Extra bit
information
Bonus bit
0.0
0
2
4
2
Training steps (x10*)
Training steps (x (x10%)
Our method learns diverse set of multiple emergent features from the same system. Training two representation learners to learn independent emergent feature on the synthetic dataset. Both learned features were emergent (top row). The first learner (left column) yielded feature that has high mutual information with the system's 's parity bit (bottom left), while for the second learner (right column) it had high mutual information with the bonus bit (bottom right).
Figure 3:
These findings demonstrate that our approach can learn meaningful emergent features complex, high-dimennsonal systems like the Game of Life, capturing collective behaviors that are not localized to individual components.
3.2 Learning diverse features in the bit-string dataset
As our next step, we set out to test our algorithm to learn diverse features in the same bit-string dataset. For this set experiments we used the representation learner with skip connections described in Section 2.2, ran our algorithm as above until convergence, and fixed the representation learner's weights (denoted by 04).. We then trained a second representation learner with the regular y objective plus penalty term (VV; VAA) to obtain new feature that is statistically independent from parity bit.
Interestingly, this process revealed new, unexpected feature that was not originally designed (Fig. 3): the XOR of the parity and extra bit (which we will refer to simply as the bonus bit). This bit is emergent (since it cannot be predicted by any of the X; and is cuto-correlated, since both the parity and extra bits are), albeit with lower V than the parity bit which fits well with the result that our model did not learn this feature spontaneously. This shows the capability of our method to discover new aspects of a system under study, even the case of simple and explicitly constructed systems.
3.3 Comparison to baselines and ablation studies
Comparison with RNNs. In order to investigate whether the failure of the mrdiction-onf, baseline to learn emergent features was do with the capacity of the architecture, we trained standard RNN on the bit-string dataset using mean squared eтror (MSE) loss, with its hidden state serving as representation
decoding. The hidden state dimension matched that of our emergent feature network fo for fair comparison. As shown in Fig. 4, the RNN consistently learned the non-emergent extra bit but encoded negligible information about the two emergent bits, resulting in V A = This indicates that the RNN did not capture any emergent features.
7

---

RNN only
RNN & Emergence
1.0
1.0
2.0
Decoding MI
Predictive MI
0.5
0.5
Parity bit
1.0
Extra bit
Bonus bit
0.0
0.0
0.0
0.5
1.5
100
200
300
RNN Emergence Both
Training steps (x (x103)
Training steps
learner
Figure 4: Standard methods do not learn emergent features, and their performance increases when combined with emergent features.. The hidden state of an RNN trained the bit-string dataset has negligible V A. indicating no emergent feature learned (left). Accordingly, the mutual information between the hidden state and the extra, parity, and bonus bits shows that only the non- emergent extra bit is encoded (middle). Interestingly, representations learned by an RNN and by our method can be combined to yield better predictions of the future state of the system (right).
Combining emergent and non-emergent features. The fact that standard architectures learn representations that are predictive, yet not emergent, suggests they may be learning different aspects of the data compared emergent features from our method with representations from standard models would enhance the predictive performance of either. As proof of concept, trained both RNN and emergence learner fo on the bit-string dataset, and compared the mutual information between X₃ and:
method. To test this hypothesis, investigated whether combining
1. The hidden state of the RNN.
2. The emergent feature Vn learned by fe.
3. The concatenation of V/ with the hidden state of an RNN.
As shown in Fig. 4, the combined representation encoded more information about the future state of the system, Xy., than either the RNN the emergence learner alone. While this result is only shown on a very simple synthetic dataset, it suggests that our emergence learner extracts meaningful features that are difffrrnt from, and can be effectively combined with, other representation learning architectures. We hypothesise that standard representation learning techniques can more effectively capture microscale properties, and since method captures macroscale properties, both can be naturally combined to enhance performance in downstream tasks.
Ablations emergence loss. The main emergence loss function we propose in this work has three terms: predictive MI term estimating (V;; Vy.), marginal MI terms estimating / (Xf; Vy),, and diversity term estimating (V,A; Vi). In order understand the contribution behaviour of the model, we performed ablation studies removing each one of these. We describe each ablation in turn:
each term to the
1. As seen Sec. 3.2 (and later in Sec. 3.4), ablating the I(V;: Vy.) term from objective results in the network learning features that are not as correlated in time as they would otherwise be, but that still satisfy the criteria for emergence. Therefore, our proposed architecture with skip connections may learn slightly less emergent features, but with the advantage of more stable training dynamics.
2. Ablating the T T(X;; Ve) terms objective results in an algorithm that only maximises predictive information T(V;; Ve,,, akin to the Deep InfoMax method [23]. As seen in Sec. 3.1, and as expected, training without the marginal terms results in representations that are not emergent.
'We retrained both models using half the representation size, to ensure a fair comparison between all three cases.

---

3. Ablating the diversity term in the experiments in Sec. 3.2 and training multiple runs on the bit-string dataset results in the representation network learning the extra bit 91% of runs, and the bonus bit 9% of runs. This implies there are strong inductive biases towards which emergent features are learned, and it is thanks to the diversity term that we can reliably recover both.
3.4 Learning emergent features in brain activity data
Finally, we demonstrate the scalability of our method by learning emergent features in three types of nigh-dimensional brain activity datasets ECoG, MEG, and fMRI to evaluate its ability learn emergent features across different neural recording modalities and spatial scales.
ECoG data. For the ECoG dataset [10], it was crucial skip connections in the representation learner, since the method as used in Section 3.1 was consistently unable to find emergent features (despite their known presence in this dataset [43]).
With this configuration. our method is able to successfully learn emergent representations of the ECoG data (Supp. Fig. 9a). The resulting feature was verified to be emergent by a positive post-hoc V/ A test. Interestingly, using an Se with skip connections we obtained emergent features even when the macroscopic MI term [V;; Vi+1) completely removed from the objective function, which further simplified learning dynamics. Empirically, we found that this does not cause I (fe (X,): (xx++))) to decrease over training substantially, suggesting that using just the marginal MI terms good objective function to remove information about the parts while preserving information about the whole that present initialisation.
As final experiment, we also found that our method could learn a second emergent feature, also verified with post-hoc V check (Supp. Fig. 10).
MEG data. We extended our analysis to magnetoenceephalorrpphy (MEG) recordings from healthy human participants [36]. Training on data from multiple subjects, our method readily learned emergent features, as evidenced by a positive post-hoc V/ a check (Supp. Fig. 9b). The consistency of emergent features across different individuals suggests that our model captures fundamental aspects of neural dynamics common human brain activity.
Notably, emergent features were learned more easily from MEG data compared to ECoG data, potentially due to differences in spatial resolution and the nature of the recorded signals.
MMRI data. Finally, we applied our method functional magnetic resonance imaging (fMRI) data from 100 unrelated human participants [45]. Despite the coarser spatial and temporal resolution of [MRI, our method was able to learn emergent features, as indicated by a positive V value (Supp. Fig. 9c). This demonstrates that emergent dynamics are present even at the macroscopic scale of brain activity. this dataset, we also determined that standard MLP did not learn emergent features in its representation (see Supp. Sec. B.2 for more details).
The successful extraction of emergent features across all three datasets highlights the robustness of our method. It underscores its capability to uncover collective neural dynamics in complex, high-dimensional brain activity data, regardless of the recording modality or spatial scale.
Table 1: Our method learns emergent features in multiscale datasets of brain activity. Final values of Ve and V A as found during the training run of fo and post-hoc evaluation respectively. All values are greater than zero, indicating that in each case an emergent feature has been found.
ECoG
MEG
fMRI
Training V
0.8 + 0.12
3.0 + 0.30
2.7 + 1.3
Post-hoc VA
1.2 + 0.19
2.6 + 0.52
3.3 + 0.44

---

4 Related work and future directions
The method we have proposed here is part of a growing literature leveraging methods from information theory to enhance deep learning architectures, and representation learning algorithms in particular. A small, far from exhaustive list of some noteworthy examples includes InfoNCE [38], deep variational information bottleneck [2], 3-VAE [8], TC-VAE [11]. In future work, it would be interesting to see our method is competitive or can be combined with recent representation learning methods on standard benchmarks [22, 38, 52].
In practice, one the challenging aspects of this method is to overcome the instability of training to make sure a valid emergent solution is found (i.e. one where the post-hoc check still shows V > 0). One potential approach is to incorporate stabilisation techniques from the GAN literature (such as e.g. spectral normalisation [35]). Another potential approach is to mutual information upper bound (instead our current lower bound) for the marginal MI terms. Although mutual information upper bounds not common as lower bounds, there a few options available [13, 12]. In practice, we found both of these to be less effective than our skip connection method, but they remain promising avenue for future work.
There a growing body of machine learning research that engages directly with PID, the information- theoretic backbone of our emergence theory. For example, recent work has proposed differentiable redundancy measure [32], which has been used as an objective function to train deep neural net- works [21]. Alternatively, there are also methods that estimate redundancy using deep neural networks [26]. analogous to SMILE. Although none of these estimators can be directly applied to estimate the unique information that underlies the definition of emergence (Section 2.1), expect that extensions of these exciting developments will also open new possibilities in the study of emergence.
Finally, it is worth mentioning that there are a number of other approaches that focus on different aspects of emergence (see Supp. Sec. D). The approach presented here was chosen for two reasons: (i) its intuitive nature, focusing on how emergent properties arise from collective interactions that cannot be fully explained by examining components in isolation; and (ii) the existence of efficient, scalable, and differentiable proxies for its estimation. Constructing similar methods as the one proposed here for other metrics of emergence would be a challenging but extremely interesting line of future work.
5 Conclusion
Emergence, the phenomenon whereby a system becomes 'more than the sum of its parts', is an promising conceptual tool investigate cognitive processes artificial and biological systems. In this paper, we proposed a machine learning method for discovering emergent variables in time series data that leverages recent cnermation-thhoretee characterisation emergence [43] and advances in mutual information estimation from data with neural networks [50].
We first showed in a synthetic dataset that our method can estimate emergence and successfully discover known emergent feature. Interestingly, pure information maximisation objective struggled learn this feature, suggesting that our method facilitates the identification of complex features of the data. Furthermore, we also proposed slight modification of our method that can learn a diverse set of features from the same system. Finally, we also showed that our method can scale up to learn emergent features in real-world brain activity data.
Overall, method opens up a range of possibilities for the practical study of emergence, well as for other machine learning problems more broadly. For example, our method may be explored in conjunction with other representation learning algorithms capture aspects of complex systems that are otherwise difficult to learn. From an application perspective, we hope this method can be further leveraged in neuroscience reveal new aspects of brain function.
Software availability
Code implementing our proposed architecture and reproducing our key results is available at https: // /i thub. com/Inperiall-MND- wataaa/aaaalyyaaaaaaaallaaaallaa http:///cualllleeeeeeeeereerreee -erreeete-eeeeeeetttttaiioo
10

---

References
[T] A. Adamatzky. Game of Life Cellular Automata, volume 1. Springer, 2010.
[2] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. 1rXiv: 2016.
[3] L.. Barnett and A. K. Seth. Dynamical independence: Discovering emergent macroscopic processes in complex dynamical systems. Physical Review E, 108(1), July 2023.
[4] A. B. Barrett. Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Physical Review E, 91(5):052802, 2015.
[5] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and R. D. Hjelm. Mine: mutual information neural estimation. arXiv preprint /rXiv:/801.0400 2018.
[6] W. Bialek, L. Nemenman, and N. Tishby, Predictability, complexity, and learning. Neural 2001.
Computation,
[7] T. Bossomaier, L. Barnett, M. Harré, J. T. Lizier, T. Bossomaier, L. Barnett, M. Harré, and J. T. Lizier. Transfer entropy. Springer, 2016.
[8] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Understanding disentangling in 3- B-VA 1rXxv:8844003999 2018.
[9] R. L. Carrast-Harris, S. Muthokumaraswamyy L. Roseman, M. Kaelen, W. Droog, K. Murphy, E. Tagliazucchi, Schenberg, T. Nest, C. Orban, et al. Neural correlates the LSD experience revealed by multimodal neuroimaging. Proceedings of the National Academy of Sciences, 113(17)44853-448) 4858,,488,,00 2016.
[10] Z. C. Chao, Y. Nagasaka, and N. Fujii. Long-term asynchronous decoding of arm motion using electrocorticographic signals monkey. Frontiers neuroengineering. 3:1189, 2010.
[11] R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autocncoders. Advances in Neural Information Processing Systems, 31, 2018.
[12] P. Cheng, W. Hao, and L. Carin. Estimating total correlation with mutual information bounds. In NeurIPS 2020 Workshop: Deep Learning through Information Geometry, 2020.
[13] P. Cheng. W. Hao, S. Dai, Liu, Z. Gan, and L. Carin. Club: A contrastive ratiratio upper bound of mutual information. International Conference on Machine Learning, pages 1779-1788. PMLR, 2020.
[14] K. Conrad. Probability distributions and maximum entropy. Entropy, 6(452):10, 2004.
[15] T. M. Cover and J. Thomas. Elements of Information Theory. John Wiley & Sons, 1999.
[16] A. Creswell, T. White, V. Dumoulin, Arulkumaran, Sengupta, and A. Bharath. Gen- erative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1):53-655 2018.
[17] J. P. Crutchfield. The calculi of emergence: computation, dynamics and induction. Physica D: Nonlinear Phenomena, 75(1-3):11-54 1994.
[18] J. P. Crutchfield. The origins of computational mechanics: A brief intellectual history and several clarifications. arXiv preprint
2017.
[19] A. Dorin, J. McCabe, J. McCormack, G. Monro, and M. Whitelaw. A framework for under- standing generative art. Digital Creativity, 23(3-4):239-255 2012.
[20] M. F. Glasser, S. N. Sotiropoulos, J. A. Wilson, T. S. Coalson, B. Fischl, J. L. Andersson, J. Xu, Jhabdi, M. Webster, R. Polimeni, et al. The minimal preprocessing pipelines for the Human Connectome Project. Neurolmage, 10:105-124, 2013.

---

[21] M. Graetz, A. Makkeh, A. C. Schneider, D. A. Ehrlich, V. Priesemann, and M. Wibral. Infomor- phic networks: Locally learning neural networks derived from partial information decomposition. D.Xiv:3306.02149, 2023.
[22] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning, Advances in neural information processing systems, 33:21271-21286 2020.
[23] R. D. Hjelm, A. Fedorov, S. Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
K. Grewal, P. Bachman, A. Trischler, and
[24] E. P. Hoel, L. Albantakis, and G. Tononi. Quantifying causal emergence shows that macro can beat micro. Proceedings of the National Academy of Sciences, 110(49)19790-1199)) 2013.
[25] H. Jensen. Complexity Science: The Study of Emergence. Cambridge University Press, 2022.
[26] M. Kleinman, A. Achille, S. Soatto, and J. C. Kao. Redundant information neural estimation. Entropy, 23(7):922, 2021.
[27] A. Kraskov, H. Stögbauer, and P. Grassberger. Estimating mutual information. Physical Review E, 69(6):066138, 2004.
[28] M. Levin. The computational boundary of "self": Developmental bioelectricity drives multicellularity and scale-free cognition. Frontiers in Psychology, 10:493866, 2019.
[29] J. T. Lizier. JIDT: An informattin--theoretie toolkit for studying the dynamics of complex systems. Frontiers Robotics and AI, 1:11, 2014.
[30] A. I. Luppi, P. A. Mediano, F. E. Rosas, J. Allanson, J. D. Pickard, G. B. Williams, M. M. Craig, P. Finoia, A. R. Peattie, P. Coppola, et al. Reduced emergent character of neural dynamics in patients with disrupted connectome. Neurolmage, 269:1 1 19926, 2023.
[3]] A. I. Luppi, P. A. Mediano, F. E. Rosas, N. Holland, T. D. Fryer, J. T. O'Brien, J. B. Rowe, D. K. Menon, Bor, and E. A. Stamatakis. synergistic core for human brain evolution and cognition. Nature Neuroscience, 25(6):771-782 2022.
[32] A. Makkeh, A. J. Gutknecht, and M. Wibral. Introducing differentiable measure of pointwise shared information. Physical Review E, 103(3)x03214, 2021.
[33] P. A. Mediano, F. E. Rosas, A. I. Luppi, R. L. Carhart-Harris, D. Bor, A. K. Seth, and A. B. Barrett. Towards an extended taxonomy of information dynamics via Integrated Information Decomposition. arXiv:2109. 1:Xi::2009131866 2021.
[34] P. A. Mediano, F. E. Rosas, A. I. Luppi, H. Jensen, A. K. Seth, A. B. Barrett, R. L. Carhart- Harris, and D. Bor. Greater than the parts: review of the information decomposition approach to causal emergence. Philosophical Transactions of the Royal Society A, 380(222)))202222446 2022.
[35] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. arXiv: arXiv: 100.009572 2018
[36] S. Muthukumaraswamy. PharmacoMEG 1/f data, 2018.
[37] X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-53)
2010.
[38] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018
[39] B. Poole, S. Ozair, van den Oord, A. A. Alemi, and G. Tucker. On variational lower bounds of mutual information. In NeurIPS Workshop Bayesian Deep Learning, 2018.
12

---

[40] A. M. Proca, F. E. Rosas, A. I. Luppi, D. Bor, M. Crosby, and P. A. Mediano. Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks. arXiv preprini arXiv:2210002996, 2022.
[41] F. E. Rosas, B. C. Geiger, A. I. Luppi, A. K. Seth, D. Polani, M. Gastpar, and P. A. Mediano. Software in the natural world: A computational approach to hierarchical emergence. Preprint arxiv, 2402:v2, 2024.
[42] F. E. Rosas, P. A. Mediano, M. Gastpar, and H. J. Jensen. Quantifying high-order interdependen- cies via multivariate extensions of the mutual information. Physical Review E, 100(3):032305, 2019.
[43] E. Rosas, P. A. Mediano, H. Jensen, A. Seth, A. B. Barrett, R. L. Carhart-Harris, and D. Bor. Reconciling emergences: An information- approach to identify causal emergence in multivariate data. PLoS Computational Biology, 16(12):e108228, 2020.
[44] A. Rupe and J. P. Crutchfield. principles of emergent organization. Physics Reports, 1071:1-47, 2024.
[45] A. Schaefer, R. Kong, E. M. Gordon, T. O. Laumann, X.-N. Zuo, A. J. Holmes, S. B. Eickhoff, and B. T. Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cerebral Cortex, 28(9):305-311 -3114,,, 2018.
[46] T. Schreiber. Measuring information transfer. Phys. Rev. Lett., 85:461-466 Jul 2000.
[47] A. K. Seth. Measuring autonomy and emergence via granger causality. Artificial life, 16(2):179- 196, 2010.
[48] C. R. Shalizi. Causal architecture, complexity and self-organization in time series and cellular automata. The University of Wisconsin-Madison 2001.
[49] R. Solé. Phase Transitions, volume 3. Princeton University Press, 2011.
[50] J. Song and S. Ermon. Understanding the limitations of variational mutual information estima- tors. In International Conference on Learning Representations, 2019.
[51] H. Thanh-Tung and T. Tran. Catastrophic forgetting and mode collapse in gans. In 2020 international joint conference on neural networks (ijenn), pages 1-10. IEEE, 2020.
[52] N. Tomasev, I. Bica, B. McWilliams, L.. Buesing, R. Pascanu, C. Blundell, and J. Mitrovic. Pushing the limits of self-super resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119 2022.
[53] D. C. Van Essen, S. M. Smith, D. M. Barch, T. E. Behrens, E. Yacoub, K. Ugurbil, W.-M. H. Consortium, et al. The WU-Minn human connectome project: An overview. Neurolmage, 00:62-79,20 2013.
[54] P. L.. Williams and R. D. Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.250 2010.
13

---

A Model pseudocode
Algorithm I: Pscudocode for the causally emergent representation learning algorithm.
Input: Dataset 'D ((1,, num_epochs, number of atoms n, Optionallpretrained representation network [8*]
Initialize representation network fo. predictive critic Jo. marginal critics (her) }",
Optional[diversity critic kr]
for epoch + 1 to num num_epochs do
for 114+1 in D do
Us, vt41 +- fo
Ter -- (Vt, 0(+1)
Update 9G to maximise SMILE lower bound (Eq. 4)
for i 4- 10 n do
Let. t (21,
end
Update {/g: 3r to maximise SMILE lower bound (Eq. 4)
= =i Ei= Tsgmess
if training diverse emergent feature then
Ter.son +- lio (Vz, So (2x))
loss x- - + Ine-So
else
loss -V
end
Update fø minimize loss
end
end
14

---

B
Supplementary results
B.1 Supplementary figures
Full emergence objective
Prediction-only objective
1.0
1.0
(estimated)
Emergence
0.5
Predictive MI
estimates
(ground truth)
0.0
0.0
0
10
20
30
0
10
20
30
1.0
1.0
Parity bit
Decoding 0.8 accuracy
0.8
Extra bit
Chance level
0.6
0.6
2
4
8
10
0
2
6
8
10
Training steps (x 10
Training steps (x 10
Figure 5: Replication of main results with noisier data. Same in Fig. 2, but with Jparity
Textra = 0.9.
0.2
0.1
0.0
2
Training steps (x103)
Figure 6: Replication of main results with even noisier data. Emergence values on the bit-string
dataset with Pparity = Textra 0.7.
15

---

time
Figure 7: Glider states in the Game of Life simulation. Gliders cycle deterministically between these four states. To interpret the learned emergent feature in the experiments in Section 3.1, trained a standard classification MLP to predict these four states from the value of V,-
a)
Emergence check
b)
Glider decoding
0.5
Accuracy
1.0
0.4
0.3
0.0
0.1
0.2
0
50
100
Training steps (x 10%)
Training steps
Figure 8: Our method learns interpretable emergent features in Conway's Game of Life. a) Post- hoc checks reveal A > 0, confirming the learned feature is emergent. b) Classification accuracy of the state of the glider (c.f. Supp. Fig. 7) held-out test set. Dashed black line represents chance level at 25%.
a)
ECoG
b)
MEG
c)
fMRI
1.5
4.0
2.0
1.0
V
2.0
0.5
0.0
0.0
0.0
50
100
150
100
200
100
200
Training steps (x103)
Training steps
Training steps
Figure 9: Learning emergent features from brain activity datasets. Post-hoc V checks for our representation learner trained on a) primate ECoG data, b) human MEG data, and e) human fMRI data. Emergent features were successfully leamed in all cases.
16

---

a)
Emergence check
b)
Diversity term
6.0
1.5
1(V;; VIA)
4.0
1.0
2.0
0.5
0.0
0.0
2
10
15
20
Training steps (x (x10*)
Training steps (x10%)
Figure 10: Learning diverse emergent features from ECoG brain activity data. a) After learning one emergent feature on the ECoG data (c.f. Supp. Fig. 9, adding the diversity loss term and training a new representation network results in another emergent feature, verified by post-hoc V A check. b) We further verify that the new feature Vú is nearly independent from the previous one V.A. as their mutual information (estimated with SMILE and the critic k,) vanishes.
B.2 MLP comparison on IMRI Dataset
In order to investigate further whether other representation learning methods could naturally learn emergent features, we also trained an encoder-decoder pair of multilayer perceptrons (MLPs) on the fMRI dataset to predict X₃ from X, using an MSE objective. The encoder mapped X, representation Vú with the same dimension as fo output. A post-hoc test revealed that the encoder did not learn an emergent feature, as A < 0 (after training, model stabilised at V -28 5), indicating that standard autoencoder-like architectures may not capture emergent properties in complex data.
C Adjusted emergence metric
Let us present an informal derivation the correction that leads to the refined criteria for emergnece presented in Eq. 2. This formulation relies the Partial Information Decomposition framework [54], on which our emergence metric is based.
Let us simplify the notation and use B = V/+1 for the target variable and A, Xj for the i-th source of information. Taking inspiration in Ref. [43, Appendix A], the total information about B provided by A = (A1...., A,) can be expressed as
I(A; B) Red A; B) Un(A; B) + (yn(A;
(6)
where (ed(A; B) is the redundancy between sources, Syn( AA B) is their synergy. and Un(A: B) accounts for the unique information provided by each predictor and not the others. Of these, the synergistic component is the one associated to emergence by the theory in Ref. [43]. By using this decomposition and the chain rule of the mutual information [15], one can find that [AA;; B) Red(A; B) + O where θ accounts for other terms unique to source i, and hence
Σ I(A; Red (A + Σ θ
This shows that, when summing the marginal mutual informations, the term Red B) gets double- counted times, resulting in containing this redundancy term n times. Hence, is natural to propose refined criteria for emergence that avoids this double-counting, given by
V = IVv; - [Red (oc((AAD) B) + Σo.) = + (n Red A: B).
(8)
17

---

D Alternative Formalizations of Emergence
Several frameworks have been proposed to formalize the concept of emergence in complex systems. Below, we provide brief overview of some approaches that are particularly well-suited to investigate time-series.
D.1 Emergence as dynamical independence
Barnett [3] introduces formalization of emergence based on information theory, defining emergent macroscopic process Vi = f(x, one where the dynamics of Vú are conditionally independent of the microscopic history X, given its own history V,- Accordingly, the condition for emergence is defined using transfer entropy [46, 7] follows:
T2(X -) XVV
-VV [V 0.
This framework captures the idea that the macroscopic state's 's future is determined by its own history, and additional knowledge of the microstate does not enhance prediction.
D.2 Emergence via computational mechanies
Computational mechanics [17, 18, 48] defines emergence through the concept of causal states, which are equivalence classes of histories leading to the same conditional distribution over futures S. This leads to establish two keys quantities: the statistical complexity C₁ and excess entropy E, which are calculated as
C, = H(S), E= 1(;35)
(10)
Above, the excess entropy quantifies how much information shared between past and future, and the statistical complexity measures how much information processing is needed to enable such prediction. Building on these ideas, emergence has been characterised in different ways. One approach has been to identify a process Sy = f(55.) e) as emergent when increases the efficiency of prediction E/C₁, compared to the microscopic description [48, Sec. 11.2). Another approach has been to assign emergence when there is new type of dynamics observed at the macroscopic level from the one observed at the micro [17, 44]: for example, dynamical patterns associated computational class [17, Sec. 5.1]. Finally, another approach has been to ascribe emergence to macro procoepp whose causal states are a coarse- graieing of the causal states of the micro [41], which leads to view closely related with the one presented in Section D.1.
different
D.3 Granger emergence
Seth [47] proposes Granger emergence, where macroscopic process V2 is emergent with respect to the microscopic process X, if satisfies the following conditions:
T(V;; Ve X.) > 0 and 7(X;; Ve V,, 0.
(11)
The first condition ensures that Vi contains predictive information about its future that is not present in X1, while the second condition establishes that Vú arises from X₂.
D.4 Emergence via effective information
Hoel et al. [24] introduce the concept of effective information, which measures the strength of depen- dencies using maximum entropy distributions [14]. More specifically, for a given probability kernl corresponding to p(X++1 Xx)) the corresponding effective information is Ef(X) = 1(X;; X where X, assumed follow uniform distribution. Then, emergence is assigned when the kernel p(V++ilV) that results when applying the coarse saase-grainiig V = f(X,) satisfies
Ef(V) Ef(X).
(12)
In other words, there is more information going through the macroscopic than the microscopie level when considering maximum-entrope inputs both of them.
18

---

D.5 Comparison with our framework
While these alternative formalisms provide valuable insights into emergence, they differ from the approach we use, which emphasizes the part-whole relationships by considering how the emergent feature encodes information about the microscopic components, particularly focusing on synergistic interactions. Our framework aligns with the view that emergent properties arise from collective interactions among parts that cannot fully understood by examining components isolation. Moreover, it is important to highlight that the metrics associated with most of these approaches are computationally expensive, and don't scale well with system size. Enabling similar methods as the one proposed here but with these other metrics of emergence, and comparing the resulting findings, would be an extremely interesting line for future work.
Hyperparameters
Hyperparameters for the models trained on the synthetic and brain datasets be seen in the below tables. The hyperparameters do not change when learning diverse sets of emergent features. For each training the clip was fixed in the SMILE mutual information estimators and the optimizer used was AdamW.
Table 2: Hyperparameters for causal emergence representation learning on synthetic data
Hyperparameter
Value
Number of bits in X₁
5 +
Number of training steps
5e6
Parity autocorrelation 7/parity
0.99
Extra bit autocorrelation Textra
0.99
Batch size So layer sizes
1000
[256, 256, 1]
Microscopic variable critic layer sizes
[512, 512, 512, 256]
Macroscopic variable critic layer sizes
[512, 512, 512, 256]
Number of steps between fe updates fo weight decay
5
le-3
fe learning rate
le-4
Critic learning rate
le-3
Table 3: Hyperparameters for causal emergence representation learning ECoG data
Hyperparameter
Value
Number of variables in Xx
64
Number of training epochs
60
Batch size
1000
Te layer sizes
[256, 256, 256, 256, 256, 3]
Number of batches to pretrain critics
300
Microscopic variable critic layer sizes
[512. 512, 512, 256, 32]
Macroscopic variable critic layer sizes
[512, 512, 512, 32]
Number of steps between fo updates fo weight decay
le-3
fo learning rate
le-4
Critic learning rate
le-3
19

---

Table 4: Hyperparameters used to train emergent feature network on MEG and fMRI dataset with V
Hyperparameter
Value
Batch size
1000
Macroscopic variable critic layer sizes
[512, 512, 512]
Microscopic variable critic layer sizes
[512, 1,024, 1,024, 512, 32] 1e-3
Critic learning rate
Number of training epochs
fo layer sizes
[256, 256, 256, 256, 256, 32, 3] le-4
So learning rate
So weight decay
1e-6
Number of steps to pretrain critics
150
Number of steps between fe updates
Table 5: Hyperparameters for Game of Life emergent feature learning with V criterion
Hyperparameter
Value
Grid size
15
Batch size
1000
Number training epochs
10
Feature size
1
Number steps to pretrain critics
150
Number of steps between fo updates
5
Convnet encoder layer sizes Downward critic learning rate Macroscopic variable critic layer sizes Microscopic variable critic layer sizes Critic learning rate
Conv[32, 64, 128, 256], FC[256, 128, 3]
1e-3
[512, 512, 512, 32]
[512, 512, 512, 32] le-3
20

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
No new theory, all theory is cited.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Co///oooeeeeeeeeeee
/nips wwtppcccc//caeeccccccccccccccace / swssssosiicgy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: No statistical claims are made, simply demonstrate that certain conditions can be met using estimations of MI.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics (tctpa///eeurpge wttpph/wwww..ppppppptetpppctttce cttpe//eccpocccttticccccdedeeeed
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not further the capabilities of AI, and instead aims to develop methods that further understanding complex systems. Our models have no path to harm.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The data and models have no such risks, they are small datasets of the ECoG and the models are small MIPs
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing was used, and no new data from human subjects subjects collected.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Same above.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "m16sJVMc1u", "reviewer": "Reviewer_pckS", "summary": "The paper introduces a novel objective function and deep learning architecture that are targeted to extract emergent latent representations from time series data. Motivation is very clear. The definition of emergent latent representation interesting and useful. The utilization of mutual information estimators (lower bounds thereof) is smart. Evaluations are restricted to a fully artificial and a fully neurobiological dataset.", "questions": "Emergence comes in many forms – I wonder if you could discuss alternative definitions / alternative perspectives on the matter?\n\nLine 52 – a “not” too much. \n\nEq (2): can you motivated the adjustment term further the summand (n-1)min_i… ? Are there alternatives to this that would be more targeted towards actually identifying true redundant mutual information? \n\nEq (4). Could you motivate the clip operator slightly more? \n\nLine 102 should read “accurately”\n\nParagraph 113ff: maximize / minimize information – I am not sure if this is worded the right-way round – could you double check and slightly reword? \n\nLine 137 – should not read “also”\n\nLine 142 – one “is” too much\n\nLine 190ff. removing the macroscopic MI term seems not to save much – or does it? The observation is interesting, but I wonder if the authors want to make a computational argument here as well. \n\nAbout the biological dataset – this is very ad-hoc somewhat. What does this analysis tell us really except that there are some complex spatio-temporal statistical correlations in the data? I find this one marginally useful.", "limitations": "Unclear how robust this is as ablations and comparisons are not very extensive.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The study of emergence and its conceptual and mathematical formalization is of general interest to neural information processing and the involved (part-of) cognitive science subdiscipline within. The authors utilize an existing definition thereof [30] as well as an approximation technique of a lower bound on mutual information (SMILE, [32]), which they combine in a highly elegant manner to yield their learning architecture. \n\nThe usage of a linear temporal predictor with learnable residuals is a great way to bootstrap the system’s abilities. \n\nEven multiple emergent latents can be successfully extracted. \n\nA real-world dataset indicates applicability beyond artificial data. \n\nPaper is very well written – relatively easy to comprehend and all steps taken are very well motivated and sufficient background is provided.", "weaknesses": "System evaluations are minimalistic and not as convincing as I had hoped. Both, comparisons to potential baseline algorithms as well as more ablations are missing. \n\nFurthermore, one artificial dataset and one not well-motivated real-world neural dataset seems not enough to warrant publication. \n\nIn particular, I would have expected at least one if not multiple DL/Neural Network baselines that do not pursue the information-theoretic objective but simply a standard temporal prediction objective. Those probably do not work on the parity problem at all but at least an attempt seems needed. That is, use a DREAMER-like world model learning architecture with probabilistic latents and see if structure emerges. \n\nAblations could have explored more than just the same architecture without the penalty / adjustment term or without the macroscopic estimator. Further, in the artificial dataset the correlation coefficients $\\gamma$ are quite high – was this necessary? When does this break down? Evaluations with a non-linear prediction pipeline would also be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "As mentioned in the global rebuttal, we have now conducted extensive additional evaluations, including:\n\n* Two new real-world datasets and one new synthetic dataset;\n* New analyses on the existing synthetic dataset with lower correlation coefficients; and\n* Comparisons against baseline algorithms based only on standard temporal prediction.\n\nOverall, our method was able to learn emergent features in all these new scenarios, the resulting features were interpretable for the synthetic cases where a notion of “ground truth” emergence is known, and baseline algorithms (standard RNN and MLP) do not learn any emergent features.\n\nIn response to your questions, we have fixed all the typos and rephrased the relevant sentences in the paper. Here are\nresponses to the rest of the questions:\n\n> Emergence comes in many forms – I wonder if you could discuss alternative definitions / alternative perspectives on the matter?\n\nWe totally agree that emergence comes in many forms, and we acknowledge that our paper could do a better job at placing our definition within the broader literature on emergence. For the camera-ready version we will include a discussion regarding alternative definitions of emergence, and how this could be explored in future work.\n\n> Eq (2): can you motivated the adjustment term further the summand $(n − 1) \\min_i ...$ ? Are there alternatives to this that would be more targeted towards actually identifying true redundant mutual information?\n\nThe reviewer is right in that the discussion surrounding the adjustment term could be expanded. For the camera-ready version we will include a proof of where this adjustment term comes from, and a discussion including what redundant information is and what role it plays in the formula. As a matter of fact, there are multiple measures of redundant mutual information in the literature, and a comparison in this context would be an interesting avenue for future work. However, we would like to emphasise that the current form of the adjustment is a recognised and well-defined measure of redundant information (in the sense that it satisfies many of the natural properties one would require of such a measure), as shown in the following paper:\n\nBarrett, Adam (2015). Exploration of synergistic and redundant information sharing in static and dynamical Gaussian\nsystems. *Phys. Rev. E* 91, 052802. doi: [10.1103/PhysRevE.91.052802](https://www.doi.org/10.1103/PhysRevE.91.052802)\n\n> Line 190ff. removing the macroscopic MI term seems not to save much – or does it? The observation is interesting, but I wonder if the authors want to make a computational argument here as well.\n\nWe agree, thanks for this observation. To clarify, there are in fact two arguments for doing this: one is the computational argument to save some compute; and the other is that in some cases we observed it stabilised learning dynamics (our hypothesis is that it alleviates the adversarial relationship between the different components of our objective). For the camera-ready version we will clarify these issues and include both arguments.\n\n> About the biological dataset – this is very ad-hoc somewhat. What does this analysis tell us really except that there are some complex spatio-temporal statistical correlations in the data? I find this one marginally useful.\n\nOur motivation to study emergence (as described in the introduction) is to understand how cognitive systems encode information in macroscopic variables, which naturally motivates us to test our methods on data from biological brains. The fact that the same method that successfully identifies particles in the Game of Life detects emergent dynamics in brain data is extremely encouraging, suggesting that neural systems may encode information into “glider-like” collective features. This paper provides direct empirical evidence supporting this idea. Moreover, it is particularly encouraging that the emergent character is observed in multiple neuroimaging modalities that cover multiple spatial and temporal scales of the brain's dynamics.\n\nNeedless to say, there is much future work to be done examining precisely what the role of emergent dynamics in the brain is, and how these emergent features contribute to brain function. We see this as a longer-term research programme, which our method is enabling for the first time. Since we do not have a sense of “ground truth” of what the emergent features of brain activity should be, this is difficult to do as part of this paper (although note that for cases where such ground truth is known, such as the bitstrings and the Game of Life, the results align with our intuitions). For the camera-ready version, we will emphasise the significance of these findings and the need for future work interpreting the learned emergent features in real-world data.\n\nFinally, in response to the limitation:\n\n> Unclear how robust this is as ablations and comparisons are not very extensive.\n\nAs mentioned in the global rebuttal, we have now performed:\n1. Further ablation analyses, showing the expected behaviour that key properties of the architecture (e.g. ability to learn diverse features) are lost if the corresponding elements of the objective are removed; and\n2. Further comparisons with other, non-information-based architectures (e.g. RNNs, MLPs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aC6COYJ4FS", "reviewer": "Reviewer_aLEM", "summary": "The paper presents a method for identifying emergent variables in time series data through a novel machine learning architecture. It uses unsupervised learning for representation and information theory to find emergent properties in systems, which are often complex and not easily describable at the microscale level. The paper is motivated by the fact that unsupervised learning can be a powerful tool for identifying emergent properties, but current approaches limit to only information theory.\n\nThe method rests on maximizing an objective defined by subtracting the mutual information of state variables at time t and the coarse graining at time t + 1 from the mutual information of the coarse grainings at t and t + 1. In other words, the amount of emergent information. Information theoretic definition of emergence is thus used to facilitate unsupervised learning. The method is tested on synthetic data and real-world ECoG brain activity data, demonstrating its ability to identify known and novel emergent features effectively.\n\nExperiments are conducted on a synthetic dataset and a macaque brain activity dataset. For the synthetic dataset, the method is able to estimate the ground-truth value of psi (the difference that is central to the objective function). For ECoG data, skip connections were introduced into the architecture, and once again found emergent representations. \n\nThe paper concludes with a discussion on related (info theory) work, limitations, and future steps.", "questions": "None", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "### Clarity \n- Diagram features are well designed and results features are clear and salient\n- Though writing is somewhat unstructured, the shorter-range explanations are well-done\n- Methodology is given in detail. Lots of helpful explanation of relevant information theory, as well as the overall approach\n\n### Quality\n- Good to have primate brain data, though more interpretation would help \n- Covers all the basic needs for a new method: real data, novel setup, suitable metrics (though they need more explanation)\n- Experimental setup is well-designed to demonstrate that emergent variables are being learned \n\n### Originality\n- As far as I know, applying the information theoretic definition of emergent variables as an objective and training in this setting is novel\n\n### Significance\n- An innovative idea that shows promise. While there could be more experimentation, this is a promising and new direction.", "weaknesses": "### Clarity\n- It's not immediately clear how to interpret results. The paper shows figures, but it doesn't explain them much. Interpreting them requires a lot of re-reading the methods section\n- Writing is somewhat verbose and unstructured, and occasionally reads like a process statement\n\n### Quality \n- This idea is compelling and innovative! The loss built on MI of coarse grainings and state variables is intuitive while creating a solid foundation for taking advantage of the capabilities of unsupervised learning\n- On the ECoG dataset, giving intuition/semantic understanding of emergent features (or at least attempting interpretation) would be cool\n- Limited experiments on real data in general - ultimately, only one experimental setting is shown as far as I understand. The synthetic problem, while useful, is simple\n- Lacking baselines or extensive comparison to existing methods, even if purely information-theoretic\n\n### Significance\n- It would help to have clearer comparison to existing methods so that we could see the value-add of this innovation, not just the novelty and value alone", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In response to the specific weaknesses identified:\n\n> It’s not immediately clear how to interpret results. The paper shows figures, but it doesn’t explain them much. Interpreting them requires a lot of re-reading the methods section\n\nWe apologise for the lack of clarity in our explanations. For the camera-ready version we will make sure the figures are explained in more detail and the text flows more smoothly.\n\n> On the ECoG dataset, giving intuition/semantic understanding of emergent features (or at least attempting interpretation) would be cool\n\nWe enthusiastically agree with the reviewer that obtaining a semantic understanding of the learned emergent features in ECoG would be definitely cool. However, we see this as a longer-term research programme that our method is enabling. Since we do not have a sense of “ground truth” in terms of what the emergent features of brain activity are, this is difficult to do as part of this paper.\n\nIn cases where we do have a reasonable sense of what these features might be (e.g. in the synthetic dataset and in Conway’s Game of Life), we conducted post-hoc analyses to interpret these features, and found that they align with our intuitions. For the camera-ready version we will emphasise the need for future work interpreting the learned emergent features in real-world data.\n\n> Limited experiments on real data in general - ultimately, only one experimental setting is shown as far as I understand. The synthetic problem, while useful, is simple\n\nAs mentioned in the global rebuttal, we have now added experiments on two more real-world datasets, and demonstrate our method can successfully learn emergent features on both. We also add one more synthetic,but far less simple, dataset using Conway’s Game of Life system, and use our method to learn emergent features from it and interpret them with post-hoc analyses.\n\n> Lacking baselines or extensive comparison to existing methods, even if purely information-theoretic\n\nIn addition to new ablation studies, showing comparisons against other information-theoretic loss functions, we now compare our method against a standard RNN and a standard MLP architecture (without any information theory). The results suggest that RNNs and MLPs do not learn emergent features on their own, supporting the value of our method.\n\n> It would help to have clearer comparison to existing methods so that we could see the value-add of this innovation, not just the novelty and value alone\n\nWe agree that comparisons against existing methods can make a clearer argument in favour of our paper. As mentioned above, we now show that standard RNN or MLP do not spontaneously learn emergent features. Additionally, we show in post-hoc analyses that the combination of the learned emergent feature of our method and an RNN representation enables better predictions than either of them in isolation (informally: a small RNN plus an emergent feature is better than a large RNN), making a direct case for the value-add of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PIkakmKRkD", "reviewer": "Reviewer_PMxp", "summary": "This paper introduces a method for learning the causally emergent representation of time series data. Based on the Partial Information Decomposition (PID) and ΦID definition of emergent variables, the paper utilizes variational information lower bounds to estimate and optimize the emergence objective function. The paper further includes a Minimum Mutual Information term and a penalty term, to reduce redundancy and discover diverse emergent variables, respectively. Experiments on a synthetic dataset and a primate brain activity dataset show that the method is able to discover diverse causally emergent representation.", "questions": "N/A", "limitations": "The authors did not explicitly state the limitation of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Discovering causally emergent representations is a very interesting topic, and has significance in a wide range of scientific disciplines. The paper is inspirational and written clearly. Although the components of the method, i.e. definition of emergence objective function, and variational bounds for mutual information, are not new, their combination to discover causally emergent representations in a learnable way is interesting, and to my knowledge, novel.", "weaknesses": "As discussed above, the novelty is a little limited. This can be compensated by solid evaluations with a wide range of interesting datasets. I think the place the paper needs most improvements is more diverse and extensive evaluations. The paper can benefit from a few more datasets, both synthetic and real world, including the other datasets used in [1] and other references. If there exists baselines for discovering causally emergent representations, those baselines should also be compared against.\n\n\n\nReference:\n\n1. Rosas, Fernando E., et al. \"Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data.\" PLoS computational biology 16.12 (2020): e1008289.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "As mentioned in the overall rebuttal, we have now conducted a wider range of evaluations, both using the same architecture on more datasets and using other architectures on the same datasets.\n\n* Specifically, we add two more brain activity datasets which capture different aspects of neural dynamics: fMRI, with high spatial resolution and low temporal resolution; and MEG, with low spatial resolution but high temporal resolution. Our method is able to find emergent features in both cases.\n\n* We ran experiments on one more synthetic dataset used in Rosas et al. 2020, Conway's Game of Life. Here we found that our method can learn emergent features, and that these can be interpreted as encoding the state of the particles, as one would expect.\n\n* We compare against standard RNN and MLP architectures that predict the future state of the time series, without any information-theoretic loss function. We confirmed that neither the RNN or the MLP are able to learn emergent features.\n\nTo the best of our knowledge, there are no established baselines for discovering causally emergent representations – if anything, the Game of Life is one of the most long-standing canonical examples of emergent behaviour generated by relatively simple rules, and we now show our method works on it. We would of course be happy to consider any other suggestions.\n\nFinally, in response to the limitation:\n\n> The authors did not explicitly state the limitation of the paper.\n\nWe apologise for the lack of clarity on this regard. We state some of the limitations of our work in Section 4, although we acknowledge this needs to be more clearly laid out. For the camera-ready version we will add an explicit discussion of our method's limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ILd0j0N52p", "reviewer": "Reviewer_7mVS", "summary": "The article proposes a learning scheme aimed at detecting emergent quantities from time series data of systems made of many interacting parts, such as the brain. To this end the authors combine \"minimum mutual information\", a previously introduced emergence criterion, with SMILE, a differentiable lower bound estimator for mutual information. Differentiability is crucial for the loss function to be optimizable efficiently. They apply this architecture to two examples: First, a series of random bit strings with time-correlated parity, where parity is considered the emergent quantity. Second, real-world data of macaque brain activity. The approach successfully identifies parity in the first example. The authors claim that an emergent feature has been learned for the second example also.", "questions": "1) How does Figure 4 prove that an emergent feature was found? What is the quantity on the x-axis in this figure?\n2) Can you share the code of the architecture and its training or describe it in more detail?\n3) Would you expect this method to find emergent features in standard, well-understood systems, such as Ising models?\n4) Can you explicitly write down the prediction-only objective in Fig.2?\n\nOverall, I find the approach promising but not yet tested on enough examples to support its ability to find emergent features.", "limitations": "There is no open access to the code.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "While the individual parts of the learning scheme are not new, their combination into a differentiable architecture is original and seems like a promising direction to me. The analysis seems sound, even though I found the presentation at times a bit hard to follow as some parts seem to be missing. The individual quantities are mostly clearly defined and the individual results are statistically significant in terms of error bars.", "weaknesses": "From the article alone, I could not fully understand the architecture and its training procedure that is illustrated in Fig. 1. I could not find code that would reproduce it, or a detailed pseudo-code description of the algorithm. It is unclear to me what emergent feature was found for the monkey example, or how Fig. 4 proves that any such feature was found. While the architecture and direction seems promising to me, a few more benchmarks would help make the case that this scheme can find emergent features in many settings. The two examples shown are one toy example with unnatural time dynamics and one real world example where it is hard to understand the dynamics from first principles. Benchmarking this new method on more standard examples with emergent behavior, such as Ising models, would be more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the feedback. We apologise for the lack of clarity in some of our explanations. We will improve these and add pseudo-code in the camera-ready version of the paper.\n\nRegarding the concern about not having enough examples, as we describe in more detail below and in the global rebuttal, we have conducted several additional benchmarks and evaluations. These new analyses include:\n\n1. Two new real-world datasets, showing our method works well in both cases.\n2. One more synthetic dataset, Conway’s Game of Life, which is arguably one of the most well-known examples of emergent behaviour. Since in this system there is a notion of what the emergent feature should be (known in the literature as “particles”), we can show that the learned feature captures the state of these particles.\n3. Comparisons against baseline algorithms, making the case that our method finds features that standard algorithms based only on temporal predictions do not.\n\nIn response to your questions:\n\n> How does Figure 4 prove that an emergent feature was found? What is the quantity on the x-axis in this figure?\n\nWe apologise for the oversight in the labelling of this figure. The X-axis represents training time, and the Y-axes represent the emergence measure $\\Psi$ and its adjusted variant $\\Psi_A$. The plot proves that an emergent feature was found because the emergence measure is greater than zero, which is a sufficient condition for the feature being emergent (as shown by Rosas et al., 2020; Ref. [30] in the paper).\n\n> Can you share the code of the architecture and its training or describe it in more detail?\n\nDue to the constraints of the double-blind review we have been unable to share a Github link, but we will include it in the camera-ready version of the paper. Also, we apologise for the lack of clarity in our description – we will improve the description of the method and add a pseudo-code algorithm to the paper in the camera ready version.\n\n> Would you expect this method to find emergent features in standard, well-understood systems, such as Ising models?\n\nAlthough the behaviour of the Ising model is often described as emergent, our method is not readily applicable to the Ising model. This is because our method is explicitly designed for time series data (as the title indicates), but the Ising model doesn’t have time per se – the Ising model specifies a probability distribution over an ensemble of spin configurations, without any dynamics. There are of course several options to add dynamics to the spins (e.g. Metropolis-Hastings, Glauber, Kawasaki, etc.), but none of these are an integral part of the canonical Ising model. For this reason, if one were to choose one of these dynamics and apply our method, then the kind of conclusions one may draw is whether or not e.g. Glauber dynamics display emergent features, and not about whether the Ising model does.\n\nFor this reason, we have chosen instead a different well-understood system that has dynamics and is widely considered to display emergence: Conway's Game of Life. As mentioned above, we show that our method can successfully find emergent features from Game of Life time series and that these features capture the state of the particles in the system.\n\n> Can you explicitly write down the prediction-only objective in Fig.2?\n\nThe prediction-only objective is $I^\\text{S}_\\varphi(f\\_{\\theta}(X_t); f\\_{\\theta}(X\\_{t+1}))$, i.e. the mutual information between the past and future of the learned representation. This quantity depends on the parameters of the critic $\\varphi$ and the parameters of the representation learner $\\theta$, and is calculated using the SMILE estimator as defined in Eq. (4).\n\nFinally, in response to the limitation:\n\n> There is no open access to the code.\n\nAs mentioned above, we didn’t include a link due to the constraints of the double-blind review. We will add a link to a publicly accessible Github repository with all the code once the requirements of double-blind review are lifted. In the meantime, we have sent the AC an anonymised link to the code for the model architecture.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their time and thoughtful comments on our paper. We are encouraged by the positive feedback, and are thankful for the constructive suggestions that have let us identify and address several limitations of our paper.

We have added responses to each reviewer’s specific comments, and provided answers to all of the reviewers’ questions (where applicable). More broadly, we have identified a number of overarching themes that multiple reviewers brought up and would like to address here. Below is a non-exhaustive list of work we have done motivated by the reviewers’ suggestions, together with a description of changes we are ready to make for the camera-ready version of the paper:

* **Evaluations in more real-world datasets**. We have conducted experiments on two more real-world datasets involving two different brain scanning modalities: functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). Results reveal that our method was able to successfully learn emergent features in both datasets (rebuttal Fig. 1). This shows that our method is flexible and effective; and it provides a stronger motivation for the real-world data analyses as showing emergent behaviour is relevant across multiple scales of brain
activity – microscale with ECoG, mesoscale with MEG, and macroscale with fMRI.

* **Evaluation in a new synthetic dataset and additional ablation analyses**. We have also used our method to successfully find emergent features in Conway’s Game of Life (a prominent and long-standing example of emergent behaviour), which complements our results via a second synthetic dataset from a well-understood system (rebuttal Fig. 2). In addition, we have also conducted further ablation studies, which clarify the specific role of each component of our architecture. Finally, we also replicated our results for additional $\gamma$ values in our synthetic dataset, which provide further evidence confirming our previous findings (rebuttal Fig. 3).

* **Better interpretation of learned features**. Although reverse-engineering the function of emergent features of the brain is a larger, long-term research problem, we are able to provide clear interpretations of learned emergent features in systems where some notion of “ground truth” is available. Specifically:
  1. For the “random bits” synthetic dataset, we show that the learned feature can be interpreted as capturing the parity of the bitstring (paper Fig. 2).
  2. For the new results on the Game of Life, we show that the learned feature captures the state of the so-called particles (i.e. gliders) in the system (rebuttal Fig. 2).

* **Comparison against standard architectures**. We have compared the performance of our method against a standard RNN and a standard MLP trained only on a temporal prediction objective, both on real-world (rebuttal Fig. 1) and synthetic data (rebuttal Fig. 4). The results show that neither the RNN nor the MLP by themselves learn the emergent features of the data, and thus our method can provide a unique value for predicting datasets with emergent dynamics. As quantitative evidence for this claim, we show that the combination of a learned emergent feature and the RNN representation can predict the future state of the synthetic dataset better than either of them in isolation, highlighting the value-add of our method in conjunction with other algorithms.

* **Improved presentation**. In the camera-ready version we will improve the description of our architecture and our figures, provide a pseudo-code algorithm, and discuss our framework in the broader context of alternative definitions of emergence.

* **Code availability**. We will add a link to a publicly available Github repository once the requirements of the double-blind review are lifted. In the meantime, we have sent the AC an anonymised link to the code for the model architecture.

We hope this new work addresses some of the reviewers’ concerns, particularly regarding the addition of a broader range of evaluations on new real-world and synthetic datasets. Please do not hesitate to ask more questions if any aspect of our work remains unclear. And, once again, we would like to thank all the reviewers for their extremely valuable feedback.

Meta-review: This work attempts to combine insights from the literature on emergence with differentiable estimators of mutual information to look for emergent features in time series data. The authors explore the presence of emergent features in some synthetic data sets and in data from neuroscience experiments. 

Reviewers appreciated the problem formulation and the combination of ideas from information decomposition with differentiable estimators. Reviewers also found the problem statement clear and of interest to the NeurIPS community. Additional experiments performed by the authors in response to reviews were also judged to substantially strengthen the paper. One weakness noted by all reviewers was the difficulty of interpreting what emergent features were learned in the real data, as the finding is based on the estimated value of $\Psi$, without any particular insight as to what that implies.

On balance, all reviewers considered this a valuable contribution, and I concur. Accept.
Decision: Accept (poster)