Learning diverse causally emergent representations from time series data

Authors: david  mcsharry
christos  kaplanis
google  deepmind
fernando e rosas
pedro a m mediano
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:19:03

Abstract: Cognitive processes usually take place at a macroscopic scale in systems characterised by emergent properties, which make the whole ‘more than the sum of its parts.’ While recent proposals have provided quantitative, information-theoretic metrics to detect emergence in time series data, it is often highly non-trivial to identify the relevant macroscopic variables a priori. In this paper we leverage recent advances in representation learning and differentiable information estimators to put forward a data-driven method to find emergent variables. The proposed method successfully detects emergent variables and recovers the ground-truth emergence values in a synthetic dataset. Furthermore, we show the method can be extended to learn multiple independent features, extracting a diverse set of emergent quantities. We finally show that a modified method scales to real experimental data from primate brain activity, paving the ground for future analyses uncovering the emergent structure of cognitive representations in biological and artificial intelligence systems.

Main paper: 
 introduction: 
 Cognitive processes usually take place in systems made of multiple interacting parts, e.g. neurons composing the nervous system of an organism. Importantly, cognitive processes themselves don't seem to take place at a 'microscopic' level of individual units, but at 'macroscopic' levels involving assemblies of several coordinated units [28] . Hence, when trying to unveil the inner workings of a -natural or artificial -cognitive system, it is crucial to be able to identify relevant macroscopic variables that best characterise the corresponding cognitive processes.The identification of macroscopic variables has traditionally been driven by intuition and expert knowledge. For example, the investigation of collective behaviour in statistical physics is based on macroscopic variables known as 'order parameters,' which are typically identified heuristically and then used to describe phase transitions and other phenomena of interest [49] . Unfortunately, identifying relevant macroscopic variables is often more an art than a science, being heavily dependent on prior knowledge and expectations. Having automated procedures to identify relevant macroscopic variables of cognitive systems would open important avenues for investigating the inner workings of different cognitive architectures.A promising approach to identify empirically useful macroscopic variables is provided by unsupervised representation learning [38, 22, 52] . For example, information maximisation has proven to be a powerful objective for learning representations within neural networks [23, 38] . In this paper we combine this approach with recent breakthroughs in our ability to formally characterise emergent phenomena [43, 34] , which have proven to be not only theoretically sound but also empirically powerful [30, 40] .Building on this literature, in this paper we leverage recently proposed metrics of emergence to identify representations that display emergent properties. Specifically, we propose an end-to-end differentiable architecture that can learn maximally emergent representations of multivariate time series data. Our results show that causal emergence speeds up learning of more complex features of the data relative to pure mutual information maximisation, and we demonstrate the scalability of our method through an analysis of real-world brain activity data. 
quantifying emergence: 
 Consider a scientist measuring a system of interest composed of n parts, and let X i t denote the state of part i at time t. The information that the joint process carries together from t to t ′ can be quantified by the standard Shannon mutual information I(X t ; X t ′ ), where X t = (X 1 t , ..., X n t ) is the joint state of the system at time t.How can one characterise an emergent macroscopic variable of such system? Following Ref. [43] , one can define emergent variables V t as satisfying two key criteria: (i) Supervenience: there exists a function (or coarse-graining) f such that V t = f (X t ).(ii) Unique information: V t holds unique predictive information about the future evolution of the system X t ′ that cannot be found in the individual parts X 1 t , . . . , X n t by themselves. Critically, the unique information that V t holds about X t ′ can be rigorously quantified using the framework of Partial Information Decomposition (PID, [54] ), and its recent extension to time series data (ΦID, [33] ). Emergence, therefore, is defined as the capability of a supervenient variable to provide predictive power that cannot be reduced to underlying microscale phenomena.Quantifying unique information in high-dimensional systems can be highly non-trivial. Luckily, the ΦID formalism allows us to derive simpler measures that provide sufficient criteria for emergence. In particular, it has been shown that the following is a sufficient condition for causal emergence [43] :EQUATIONImportantly, Ψ is comparatively easy to calculate, as it relies only on pairwise marginal distributions and on standard Shannon mutual information. These key features allow the framework to be applicable on a wide range of scenarios, as illustrated by the applications reviewed in Ref. [34] . Note that here we take t ′ = t + 1, but in principle any t ′ > t is valid.The reason why Ψ > 0 is only a sufficient, but not necessary, condition for emergence is that in some systems multiple X i t can have the same information about V t+1 , and hence the sum of terms I(X i t ; V t+1 ) may 'double-count' information -resulting in a negative bias in Ψ. This doublecounting can be alleviated by discounting from Eq. ( 1) the redundant information between the X i t 's about V t+1 . Here we do this using a measure of redundancy known as 'Minimum Mutual Information' [4] , which yields the following 'adjusted' emergence criterion (Supp. Sec. C):EQUATIONMaximising emergence. Our aim is to establish an automated procedure to identify emergent macroscopic variables V t with respect to a microscopic substrate X t . For this, we investigate parametric coarse-grainings V t = f θ (X t ) that can be optimised to maximise Ψ via a differentiable objective function. A representation network f θ applied to the data X t learns a feature V t . This feature is trained to optimise Ψ made up of predictive and marginal mutual information terms estimated by g φ and h ξ i respectively (left). A further critic k σ may be added to calculate the mutual information with another emergent feature V A t , to encourage the learning of a diverse set of emergent features (right).A key ingredient to maximising Ψ is employing a suitable estimator of Shannon's mutual information.Although many estimators of mutual information exist [27, 29] , most are not differentiable, and thus not suitable for optimisation with standard representation learning architectures. Fortunately, a number of differentiable estimators of mutual information exist [39, 50] .We use the Smoothed Mutual Information "Lower-bound" Estimator (SMILE, [50] ), which is one of a family of approaches that formulates mutual information estimation as a variational problem, and was specifically designed to address the issue of high variance in existing estimators such as NWJ [37] and MINE [5] . The SMILE mutual information estimator is given byEQUATION≥ E p(x,y) [g φ (x, y)]log E p(x)p(y) clip e gφ(x,y) , e -τ , e τ ≜ I S φ (X; Y ) , (4)where g φ is a parameterised function that estimates the log density ratio log (p(x, y)/(p(x)p(y))), clip(v, l, u) = max (min (v, u) , l) and τ ≥ 0 is a hyperparameter. As τ → ∞, I S converges to the MINE estimation [5] , but a finite τ prevents the potentially exponential growth of the variance of the estimate with MI, which MINE suffers from [50] .Equipped with this estimator, we can now formulate our representation learning algorithm for causally emergent features. The architecture is schematically shown in Fig. 1 . Our main method involves three learnable functions:1. A representation network f θ , that learns a supervenient variable V t = f θ (X t ).2. A critic for the macroscopic variable g φ , that controls the estimation of I(V t ; V t+1 ).A set of critics for the microscopic variable h ξ i , that control the estimation of I(X i t ; V t+1 ). The number of critics equals the number of constituent atoms of the system. 1We will demonstrate that the representation critic can learn emergent features on an algorithmic dataset by maximising the following metric:EQUATIONWe refer to Ψ S as the emergence objective function, and the first term in the RHS of Eq. ( 5) as the predictive mutual information [6] -since (by the data processing inequality [15] ) it represents a lower bound on the joint mutual information between the past and future states of the whole system, I(X t ; X t ′ ). As a control condition, we also ran experiments with an objective function consisting of only the predictive information, removing the marginal mutual information terms. After training has converged, we ran 'post-hoc' tests to further verify that the learned feature is emergent by freezing the representation network f θ , retraining the critics to accurately estimate I(V t ; V t+1 ) and I(X i t ; V t+1 ), and calculating Ψ A . If this resulting Ψ A is positive, we conclude the feature is emergent. For convenience, pseudocode for the algorithm used to train the model is provided in Supp. Sec. A.Learning diverse emergent features. A well-known fact about complex systems is that they can have more than one emergent property, and/or be described by multiple order parameters [25] .To learn a diverse set of emergent features, we consider a scenario where an emergent feature V A t has been learned and its corresponding parameters θ A are fixed. Intuitively, our goal is to find a new feature that is substantially different from the existing one by incentivising it to be statistically independent from V A t . To implement this, we add a penalty termI S σ (f θ A (X t ); f θ (X t ))to the objective function and include a fourth learnable function to our method: a critic k σ that estimates this mutual information, also using SMILE.Improving training stability. An important point to note of the proposed method is that there is an adversarial relationship between the microscopic critics h ξ i and the macroscopic critic g φ through the representation network f θ , reminiscent of the adversarial learning dynamics in GANs [16] . In essence, the former pushes the representation network f θ to maximize information about the system's state, while the latter pushes it to minimize it. This could lead to potentially unstable learning dynamics and various failure modes -for example, a representation network could "trick" the microscopic critics into reducing their MI estimate by exploiting their mistakes, rather than genuinely reducing I(X i t ; V t+1 ), resulting in an artificially inflated value of Ψ. Fortunately, unlike in GANs, it is not harmful for the critics to be overpowered compared to the representation learner2 because the SMILE estimation is bounded from above by the true MI [50] . Following this reasoning, we i) update the critics multiple times for each representation learner update, using a higher learning rate and more parameters; and ii) pre-train the critics before we start training the representation learner, so that they provide a more robust MI estimation and better training signal. This results in slower, but more stable, learning dynamics for f θ .Another method to increase training stability is to use a dedicated choice of the architecture of the representation network. f θ first linearly projects the input to a higher-dimensional space, then passes it through an MLP with residual ('skip') connections, and then projects it linearly down to the output vector V t . Because of the skip connections (and because of the auto-correlation of X t ), at initialisation the output already contains useful information about the system and yields a high value of the macroscopic information I S φ (f θ (X t ); f θ (X t+1 )) before θ has been trained. Therefore, this allows us to reduce the contribution of this term in the objective function, and thus reduce the influence of g φ on f θ . 3Bit-string dataset. We evaluate our method for learning causally emergent representations by applying it to sequences of random bit-strings of length n with two constructed temporal correlations:1. The parity of the first n -1 bits is auto-correlated across time, such thatP ⊕ n-1 i=1 X i t+1 = ⊕ n-1 i=1 X i t = γ parity > 1 2 ,where ⊕ represents modulo-2 addition.2. The last (or extra) bit in the bit-string X n is auto-correlated across time, such thatP X n t+1 = X n t = γ extra > 1 2 .Since parity is a synergistic function of the bits of a bit-string (i.e. it cannot be predicted from any of the input bits individually [42] ), and since the parity predicts some information about the future evolution of the system, V t = ⊕ n-1 i=1 X t is an emergent feature of the system. Despite its simplicity, this dataset has two important advantages: there is a known emergent feature (the parity), and one can calculate the mutual information and the emergence measure analytically. 4These properties will allow us to verify that the model has successfully extracted the expected emergent properties and that mutual information is being accurately estimated.Conway's Game of Life. To further evaluate our method on a more complex system, we apply it to simulations from Conway's Game of Life (GOL) -a canonical setting for the study of emergence [1] . GOL is a cellular automaton on an N × N grid where each cell's state evolves based on simple, deterministic rules. We initialize a "glider" pattern [19] at a random position within the grid, and this glider moves diagonally by cycling through four distinct states until it reaches a boundary and the simulation concludes. The dataset consists of multiple such simulations concatenated to form a continuous sequence. To effectively capture the spatial dependencies inherent in the Game of Life, we replace the multilayer perceptron used in previous experiments with a convolutional neural network for the representation learner f θ , allowing us to exploit the 2D structure of the grid-based simulations.Primate ECoG dataset. We evaluate our method on a dataset of electrocorticography (ECoG) brain activity data from a macaque monkey, originally reported by Chao et al. [10] . The dataset contains long time series of electrical activity measured with 64 electrodes placed across the monkey's brain surface. We minimally pre-process the data by applying a second-order Butterworth high-pass filter with a 1 Hz cutoff, downsampling the signals to 300 Hz, and finally standardising the data before applying our method to learn emergent features from it.Human MEG dataset. We evaluate our method on a magnetoencephalography (MEG) brain activity dataset from healthy human participants, originally reported in four pharmaco-MEG studies [36] . The dataset includes resting-state recordings collected before the administration of various pharmacological agents. MEG signals were sampled at 600 Hz with a 0-300 Hz bandpass filter. For our analysis, we focus on the placebo condition (pre-drug administration) and train our model f θ on data from multiple subjects, using each epoch to represent a different participant to capture features common across individuals. Data was pre-processed following standard procedure [9] and standardised before applying our method to learn emergent features.Human fMRI dataset. We also evaluate our method on a functional magnetic resonance imaging (fMRI) dataset from 100 unrelated human participants from the "minimally preprocessed" release by the Human Connectome Project [53, 20] . We performed additional preprocessing following Luppi et al. [31] and calculated 100 time series capturing the activity of each of the regions in the Schaefer brain atlas [45] . We further standardise the data before applying our method to learn emergent features from it.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
results: 
 In this section, we discuss results on learning emergent features on the synthetic and real world datasets introduced above, as well as comparisons to baselines and some ablation studies. All experiments can be run on a single A10G GPU in less than two hours. Bit-string dataset. Results show that our proposed architecture can accurately estimate the groundtruth value of Ψ in the synthetic dataset, confirming it is able to learn causally emergent representations (Fig. 2 ). To interpret the contents of the learned representation, we trained decoders with standard supervised learning to predict both the parity of the first n -1 bits (the parity bit) and the last auto-correlated bit (the extra bit). We found that the parity bit could be decoded with high accuracy but the extra bit could not, confirming that the learned representation indeed corresponded to an emergent feature. For additional verification, we fixed the representation learner's weights θ and re-trained both critics to estimate both Ψ and Ψ A , which were both positive and confirmed our results.As expected, when the marginal MI terms are removed from the objective function (Fig. 2 , right column), the model is no longer able to obtain the correct Ψ value -and, interestingly, only the extra bit (but not the parity bit) is encoded in the representation. We hypothesise that, in the absence of the regularisation induced by the marginal MI, the system's inductive biases lead it towards learning "low-order" (i.e. non-emergent) representations. Note that, despite having a constraint removed, the model without marginal MI loss is unable to extract the full predictive information of the system (which equals approximately 1.84 bit), showing that using the full emergence loss could incentivise the system to learn features that provide information about the system's dynamics that would otherwise be ignored (we elaborate on this further in Section 3.3). We obtain qualitatively similar results with a noisier version of the same data generating process (Supp. Fig. 5 ).Conway's Game of Life. By employing a convolutional neural network for the representation learner f θ , we successfully learned emergent features from this dataset, evidenced by a positive Ψ A value after freezing the representation network and retraining the critics (Supp. Fig. 8a ). To interpret the emergent feature, we attempted to predict the glider's position and state from the representation. While the position prediction performed at chance level, the state prediction achieved an accuracy of 53%, significantly higher than the random baseline of 25% (Supp. Fig. 8b ). This suggests that the emergent feature encodes information about the glider's state. When f θ is trained with a predictiononly objective, the state prediction achieved an accuracy of 35%, suggesting the learned feature encodes less information about the state of the glider when trained on this objective.Training steps (×10 Training two representation learners to learn independent emergent feature on the synthetic dataset. Both learned features were emergent (top row). The first learner (left column) yielded a feature that has high mutual information with the system's parity bit (bottom left), while for the second learner (right column) it had high mutual information with the bonus bit (bottom right).These findings demonstrate that our approach can learn meaningful emergent features in complex, high-dimensional systems like the Game of Life, capturing collective behaviors that are not localized to individual components.As our next step, we set out to test our algorithm to learn diverse features in the same bit-string dataset. For this set of experiments we used the representation learner with skip connections described in Section 2.2, ran our algorithm as above until convergence, and fixed the representation learner's weights (denoted by θ A ). We then trained a second representation learner with the regular Ψ S objective plus a penalty term I(V t ; V A t ) to obtain a new feature that is statistically independent from the parity bit.Interestingly, this process revealed a new, unexpected feature that was not originally designed (Fig. 3 ): the XOR of the parity and extra bit (which we will refer to simply as the bonus bit). This bit is emergent (since it cannot be predicted by any of the X i t and is auto-correlated, since both the parity and extra bits are), albeit with a lower Ψ than the parity bit -which fits well with the result that our model did not learn this feature spontaneously. This shows the capability of our method to discover new aspects of a system under study, even in the case of simple and explicitly constructed systems.Comparison with RNNs. In order to investigate whether the failure of the prediction-only baseline to learn emergent features was to do with the capacity of the architecture, we trained a standard RNN on the bit-string dataset using mean squared error (MSE) loss, with its hidden state serving as the representation for decoding. The hidden state dimension matched that of our emergent feature network f θ for a fair comparison. As shown in Fig. 4 , the RNN consistently learned the non-emergent extra bit but encoded negligible information about the two emergent bits, resulting in Ψ A = 0. This indicates that the RNN did not capture any emergent features. . Accordingly, the mutual information between the hidden state and the extra, parity, and bonus bits shows that only the nonemergent extra bit is encoded (middle). Interestingly, representations learned by an RNN and by our method can be combined to yield better predictions of the future state of the system (right).Combining emergent and non-emergent features. The fact that standard architectures learn representations that are predictive, yet not emergent, suggests they may be learning different aspects of the data compared to our method. To test this hypothesis, we investigated whether combining emergent features from our method with representations from standard models would enhance the predictive performance of either. As a proof of concept, we trained both an RNN and our emergence learner f θ on the bit-string dataset, and compared the mutual information between X t ′ and:1. The hidden state of the RNN.2. The emergent feature V t learned by f θ .3. The concatenation of V t with the hidden state of an RNN. 5As shown in Fig. 4 , the combined representation encoded more information about the future state of the system, X t ′ , than either the RNN or the emergence learner alone. While this result is only shown on a very simple synthetic dataset, it suggests that our emergence learner extracts meaningful features that are different from, and can be effectively combined with, other representation learning architectures. We hypothesise that standard representation learning techniques can more effectively capture microscale properties, and since our method captures macroscale properties, both can be naturally combined to enhance performance in downstream tasks.Ablations to emergence loss. The main emergence loss function we propose in this work has three terms: a predictive MI term estimating I(V t ; V t ′ ), marginal MI terms estimating I(X i t ; V t ′ ), and a diversity term estimating I(V A t ; V t ). In order to understand the contribution of each term to the behaviour of the model, we performed ablation studies removing each one of these. We describe each ablation in turn:1. As seen in Sec. 3.2 (and later in Sec. 3.4), ablating the I(V t ; V t ′ ) term from Ψ objective results in the network learning features that are not as correlated in time as they would otherwise be, but that still satisfy the criteria for emergence. Therefore, our proposed architecture with skip connections may learn slightly less emergent features, but with the advantage of more stable training dynamics.2. Ablating the I(X i t ; V t ′ ) terms objective results in an algorithm that only maximises predictive information I(V t ; V t ′ ), akin to the Deep InfoMax method [23] . As seen in Sec. 3.1, and as expected, training without the marginal terms results in representations that are not emergent.3. Ablating the diversity term in the experiments in Sec. 3.2 and training multiple runs on the bit-string dataset results in the representation network learning the extra bit 91% of runs, and the bonus bit 9% of runs. This implies there are strong inductive biases towards which emergent features are learned, and it is thanks to the diversity term that we can reliably recover both.Finally, we demonstrate the scalability of our method by learning emergent features in three types of high-dimensional brain activity datasets -ECoG, MEG, and fMRI -to evaluate its ability to learn emergent features across different neural recording modalities and spatial scales.ECoG data. For the ECoG dataset [10] , it was crucial to use skip connections in the representation learner, since the method as used in Section 3.1 was consistently unable to find emergent features (despite their known presence in this dataset [43] ).With this configuration, our method is able to successfully learn emergent representations of the ECoG data (Supp. Fig. 9a ). The resulting feature was verified to be emergent by a positive post-hoc Ψ A test. Interestingly, using an f θ with skip connections we obtained emergent features even when the macroscopic MI term I(V t ; V t+1 ) was completely removed from the objective function, which further simplified learning dynamics. Empirically, we found that this does not cause I S φ (f θ (X t ); f θ (X t+1 )) to decrease over training substantially, suggesting that using just the marginal MI terms is a good objective function to remove information about the parts while preserving information about the whole that was present at initialisation. As a final experiment, we also found that our method could learn a second emergent feature, also verified with a post-hoc Ψ A check (Supp. Fig. 10 ).MEG data. We extended our analysis to magnetoencephalography (MEG) recordings from healthy human participants [36] . Training on data from multiple subjects, our method readily learned emergent features, as evidenced by a positive post-hoc Ψ A check (Supp. Fig. 9b ). The consistency of emergent features across different individuals suggests that our model captures fundamental aspects of neural dynamics common to human brain activity.Notably, emergent features were learned more easily from MEG data compared to ECoG data, potentially due to differences in spatial resolution and the nature of the recorded signals.fMRI data. Finally, we applied our method to functional magnetic resonance imaging (fMRI) data from 100 unrelated human participants [45] . Despite the coarser spatial and temporal resolution of fMRI, our method was able to learn emergent features, as indicated by a positive Ψ A value (Supp. Fig. 9c ). This demonstrates that emergent dynamics are present even at the macroscopic scale of brain activity. On this dataset, we also determined that a standard MLP did not learn emergent features in its representation (see Supp. Sec. B.2 for more details).The successful extraction of emergent features across all three datasets highlights the robustness of our method. It underscores its capability to uncover collective neural dynamics in complex, high-dimensional brain activity data, regardless of the recording modality or spatial scale. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification:Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work and future directions: 
 The method we have proposed here is part of a growing literature leveraging methods from information theory to enhance deep learning architectures, and representation learning algorithms in particular. A small, far from exhaustive list of some noteworthy examples includes InfoNCE [38] , deep variational information bottleneck [2] , β-VAE [8] , or TC-VAE [11] . In future work, it would be interesting to see if our method is competitive or can be combined with recent representation learning methods on standard benchmarks [22, 38, 52] .In practice, one of the challenging aspects of this method is to overcome the instability of training to make sure a valid emergent solution is found (i.e. one where the post-hoc check still shows Ψ A > 0). One potential approach is to incorporate stabilisation techniques from the GAN literature (such as e.g. spectral normalisation [35] ). Another potential approach is to use a mutual information upper bound (instead of our current lower bound) for the marginal MI terms. Although mutual information upper bounds are not as common as lower bounds, there are a few options available [13, 12] . In practice, we found both of these to be less effective than our skip connection method, but they remain a promising avenue for future work.There is a growing body of machine learning research that engages directly with PID, the informationtheoretic backbone of our emergence theory. For example, recent work has proposed a differentiable redundancy measure [32] , which has been used as an objective function to train deep neural networks [21] . Alternatively, there are also methods that estimate redundancy using deep neural networks [26] , analogous to SMILE. Although none of these estimators can be directly applied to estimate the unique information that underlies the definition of emergence (Section 2.1), we expect that extensions of these exciting developments will also open new possibilities in the study of emergence.Finally, it is worth mentioning that there are a number of other approaches that focus on different aspects of emergence (see Supp. Sec. D). The approach presented here was chosen for two reasons: (i) its intuitive nature, focusing on how emergent properties arise from collective interactions that cannot be fully explained by examining components in isolation; and (ii) the existence of efficient, scalable, and differentiable proxies for its estimation. Constructing similar methods as the one proposed here for other metrics of emergence would be a challenging but extremely interesting line of future work. 
conclusion: 
 Emergence, the phenomenon whereby a system becomes 'more than the sum of its parts', is an promising conceptual tool to investigate cognitive processes in artificial and biological systems. In this paper, we proposed a machine learning method for discovering emergent variables in time series data that leverages a recent information-theoretic characterisation of emergence [43] and advances in mutual information estimation from data with neural networks [50] .We first showed in a synthetic dataset that our method can estimate emergence and successfully discover a known emergent feature. Interestingly, a pure information maximisation objective struggled to learn this feature, suggesting that our method facilitates the identification of complex features of the data. Furthermore, we also proposed a slight modification of our method that can learn a diverse set of features from the same system. Finally, we also showed that our method can scale up to learn emergent features in real-world brain activity data.Overall, our method opens up a range of possibilities for the practical study of emergence, as well as for other machine learning problems more broadly. For example, our method may be explored in conjunction with other representation learning algorithms to capture aspects of complex systems that are otherwise difficult to learn. From an application perspective, we hope this method can be further leveraged in neuroscience to reveal new aspects of brain function. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: No statistical claims are made, we simply demonstrate that certain conditions can be met using estimations of MI.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper does not further the capabilities of AI, and instead aims to develop methods that further understanding of complex systems. Our models have no path to harm.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The data and models have no such risks, they are small datasets of the ECoG and the models are small MLPs Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Algorithm 1: Pseudocode for the causally emergent representation learning algorithm.Input: Dataset D = {(x t , x t+1 )}, num_epochs, number of atoms n, Optional[pretrained representation network f θ A ] Initialize representation network f θ , predictive critic g ϕ , marginal critics {h ξ i } n , Optional[diversity critic k σ ] for epoch ← 1 to num_epochs do for x t , x t+1 in D do v t , v t+1 ← f θ (x t ), f θ (x t+1 ) Îvt,vt+1 ← g φ (v t , v t+1 )Update g φ to maximise SMILE lower bound (Eq. 4) Training steps (×10 3 )for i ← 1 to n do Îx i t ,vt+1 ← h ξ i (x i t , v t+1 ) end Update {h ξ i } n to maximise SMILE lower bound (Eq. 4) Ψ = Îvt,vt+1 - n i=1 Îx i t ,vt+1 if training diverse emergent feature then Îvt,f θ A ← k σ (v t , f θ A (x t )) loss ← -Ψ + Îvt,f θ A else loss ← - Ψ end Update f θ toΨ ATraining steps (×10 3 )I(V t ; V A t )Diversity term a) b) We further verify that the new feature V t is nearly independent from the previous one V A t , as their mutual information (estimated with SMILE and the critic k σ ) vanishes.In order to investigate further whether other representation learning methods could naturally learn emergent features, we also trained an encoder-decoder pair of multilayer perceptrons (MLPs) on the fMRI dataset to predict X t ′ from X t using an MSE objective. The encoder mapped X t to a representation V t with the same dimension as f θ 's output. A post-hoc test revealed that the encoder did not learn an emergent feature, as Ψ A < 0 (after training, the model stabilised at Ψ A = -28 ± 5), indicating that standard autoencoder-like architectures may not capture emergent properties in complex data.Let us present an informal derivation of the correction that leads to the refined criteria for emergnece presented in Eq. 2. This formulation relies on the Partial Information Decomposition framework [54] , on which our emergence metric Ψ is based.Let us simplify the notation and use B = V t+1 for the target variable and A i = X i t for the i-th source of information. Taking inspiration in Ref. [43, Appendix A], the total information about B provided by A := (A 1 , . . . , A n ) can be expressed asEQUATIONwhere Red(A; B) is the redundancy between sources, Syn(A; B) is their synergy, and Un(A; B) accounts for the unique information provided by each predictor and not the others. Of these, the synergistic component is the one associated to emergence by the theory in Ref. [43] . By using this decomposition and the chain rule of the mutual information [15] , one can find that I(A i ; B) = Red(A; B) + Θ i where Θ i accounts for other terms unique to source i, and henceEQUATIONThis shows that, when summing the marginal mutual informations, the term Red(A; B) gets doublecounted n times, resulting in Ψ containing this redundancy term 1n times. Hence, it is natural to propose a refined criteria for emergence that avoids this double-counting, given byEQUATIONSeveral frameworks have been proposed to formalize the concept of emergence in complex systems. Below, we provide a brief overview of some approaches that are particularly well-suited to investigate time-series.Barnett [3] introduces a formalization of emergence based on information theory, defining an emergent macroscopic process V t = f (X t ) as one where the dynamics of V t are conditionally independent of the microscopic history X - t , given its own history V - t . Accordingly, the condition for emergence is defined using transfer entropy [46, 7] as follows:EQUATION)This framework captures the idea that the macroscopic state's future is determined by its own history, and additional knowledge of the microstate does not enhance prediction.Computational mechanics [17, 18, 48] defines emergence through the concept of causal states, which are equivalence classes of histories ← -S leading to the same conditional distribution over futures -→ S . This leads to establish two keys quantities: the statistical complexity C µ and excess entropy E, which are calculated asEQUATIONAbove, the excess entropy quantifies how much information is shared between past and future, and the statistical complexity measures how much information processing is needed to enable such prediction.Building on these ideas, emergence has been characterised in different ways. One approach has been to identify a process S ′ t = f ( ← -S t ) as emergent when it increases the efficiency of prediction e = E/C µ compared to the microscopic description [48, Sec. 11.2] . Another approach has been to assign emergence when there is a new type of dynamics observed at the macroscopic level from the one observed at the micro [17, 44] ; for example, dynamical patterns associated to a different computational class [17, Sec. 5.1] . Finally, another approach has been to ascribe emergence to macro-processes whose causal states are a coarse-graining of the causal states of the micro [41] , which leads to a view closely related with the one presented in Section D.1.Seth [47] proposes Granger emergence, where a macroscopic process V t is emergent with respect to the microscopic process X t if it satisfies the following conditions:EQUATIONThe first condition ensures that V t contains predictive information about its future that is not present in X t , while the second condition establishes that V t arises from X t .Hoel et al. [24] introduce the concept of effective information, which measures the strength of dependencies using maximum entropy distributions [14] . More specifically, for a given probability kernel corresponding to p(X t+1 |X t ), the corresponding effective information is Ef(X) = I(X t ; X t+1 ) where X t is assumed to follow a uniform distribution. Then, emergence is assigned when the kernel p(V t+1 |V t ) that results when applying the coarse-grainingEQUATIONIn other words, there is more information going through the macroscopic than the microscopic level when considering maximum-entropy inputs to both of them.While these alternative formalisms provide valuable insights into emergence, they differ from the approach we use, which emphasizes the part-whole relationships by considering how the emergent feature encodes information about the microscopic components, particularly focusing on synergistic interactions. Our framework aligns with the view that emergent properties arise from collective interactions among parts that cannot be fully understood by examining components in isolation. Moreover, it is important to highlight that the metrics associated with most of these approaches are computationally expensive, and don't scale well with system size. Enabling similar methods as the one proposed here but with these other metrics of emergence, and comparing the resulting findings, would be an extremely interesting line for future work.Hyperparameters for the models trained on the synthetic and brain datasets can be seen in the below tables. The hyperparameters do not change when learning diverse sets of emergent features. For each training run the clip was fixed at 5 in the SMILE mutual information estimators and the optimizer used was AdamW. Justification:Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Alternatively, one could consider using a single critic h ξ shared across all X i t . We found this option to be computationally cheaper but less effective.In GANs this could lead to mode collapse[51].Although this could in principle lead to learned representations with reduced predictive information, in practice we observe this leads to increased stability and only small decrements in I(Vt; Vt+1).Specifically, the emergence metric is Ψ = 1 -H2(γparity) and the predictive information is I(Xt; Xt+1) = 2 -H2(γparity) -H2(γextra), where H2(p) represents the entropy of a Bernoulli distribution with parameter p. Here we set γparity = γextra = 0.99, and show results for other γparity, γextra values in the Appendix.We retrained both models using half the representation size, to ensure a fair comparison between all three cases. 


Decision: This work attempts to combine insights from the literature on emergence with differentiable estimators of mutual information to look for emergent features in time series data. The authors explore the presence of emergent features in some synthetic data sets and in data from neuroscience experiments. 

Reviewers appreciated the problem formulation and the combination of ideas from information decomposition with differentiable estimators. Reviewers also found the problem statement clear and of interest to the NeurIPS community. Additional experiments performed by the authors in response to reviews were also judged to substantially strengthen the paper. One weakness noted by all reviewers was the difficulty of interpreting what emergent features were learned in the real data, as the finding is based on the estimated value of $\Psi$, without any particular insight as to what that implies.

On balance, all reviewers considered this a valuable contribution, and I concur. Accept.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel objective function and deep learning architecture that are targeted to extract emergent latent representations from time series data. Motivation is very clear. The definition of emergent latent representation interesting and useful. The utilization of mutual information estimators (lower bounds thereof) is smart. Evaluations are restricted to a fully artificial and a fully neurobiological dataset.
strength_weakness: strength_weakness: System evaluations are minimalistic and not as convincing as I had hoped. Both, comparisons to potential baseline algorithms as well as more ablations are missing. 

Furthermore, one artificial dataset and one not well-motivated real-world neural dataset seems not enough to warrant publication. 

In particular, I would have expected at least one if not multiple DL/Neural Network baselines that do not pursue the information-theoretic objective but simply a standard temporal prediction objective. Those probably do not work on the parity problem at all but at least an attempt seems needed. That is, use a DREAMER-like world model learning architecture with probabilistic latents and see if structure emerges. 

Ablations could have explored more than just the same architecture without the penalty / adjustment term or without the macroscopic estimator. Further, in the artificial dataset the correlation coefficients $\gamma$ are quite high – was this necessary? When does this break down? Evaluations with a non-linear prediction pipeline would also be useful.
questions: questions: Emergence comes in many forms – I wonder if you could discuss alternative definitions / alternative perspectives on the matter?

Line 52 – a “not” too much. 

Eq (2): can you motivated the adjustment term further the summand (n-1)min_i… ? Are there alternatives to this that would be more targeted towards actually identifying true redundant mutual information? 

Eq (4). Could you motivate the clip operator slightly more? 

Line 102 should read “accurately”

Paragraph 113ff: maximize / minimize information – I am not sure if this is worded the right-way round – could you double check and slightly reword? 

Line 137 – should not read “also”

Line 142 – one “is” too much

Line 190ff. removing the macroscopic MI term seems not to save much – or does it? The observation is interesting, but I wonder if the authors want to make a computational argument here as well. 

About the biological dataset – this is very ad-hoc somewhat. What does this analysis tell us really except that there are some complex spatio-temporal statistical correlations in the data? I find this one marginally useful.
limitations: limitations: Unclear how robust this is as ablations and comparisons are not very extensive.
paper_summary: paper_summary: The paper presents a method for identifying emergent variables in time series data through a novel machine learning architecture. It uses unsupervised learning for representation and information theory to find emergent properties in systems, which are often complex and not easily describable at the microscale level. The paper is motivated by the fact that unsupervised learning can be a powerful tool for identifying emergent properties, but current approaches limit to only information theory.

The method rests on maximizing an objective defined by subtracting the mutual information of state variables at time t and the coarse graining at time t + 1 from the mutual information of the coarse grainings at t and t + 1. In other words, the amount of emergent information. Information theoretic definition of emergence is thus used to facilitate unsupervised learning. The method is tested on synthetic data and real-world ECoG brain activity data, demonstrating its ability to identify known and novel emergent features effectively.

Experiments are conducted on a synthetic dataset and a macaque brain activity dataset. For the synthetic dataset, the method is able to estimate the ground-truth value of psi (the difference that is central to the objective function). For ECoG data, skip connections were introduced into the architecture, and once again found emergent representations. 

The paper concludes with a discussion on related (info theory) work, limitations, and future steps.
strength_weakness: strength_weakness: ### Clarity
- It's not immediately clear how to interpret results. The paper shows figures, but it doesn't explain them much. Interpreting them requires a lot of re-reading the methods section
- Writing is somewhat verbose and unstructured, and occasionally reads like a process statement

### Quality 
- This idea is compelling and innovative! The loss built on MI of coarse grainings and state variables is intuitive while creating a solid foundation for taking advantage of the capabilities of unsupervised learning
- On the ECoG dataset, giving intuition/semantic understanding of emergent features (or at least attempting interpretation) would be cool
- Limited experiments on real data in general - ultimately, only one experimental setting is shown as far as I understand. The synthetic problem, while useful, is simple
- Lacking baselines or extensive comparison to existing methods, even if purely information-theoretic

### Significance
- It would help to have clearer comparison to existing methods so that we could see the value-add of this innovation, not just the novelty and value alone
questions: questions: None
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a method for learning the causally emergent representation of time series data. Based on the Partial Information Decomposition (PID) and ΦID definition of emergent variables, the paper utilizes variational information lower bounds to estimate and optimize the emergence objective function. The paper further includes a Minimum Mutual Information term and a penalty term, to reduce redundancy and discover diverse emergent variables, respectively. Experiments on a synthetic dataset and a primate brain activity dataset show that the method is able to discover diverse causally emergent representation.
strength_weakness: strength_weakness: As discussed above, the novelty is a little limited. This can be compensated by solid evaluations with a wide range of interesting datasets. I think the place the paper needs most improvements is more diverse and extensive evaluations. The paper can benefit from a few more datasets, both synthetic and real world, including the other datasets used in [1] and other references. If there exists baselines for discovering causally emergent representations, those baselines should also be compared against.



Reference:

1. Rosas, Fernando E., et al. "Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data." PLoS computational biology 16.12 (2020): e1008289.
questions: questions: N/A
limitations: limitations: The authors did not explicitly state the limitation of the paper.
paper_summary: paper_summary: The article proposes a learning scheme aimed at detecting emergent quantities from time series data of systems made of many interacting parts, such as the brain. To this end the authors combine "minimum mutual information", a previously introduced emergence criterion, with SMILE, a differentiable lower bound estimator for mutual information. Differentiability is crucial for the loss function to be optimizable efficiently. They apply this architecture to two examples: First, a series of random bit strings with time-correlated parity, where parity is considered the emergent quantity. Second, real-world data of macaque brain activity. The approach successfully identifies parity in the first example. The authors claim that an emergent feature has been learned for the second example also.
strength_weakness: strength_weakness: From the article alone, I could not fully understand the architecture and its training procedure that is illustrated in Fig. 1. I could not find code that would reproduce it, or a detailed pseudo-code description of the algorithm. It is unclear to me what emergent feature was found for the monkey example, or how Fig. 4 proves that any such feature was found. While the architecture and direction seems promising to me, a few more benchmarks would help make the case that this scheme can find emergent features in many settings. The two examples shown are one toy example with unnatural time dynamics and one real world example where it is hard to understand the dynamics from first principles. Benchmarking this new method on more standard examples with emergent behavior, such as Ising models, would be more convincing.
questions: questions: 1) How does Figure 4 prove that an emergent feature was found? What is the quantity on the x-axis in this figure?
2) Can you share the code of the architecture and its training or describe it in more detail?
3) Would you expect this method to find emergent features in standard, well-understood systems, such as Ising models?
4) Can you explicitly write down the prediction-only objective in Fig.2?

Overall, I find the approach promising but not yet tested on enough examples to support its ability to find emergent features.
limitations: limitations: There is no open access to the code.

Comments: title: Thanks for the response
comment: Great rebuttal with new datasets and results as well as very good answers!
The provided additional information make me confident that the final paper will turn out to be very good and very useful for the ML community - at least for those that are interested in emergence. Please make sure that you use the additional page left accordingly (submission was less than 8 pages long).
I raise my score by one (i.e., to 7 - accept).
Thanks for the response and the new work. I do think the new experiments help, and the writing looks better. In retrospect, I think I would have originally given a 5 or 6, and raised it to a 7, so I'm going to keep the current score.
Thanks for the response. The added experiments provide more solid support for the claims of the paper. Thus, I increased my score.
Thank you for your reply and clarifying my question regarding the notation.

On the overall improved and clearer presentation for the final version I will put my trust in the new github repository, and also in the judgement in some of the reviewers who already found the first version easier to read than I did. They seem more familiar with the previous work that your method is based on.

The new results, in particular using the game of life example, investigate exactly the kind of system I found missing as a benchmark in the original version of the article. I am going to raise the score in response.