SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: We conduct experiments of ablation study in six control environments including the number of samples using for supervised training, mean degree of networks (the higher mean degree results in more parents per child variable), coupling strength related to the extent of influence that parent variables have on child variables through the coupling drive, sampling frequency of activity sequence, measurement noise (Gaussian noise is added with mean zero and standard deviation of a certain proportion of the original sequence), and hidden variables (training or validation samples are randomly selected from the remaining observable variables, and test samples are randomly selected from the whole network). In the plots above, the default values of train samples is 20 (10/10 positive/negative samples), mean degree 10, coupling strength 0.2, sampling period 0.1,  $10\%$  measurement noise and  $0\%$  hidden variables, and each plot considers gradual variation of a single variable. Activity sequences are generated by the HR model on synthetic networks with 500 variables. The size of validation and test set is 100 and 400 respectively, with a uniform distribution of positive and negative samples as in the training set, which are randomly selected from all possible ordered pairs within the entire network.
4.2 Ablation study
Our method's innovation can be decomposed into three components: Firstly, a trainable transfer entropy calculator (TENE) is proposed; Secondly, TENE is combined with attention to detect region of high transfer entropy; Thirdly, the classifier is guided by coupling attention to focus on regions where the coupling effect may emerge. Here, we provide a detailed ablation study to determine which component is the key to performance improvement, and show results in Figure 3. We observe that Classifier alone (blue lines) exhibits the worse performances than our method (green lines) in most environments. This indicates that the spurious features learned by Classifier, despite quickly reaching low loss on the small training set, are less related to the properties of coupling effect and result in poor generalization capability on test set. When the size of train set increases, the performance of Classifier alone become comparable to that of ATEn (see Figure 3a). TENE (cyan lines) exhibits performance marginally superior to transfer entropy estimated using the standard Kraskov method (TE_Kriskov; purple lines). The ATEn (green lines) shows a substantial performance improvement compared with baselines, which indicates that the classifier with the guidance of coupling attention can extract features hidden in activity sequences that are closely related to coupling effect and result in superior generalization. This also demonstrates the importance of identifying and focusing on these critical regions of activity sequences. Moreover, our method is robust against changes to experimental conditions other than mean degree and measurement noise, a limitation we discuss in Sec. 5.
4.3 Insight into the coupling-drive of underlying dynamics
In Figure 4, we demonstrate the ability of coupling attention to catch the transient emergence of coupling effect. The gray lines in Figure 4 represent the change of coupling force from parent to child variable over time, and are generated by the coupling term  $\Gamma$  in Table 2. The absolute value of the coupling force rises (the gray lines spike) at occasional moments when the behavior of a
(a) Hindmarsh-Rose
(b) Izhikevich
(c) FitzHugh-Nagumo
Figure 4: Insight into the coupling drive of neural dynamics. Top panel in each subplot: segment of two activity sequences on the ordered pair of variables with a coupling relationship (blue is parent and green is child). Bottom panel in each subplot: the absolute value of coupling force (gray line), the coefficient of traditional attention (light pink line) and coupling attention (orange line) assigned along the sequences. (a) HR; (b) Izh; (c) FHN. Time  $t$  is index in the time series. In axis labels, "Power" is membrane potential, while "Strength" is membrane potential (attention coefficient) for the coupling force (attention). Results are chosen for clarity and are obtained when the training of stage 1 reaches stability.
parent variable substantially influences the evolution of its child variable, and remains almost zero at other times. The orange lines representing the coupling attention keep in step with the gray lines, indicating that the coupling attention mechanism recognizes the effect of coupling force in reducing the uncertainty of the child variable and pays attention to the areas where coupling force is significant. In contrast, the light pink lines in Figure 4 representing the traditional attention [29] remain close to their maximum value, indicating it is insensitive to changes in coupling force. This would lead its classifier to extract features from any part of the entire time series, rather than focusing on areas where coupling effects may emerge.
4.4 Performance on real networks
We test our method on six empirical neurological connectivity datasets using activity sequences data generated by four neuronal dynamic models. Compared with the baselines, our method usually substantially improves reconstruction performance on real coupled networks, as shown in Table 3. The performance of classical unsupervised methods, for which all positions in the time series are treated equally, is limited by the paucity of distinct coupling effects. In conclusion, our method slightly
Table 3: Performance comparison (model neurons coupled on real connectomes). Each point contains the mean and standard deviation of AUROC computed in ten experiments with randomly sampled training/validation/test set (20/100/1000) in C. elegans (left) and Drosophila (right) connectomes. Results on other three connectomes are shown in Appendix F.
Dataset	C.elegans	Drosophila
Dynamics	HR	Izh	Rulkov	FHN	HR	Izh	Rulkov	FHN
Granger	0.50±0.01	0.64±0.01	0.71±0.01	0.64±0.02	0.69±0.02	0.42±0.01	0.74±0.01	0.60±0.01
TE Kraskov	0.63±0.01	0.50±0.01	0.68±0.02	0.54±0.02	0.60±0.01	0.47±0.02	0.65±0.01	0.58±0.02
CCM	0.79±0.02	0.62±0.02	0.67±0.02	0.74±0.01	0.80±0.01	0.42±0.02	0.80±0.01	0.76±0.03
Latent CCM	0.78±0.01	0.55±0.04	0.53±0.04	0.68±0.01	0.81±0.02	0.47±0.02	0.76±0.01	0.73±0.01
PCMCI	0.53±0.01	0.53±0.02	0.66±0.01	0.57±0.02	0.51±0.01	0.51±0.01	0.51±0.01	0.51±0.01
PCMCI+	0.53±0.01	0.56±0.01	0.69±0.02	0.63±0.02	0.51±0.01	0.51±0.01	0.51±0.01	0.51±0.01
Classifier	0.75±0.06	0.65±0.05	0.72±0.06	0.65±0.07	0.85±0.05	0.65±0.06	0.83±0.09	0.72±0.05
TENE	0.65±0.06	0.43±0.03	0.65±0.08	0.47±0.06	0.54±0.07	0.51±0.03	0.54±0.11	0.41±0.06
ATEn	0.94±0.06	0.66±0.12	0.90±0.07	0.91±0.06	0.97±0.03	0.74±0.09	0.87±0.06	0.83±0.10
increases cost, due to the need for label collection, but obtains a substantial boost in performance compared with those unsupervised methods in this class of coupling network reconstruction tasks. We also observe that the reconstruction performances on identical dynamics across distinct connectomes are significant different (e.g., AUROC of ATEn on Izh across C.elegans in Table 3 and Rat in Appendix Table 5), indicating that the underlying network architecture plays an crucial role in shaping the behavior of coupling dynamics.
Our methodology has limitations (i.e., cases for which performance improvement is less): 1. Dense networks, where a variable is coupled with many driving variables and substantial coupling forces can emerge from distinct parents at overlapping times, making individual drivers harder to distinguish (see green line in Figure 3b); 2. Intense noise, which makes the coupling attention mechanism falsely identify high transfer entropy regions. The downstream classifier then extracts spurious features, leading to the reduction of its generalization (see Figure 3e); 3. Strongly coupled systems, which are dominated by synchronization phenomena in which the dynamic behaviors of all variables are similar.
In addition, we assume the state evolution of variables in the coupled network is uniformly governed by a group of dynamic equations. However, if the evolution behaviors of variables in the system obey different underlying dynamics, it becomes critical to carefully select the small training samples, in which the patterns of coupled driving effects are representative. Failure to do so may result in the poor generalization of the classifier. That is our next research direction.
Coupled Network Reconstruction. Several common methods of causal inference [31, 4, 6, 9] are based on conditional independence relations, but differ in the design of condition-selection strategies or choice of conditional independence test. Granger Causality [1] is extended to nonlinear dynamics by using neural networks to represent nonlinear casual relationships [8, 5]. Many methods of causal discovery assume that the coupled network is a directed acyclic graph. However, directed cyclic graphs are common in real systems. Conventional frameworks assume separability, i.e., that information about coupling drive are not contained in the parent variable itself. To address the non-separability issue, Convergent-cross mapping [3] and its variations [32, 27] measure the extent to which the historical record of child can reliably estimate states of the parent in reconstructed state space. However, sparse causal effect in neuronal dynamics, particularly in the presence of noise, may lead parent and child time series to appear statistically independent, so that their contribution to state estimation is hard to recognize.
Mutual Information Estimation. Ref. [23] built on a dual representation of KL divergence [22] to offer a parametric mutual information neural estimator (MINE) which is linearly scalable in dimension as well as sample size, and is also trainable and strongly consistent. They also discussed another version of MINE based on the  $f$ -divergence representation [33, 20]. Using the technique of Noise-Contrastive Estimation (NCE) [34], based on comparing target and randomly chosen negative samples, Van den Oord et al. [35] proposed InfoNCE loss, minimization of which maximizes a mutual information lower bound. An important application of this contrastive learning approach has been extracting high-level representations of different data modalities [36, 37, 38, 39]. In our work, we extend MINE for transfer entropy estimation.
Attention Mechanisms. The attention mechanisms identify key areas in the data by learning a set of weight distributions. Spatial-based attention [29, 40, 41] involves generating attention scores from spatial regions of feature maps, while channel-based attention [42, 43] optimises the representation of each channel. Self-attention [44, 45] encodes interactions among all input entities and cross-attention [46, 47, 48] introduces the interaction of two domains further. However, it is recognised that attention mechanisms need to be tailored to the specific problem at hand [49]. In our work, we tailor attention mechanism for coupling relationship inference by accommodating the selection of temporal regions that correspond to the transient emergence of coupling effect in neural activity sequences.
The problem of reconstructing coupling networks from observational data is fundamental in multiple disciplines of science including neuroscience, since it is a prerequisite foundation for the research about structure analysis and behavior control in coupling networks. Especially, several countries have recently launched grand brain projects, and one important goal is to map the connectomes (i.e., directed links between neurons) of different species.
We draw on coupling attention to guide machine learning models to infer coupling relationships while focusing on the specific areas where casual effect may emerge. We show that this mechanism identifies weak coupling effects ignored by classical techniques, and helps machine learning models gain insight into the coupling dynamics underlying time series data.
This work is supported by STI2030-Marjor Project 2021ZD0204500, National Natural Science Foundation of China (Grants No. T2225022, No. 12161141016, No. 12150410309), Shuguang Program of Shanghai Education Development Foundation and Shanghai Municipal Education Commission (Grant No. 22SG21), Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZDZX0100), Shanghai Municipal Commission of Science and Technology Project (Grant No. 19511132101), and the Fundamental Research Funds for the Central Universities.
[1] Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. *Econometrica: Journal of the Econometric Society*, pages 424-438, 1969.
[2] Thomas Schreiber. Measuring information transfer. Physical Review Letters, 85(2):461, 2000.
[3] George Sugihara, Robert May, Hao Ye, Chih-hao Hsieh, Ethan Deyle, Michael Fogarty, and Stephan Munch. Detecting causality in complex ecosystems. science, 338(6106):496-500, 2012.
[4] Jie Sun, Dane Taylor, and Erik M Bollt. Causal network inference by optimal causation entropy. SIAM Journal on Applied Dynamical Systems, 14(1):73-106, 2015.
[5] Meike Nauta, Doina Bucur, and Christin Seifert. Causal discovery with attention-based convolutional neural networks. Machine Learning and Knowledge Extraction, 1(1):312-340, 2019.
[6] Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. Detecting and quantifying causal associations in large nonlinear time series datasets. Science Advances, 5(11):eaau4996, 2019.
[7] Andreas Gerhardus and Jakob Runge. High-recall causal discovery for autocorrelated time series with latent confounders. Advances in Neural Information Processing Systems, 33:12615-12625, 2020.
[8] Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaie, and Emily B Fox. Neural Granger causality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4267-4279, 2021.
[9] Atalanti A Mastakouri, Bernhard Schölkopf, and Dominik Janzing. Necessary and sufficient conditions for causal feature selection in time series with latent common causes. In International Conference on Machine Learning, pages 7502-7511. PMLR, 2021.
[10] Lipeng Cui and Jack Murdoch Moore. Causal network reconstruction from nonlinear time series: A comparative study. International Journal of Modern Physics C, 32(04):2150049, 2021.
[11] Shohei Shimizu, Patrik O Hoyer, Aapo Hyvarinen, Antti Kerminen, and Michael Jordan. A linear non-Gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
[48] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. In International Conference on Learning Representations, 2022.
[49] Mohammed Hassanin, Saeed Anwar, Ibrahim Radwan, Fahad S Khan, and Ajmal Mian. Visual attention methods in deep learning: An in-depth survey. arXiv preprint arXiv:2204.07756, 2022.