SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Correlation Matrix as the Initialization of the Adjacency Matrix in AMAG. A well-initialized adjacency matrix in AMAG contributes to a more stable training process. We investigate two initialization approaches: random and correlation matrix initialization. Both initialization methods result in a similar performance in terms of the final  $R^2$  scores, i.e., 0.659 (randomly initialized) vs. 0.658 (correlation matrix initialized). However, their learning curves exhibit distinct characteristics, as depicted in Fig. 7. When the adjacency matrix is initialized randomly (dashed curves), both training and testing curves display more fluctuations. In contrast, when the adjacency matrix is initialized with the correlation matrix (solid lines), the learning curves become smoother. This indicates that the correlation matrix initialization promotes a more stable and consistent learning process. Furthermore, examining the  $R^2$  values in the initial epochs, we observe that the model initialized with the correlation matrix achieved higher performance compared to the random initialization. This finding suggests that careful initialization with the correlation matrix can accelerate the training process, leading to improved performance during the early stages of training.
Adjacency Matrix Before and After Training. We visualize the learned adjacency matrix alongside the correlation matrix in Fig. 8. We observe that the learned adjacency matrix is more sparse compared to the original correlation matrix. This indicates that during the training process, AMAG prunes and
Figure 8: Illustration of correlation matrix as the initialization of the adjacency matrix in AMAG and the adjacency matrix after training. The learned adjacency matrix is more sparse compared with the correlation matrix (fewer values close to 1).
refines the correlation matrix, identifying the relevant and informative connections between neural channels for forecasting of neural signals.
Appendix C Analysis of Forecasting Error
We provide the following proofs to understand what would be the upper bound error of the predicted neural activity. Considering the complexity of the AMAG model study, the theoretical guarantee of the temporal forecasting tasks could be a potential research direction. As a starting point, we simplify the problem as follows: considering the one-step forecasting case and removing the nonlinearities in the model, the sample-dependent weight matrix, and the multiplicative weight matrix.
Assume we have neural signal recorded from  $C$  channels at time step  $t$ ,  $\mathbf{X}_t \in \mathbb{R}^{C \times 1}$ , output of TE layer,  $\mathbf{H}_t \in \mathbb{R}^{(C \times D)}$ .  $(\mathbf{H}_0 = 0)$ . The  $\mathbf{H}_t$  is updated with Linear RNN, such that  $\mathbf{H}_t = \mathbf{X}_t W_1^e + \mathbf{H}_{t-1} W_2^e$ , where  $W_1^e$  and  $W_2^e$  are weights of linear RNN. Outputs of SI and TR are represented as  $\mathbf{Z}_t$  and  $\mathbf{R}_t$  respectively.
$$
Z _ {t} = A H _ {t}
$$
$$
\boldsymbol {R} _ {t} = \boldsymbol {Z} _ {t} W _ {1} ^ {r} + \boldsymbol {R} _ {t - 1} W _ {2} ^ {r}
$$
$$
\pmb {R} _ {0} = 0
$$
$$
\boldsymbol {Z} _ {t}, \boldsymbol {R} _ {t} \in \mathbb {R} ^ {(C \times D)}
$$
We rewrite the  $H_{t}$  and  $\mathbf{Z}_t$  such that they only depend on  $\pmb{X}$
$$
\boldsymbol {H} _ {t} = \sum_ {\tau = 1} ^ {t} \boldsymbol {X} _ {\tau} W _ {1} ^ {e} \left(W _ {2} ^ {e}\right) ^ {t - \tau}
$$
$$
\boldsymbol {Z} _ {t} = \sum_ {\tau = 1} ^ {t} \boldsymbol {A} \boldsymbol {X} _ {\tau} W _ {1} ^ {e} \left(W _ {2} ^ {e}\right) ^ {t - \tau}
$$
$$
\begin{array}{l} \boldsymbol {R} _ {t} = \sum_ {\tau = 1} ^ {t} \boldsymbol {Z} _ {\tau} W _ {1} ^ {e} \left(W _ {2} ^ {e}\right) ^ {t - \tau} \\ = \sum_ {\tau = 1} ^ {t} \left(\sum_ {q = 1} ^ {\tau} \boldsymbol {A} \boldsymbol {X} _ {q} W _ {1} ^ {e} \left(W _ {2} ^ {e}\right) ^ {\tau - q}\right) W _ {1} ^ {r} \left(W _ {2} ^ {r}\right) ^ {t - \tau} \\ \end{array}
$$
The Predicted signal at the time  $t + 1$ ,  $\hat{\pmb{X}}_{t + 1} = R_t\pmb {W}$ . Since we assume the model is stable, such that all the weight matrices should be bounded by  $\mathbf{M}$ , i.e.,  $\| W_1^e\| _2,\| W_2^e\| _2,\| W_1^r\| _2,\| W_2^r\| _2\leq 1$ . Besides,
to make sure the RNN is stable, we need to constrain the eigenvalue of for  $W_1^e, W_2^e, W_1^r, W_2^r$  to be smaller than 1, for example,  $\forall \lambda_i \in \mathrm{eig}(W_1^r), \lambda_i < 1$ . With the assumption, we remove the term  $(W_2^e)^n, (W_2^r)^n, n > 2$  which corresponds to the contribution of previous t-2 steps to  $t + 1$  step prediction. Thus we have
$$
\hat {\boldsymbol {X}} _ {t + 1} = \left(A X _ {t - 1} W _ {1} ^ {e} W _ {1} ^ {r} W _ {2} ^ {r} + A X _ {t - 1} W _ {1} ^ {e} W _ {2} ^ {e} W _ {1} ^ {r} + A X _ {t} W _ {1} ^ {e} W _ {1} ^ {r}\right) W.
$$
We rewrite the  $X_{t}$  and  $X_{t + 1}$  in form of early step plus a variation term, such that
$$
\begin{array}{l} X _ {t} = X _ {t - 1} + \delta_ {1} \\ X _ {t + 1} = X _ {t} + \delta_ {2} \\ = X _ {t - 1} + \delta_ {1} + \delta_ {2} \\ \end{array}
$$
The prediction error as  $\epsilon = \hat{X}_{t + 1} - X_{t + 1}$
$$
\epsilon = \left(A X _ {t - 1} W _ {1} ^ {e} W _ {1} ^ {r} W _ {2} ^ {r} + A X _ {t - 1} W _ {1} ^ {e} W _ {2} ^ {e} W _ {1} ^ {r} + A \left(X _ {t - 1} + \delta_ {1}\right) W _ {1} ^ {e} W _ {1} ^ {r}\right) W - X _ {t - 1} - \delta_ {1} - \delta_ {2}.
$$
Considering the first three terms that decide the  $\epsilon$ , we apply Cauchy-Schwarz inequality to get the upper bound,
$$
\begin{array}{l} \left\| \left(A X _ {t - 1} W _ {1} ^ {e} W _ {1} ^ {r} W _ {2} ^ {r} + A X _ {t - 1} W _ {1} ^ {e} W _ {2} ^ {e} W _ {1} ^ {r} + \right. \right. \\ A \left(X _ {t - 1} + \delta_ {1}\right) W _ {1} ^ {e} W _ {1} ^ {r}) W \| _ {2} \leq 3 \| A \| _ {2} \| X _ {t - 1} \| _ {2} + \| A \| _ {2} \| \delta_ {1} \| _ {2} \\ \end{array}
$$
$$
\| \epsilon \| _ {2} \leq 3 \| A \| _ {2} \| X _ {t - 1} \| _ {2} + \| A \| _ {2} \| \delta_ {1} \| _ {2} + \| X _ {t - 1} \| _ {2} + \| \delta_ {1} \| _ {2} + \| \delta_ {2} \| _ {2}
$$
So if the signal is smooth enough that that  $||\delta_1||_2 + ||\delta_2||_2$  is small, then the prediction error is bounded by the adjacency matrix  $A$ .
[1] Larry F Abbott. "Lapicque's introduction of the integrate-and-fire model neuron (1907)". In: Brain research bulletin 50.5-6 (1999), pp. 303-304.
[2] Evan Archer, Il Memming Park, Lars Buesing, John Cunningham, and Liam Paninski. "Black box variational inference for state space models". In: arXiv preprint arXiv:1511.07367 (2015).
[3] Navid Ayoobi and Elnaz Banan Sadeghian. "A self-paced BCI system with low latency for motor imagery onset detection based on time series prediction paradigm". In: arXiv preprint arXiv:2204.05450 (2022).
[4] Edoardo Balzani, Jean Paul Noel, Pedro Herrero-Vidal, Dora E Angelaki, and Cristina Savin. "A probabilistic framework for task-aligned intra- and inter-area neural manifold estimation". In: arXiv preprint arXiv:2209.02816 (2022).
[5] Rahul Biswas and Eli Shlizerman. "Statistical perspective on functional and causal neural connectomics: A comparative study". In: Frontiers in Systems Neuroscience 16 (2022), p. 817962.
[6] Rahul Biswas and Eli Shlizerman. "Statistical perspective on functional and causal neural connectomics: The Time-Aware PC algorithm". In: PLOS Computational Biology 18.11 (2022), e1010653.
[7] Martin Bjerke, Lukas Schott, Kristopher T Jensen, Claudia Battistin, David A Klindt, and Benjamin A Dunn. "Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles". In: arXiv preprint arXiv:2210.03155 (2022).
[8] Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. "Conditional time series forecasting with convolutional neural networks". In: arXiv preprint arXiv:1703.04691 (2017).
[9] Nicolas Brunel. "Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons". In: Journal of computational neuroscience 8 (2000), pp. 183-208.
[10] Luis Carrillo-Reid, Shuting Han, Dark O'Neil, Ekaterina Taralova, Tony Jebara, and Rafael Yuste. "Identification of pattern completion neurons in neuronal ensembles using probabilistic graphical models". In: Journal of Neuroscience 41.41 (2021), pp. 8577-8588.