SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evren Gokcen<sup>1</sup>, Anna I. Jasper<sup>2</sup>, Alison Xu<sup>2</sup>, Adam Kohn<sup>*2,3,4</sup>, Christian K. Machens<sup>*5</sup>, Byron M. Yu<sup>*1,6</sup>  
<sup>1</sup>Dept. of Electrical and Computer Engineering, Carnegie Mellon University  
<sup>2</sup>Dominick Purpura Dept. of Neuroscience, <sup>3</sup>Dept. of Ophthalmology and Visual Sciences, <sup>4</sup>Dept. of Systems and Computational Biology, Albert Einstein College of Medicine  
<sup>5</sup>Champalimaud Neuroscience Programme, Champalimaud Foundation  
<sup>6</sup>Dept. of Biomedical Engineering, Carnegie Mellon University  
egokcen@cmu.edu, {anna.jasper, alison.xu, adam.kohn}@einsteinmed.edu, christian.machens@neuro.fchampalimaud.org, byronyu@cmu.edu  
*Denotes equal contribution.
Modern recording techniques now allow us to record from distinct neuronal populations in different brain networks. However, especially as we consider multiple (more than two) populations, new conceptual and statistical frameworks are needed to characterize the multi-dimensional, concurrent flow of signals among these populations. Here, we develop a dimensionality reduction framework that determines (1) the subset of populations described by each latent dimension, (2) the direction of signal flow among those populations, and (3) how those signals evolve over time within and across experimental trials. We illustrate these features in simulation, and further validate the method by applying it to previously studied recordings from neuronal populations in macaque visual areas V1 and V2. Then we study interactions across select laminar compartments of areas V1, V2, and V3d, recorded simultaneously with multiple Neuropixels probes. Our approach uncovered signatures of selective communication across these three areas that related to their retinotopic alignment. This work advances the study of concurrent signaling across multiple neuronal populations.
Cortical circuits functionally involve feedforward, feedback, and horizontal interactions between many neuronal populations that span distinct areas and layers. Recording techniques now allow us to record from many neurons across these populations [1-3] (Fig. 1a). To capitalize on the scientific opportunities presented by these recordings, however, new conceptual and statistical frameworks are needed, particularly as we consider communication across multiple (more than two) populations.
Characterizing interactions between just two populations is a challenging high-dimensional problem. Dimensionality reduction techniques have therefore been increasingly used for this purpose [4-7]. Methods like canonical correlation analysis (CCA) [8] and its probabilistic variants [9], in particular, identify a low-dimensional set of latent variables that parsimoniously describe the interactions between two populations. This cross-population shared-latent model has inspired several extensions targeted toward neural recordings [10-14].
Communication between two populations, however, occurs bidirectionally and likely concurrently, and disentangling this concurrent communication is a substantial challenge in neuroscience. A
C
Figure 1: Challenges in studying concurrent signaling across multiple populations of neurons. (a) High-dimensional, simultaneous recordings across multiple neuronal populations. (b) Disentangling concurrent signal flow. (c) Distinguishing network-level interactions.
recently developed dimensionality reduction approach, delayed latents across groups (DLAG) [15], addresses this challenge by leveraging two insights: (1) communication between two populations is not instantaneous, and (2) while concurrently relayed signals might be difficult to tease apart from the raw neural activity (Fig. 1b, black trajectories in each population space), they could be pinpointed along certain dimensions or axes (Fig. 1b, "latent activity" measured along the magenta and gray dimensions; note how activity in one population leads activity in the other population).
Pairwise methods are ill-equipped, however, to analyze communication across three or more recorded populations. Suppose, for example, that we wish to study the interactions of three recorded populations, A, B, and C (Fig. 1c). One could consider applying one of the aforementioned methods to each pair of populations. However, we would encounter the following interpretational ambiguity. Suppose that populations A and B exhibit shared activity fluctuations, and populations A and C also exhibit shared fluctuations. Do populations A, B, and C all co-fluctuate together (Fig. 1c, right; black arrows: direction of influence)? Or do A and B co-fluctuate in a way that is uncorrelated with the way in which A and C co-fluctuate (Fig. 1c, left)? Only by analyzing all populations together can we differentiate these possibilities. Multi-population dimensionality reduction approaches, such as group factor analysis (GFA) [16], could be applied toward that end, but the challenge of disentangling concurrent, bidirectional signaling remains. New approaches are needed to jointly characterize the multi-dimensional, concurrent flow of signals among multiple populations.
We therefore propose multi-population DLAG (mDLAG), a dimensionality reduction framework that determines (1) the subset of populations described by each latent dimension, (2) the direction of signal flow among those populations, and (3) how those signals evolve over time within and across experimental trials. We illustrate these features in simulation, and further validate mDLAG by applying it to previously studied recordings from neuronal populations in macaque visual areas V1 and V2. Then we study interactions across select laminar compartments of areas V1, V2, and V3d, recorded simultaneously with multiple Neuropixels probes. mDLAG uncovered signatures of selective communication across these three areas that related to their retinotopic alignment. Throughout the analyses of simulated data and neural recordings, mDLAG provided improved interpretation and quantitative performance relative to alternative approaches. This work advances the study of concurrent signaling across multiple neuronal populations and its role in brain function.
2 Delayed latents across multiple groups (mDLAG)
Observation model and automatic relevance determination For population  $m$  (comprising  $q_{m}$  neurons) at time  $t$  on trial  $n$ , we define a linear relationship between observed activity,  $\mathbf{y}_{n,t}^{m} \in \mathbb{R}^{q_{m}}$ , and latent variables,  $\mathbf{x}_{n,t}^{m} \in \mathbb{R}^{p}$  (Fig. 2a):
$$
\mathbf {y} _ {n, t} ^ {m} = C ^ {m} \mathbf {x} _ {n, t} ^ {m} + \mathbf {d} ^ {m} + \varepsilon^ {m} \tag {1}
$$
$$
\varepsilon^ {m} \sim \mathcal {N} (\mathbf {0}, \left(\Phi^ {m}\right) ^ {- 1}) \tag {2}
$$
C
Figure 2: Delayed latents across multiple groups (mDLAG) (a) mDLAG directed graphical model representation. Filled circles represent observed variables. Unfilled circles represent probabilistic latent variables and parameters. Black dots represent point estimates. Arrows indicate conditional dependence relationships between variables. (b) Example mDLAG loading matrix (here the loading matrices for individual populations,  $C^1$ ,  $C^2$ , and  $C^3$  have been concatenated vertically). Each element of the matrix is represented by a square: magnitude is represented by the square's area, and sign is represented by the square's color (red: positive; blue: negative). Note the population-wise sparsity pattern of each latent variable, which is estimated from the neural activity. (c) Gaussian process auto-covariance (top) and cross-covariance (bottom) functions in the mDLAG model. Here we chose the squared exponential covariance function. The width of the auto- and cross-covariances corresponds to a timescale parameter  $(\tau_j)$ . The center of the cross-covariance between populations  $m_1$  and  $m_2$  (e.g., population A and B, respectively) corresponds to the relative time delay  $D_{m_2,j} - D_{m_1,j}$ .
where the loading matrix  $C^m \in \mathbb{R}^{q_m \times p}$  and mean parameter  $\mathbf{d}^m \in \mathbb{R}^{q_m}$  are model parameters.  $\varepsilon^m$  is a zero-mean Gaussian random variable with noise precision matrix  $\Phi^m \in \mathbb{S}^{q_m \times q_m}$  ( $\mathbb{S}^{q_m \times q_m}$  is the set of  $q_m \times q_m$  symmetric matrices). As we will describe, at time point  $t$ , latent variables  $\mathbf{x}_{n,t}^m$ ,  $m = 1, \dots, M$  are coupled across populations, and thus each population has the same number of latents,  $p$ . Because we seek a low-dimensional description of neural activity, the number of latent variables is less than the number of neurons, i.e.,  $p < q$ , where  $q = \sum_{m} q_m$ .
A core goal of the mDLAG framework is to identify and distinguish multiple network-level interactions (Fig. 1c). To do so requires identifying the number of latent variables across all recorded populations, and which subset of populations each latent involves, in a computationally tractable manner. We take a Bayesian approach to this problem (thereby avoiding computationally intensive grid search, see Discussion), and let  $\mathbf{d}^m$ ,  $\Phi^m$ , and  $C^m$  be probabilistic parameters with prior distributions.
The parameter  $\mathbf{d}^m$  describes the mean firing rate of each neuron. We set a Gaussian prior over  $\mathbf{d}^m$ :
$$
P \left(\mathbf {d} ^ {m}\right) = \mathcal {N} \left(\mathbf {d} ^ {m} \mid \mathbf {0}, \beta^ {- 1} I _ {q _ {m}}\right) \tag {3}
$$
where  $\beta \in \mathbb{R}_{>0}$  is a hyperparameter, and  $I_{q_m}$  is the  $q_{m} \times q_{m}$  identity matrix. We constrain the precision matrix  $\Phi^m = \mathrm{diag}(\phi_1^m, \ldots, \phi_{q_m}^m)$  to be diagonal to capture variance that is independent to each neuron. This constraint encourages the latent variables to explain as much of the shared variance among neurons as possible. We set the conjugate Gamma prior over each  $\phi_i^m$ , for each neuron  $i = 1, \ldots, q_m$ :
$$
P \left(\phi_ {i} ^ {m}\right) = \Gamma \left(\phi_ {i} ^ {m} \mid a _ {\phi}, b _ {\phi}\right) \tag {4}
$$
where  $a_{\phi},b_{\phi}\in \mathbb{R}_{>0}$  are hyperparameters.
The loading matrix  $C^m$  linearly combines latent variables and maps them to observed neural activity. In particular, the  $j^{\mathrm{th}}$  column of  $C^m$ ,  $\mathbf{c}_j^m \in \mathbb{R}^{q_m}$ , maps the  $j^{\mathrm{th}}$  latent variable  $x_{n,j,t}^m$  to population  $m$ . To determine which subset of populations is described by each latent, we employ automatic relevance determination (ARD), which has been used successfully in a variety of contexts [17, 18]. Specifically, we define the following prior over the columns of each  $C^m$ :
$$
P \left(\mathbf {c} _ {j} ^ {m} \mid \alpha_ {j} ^ {m}\right) = \mathcal {N} \left(\mathbf {c} _ {j} ^ {m} \mid \mathbf {0}, \left(\alpha_ {j} ^ {m}\right) ^ {- 1} I _ {q _ {m}}\right) \tag {5}
$$
$$
P \left(\alpha_ {j} ^ {m}\right) = \Gamma \left(\alpha_ {j} ^ {m} \mid a _ {\alpha}, b _ {\alpha}\right) \tag {6}
$$
<sup>1</sup>We will define all variables as they appear, but see Supplementary Section S1 for more complete notation.
where  $\alpha_{j}^{m}\in \mathbb{R}_{>0}$  is the ARD parameter for latent variable  $j$  and population  $m$ , and  $a_{\alpha},b_{\alpha}\in \mathbb{R}_{>0}$  are hyperparameters. As  $\alpha_{j}^{m}$  becomes large, the magnitude of  $\mathbf{c}_j^m$  becomes concentrated around 0, and hence the  $j^{\mathrm{th}}$  latent variable  $x_{n,j,t}^{m}$  will have a vanishing influence on population  $m$ . The ARD prior encourages population-wise sparsity for each latent variable during model fitting (see below), where the loading matrix coefficients will be pushed toward zero for latent variables that explain an insignificant amount of shared variance within a population, and remain non-zero otherwise (Fig. 2b).
The parameter  $C^m$  also has an intuitive geometric interpretation. Each element of  $\mathbf{y}_{n,t}^m$ , the activity of each neuron in population  $m$ , can be represented as an axis in a high-dimensional population activity space (Fig. 1b). Then each column of  $C^m$  defines a dimension in population  $m$ 's activity space (Fig. 1b, "activity pattern"). Dimensions that appear in two or more populations represent patterns of activity that are correlated across populations. Note that the columns of  $C^m$  are linearly independent; but they are not, in general, orthogonal. The ordering of these columns, and of the corresponding latent variables, is arbitrary.
State model For each latent variable, we seek to characterize the direction of signal flow among the involved populations (determined by ARD) and how those signals evolve over time within and across trials. We therefore employ Gaussian processes (GPs) [19], and define a GP over all time points  $t = 1,\dots ,T$  for each latent variable  $j = 1,\ldots ,p$  as follows (Fig. 2c):
$$
\left[ \begin{array}{c} \mathbf {x} _ {n, j,:} ^ {1} \\ \vdots \\ \mathbf {x} _ {n, j,:} ^ {M} \end{array} \right] \sim \mathcal {N} \left(\mathbf {0}, \left[ \begin{array}{c c c} K _ {1, 1, j} & \dots & K _ {1, M, j} \\ \vdots & \ddots & \vdots \\ K _ {M, 1, j} & \dots & K _ {M, M, j} \end{array} \right]\right) \tag {7}
$$
Under equation 7, latents are independent and identically distributed across trials. The diagonal blocks  $K_{1,1,j} = \dots = K_{M,M,j} \in \mathbb{S}^{T \times T}$  describe the autocovariance of each latent, and each  $T$ -by-  $T$  off-diagonal block describes the cross-covariance that couples two populations.
To define these matrices, we introduce additional notation. Specifically, we indicate populations with two subscripts,  $m_{1} = 1,\ldots ,M$  and  $m_{2} = 1,\dots ,M$ . Then, we define  $K_{m_1,m_2,j}\in \mathbb{R}^{T\times T}$  to be either the auto- or cross-covariance matrix between latent variable  $\mathbf{x}_{n,j,:}^{m_1}\in \mathbb{R}^T$  in population  $m_{1}$  and latent variable  $\mathbf{x}_{n,j,:}^{m_2}\in \mathbb{R}^T$  in population  $m_{2}$  on trial  $n$ . We choose to use the squared exponential function for GP covariances (Fig. 2c). Therefore, element  $(t_{1},t_{2})$  of each  $K_{m_1,m_2,j}$  can be computed as follows [15, 20]:
$$
k _ {m _ {1}, m _ {2}, j} \left(t _ {1}, t _ {2}\right) = \left(1 - \sigma_ {j} ^ {2}\right) \exp \left(- \frac {\left(\Delta t\right) ^ {2}}{2 \tau_ {j} ^ {2}}\right) + \sigma_ {j} ^ {2} \cdot \delta_ {\Delta t} \tag {8}
$$
$$
\Delta t = \left(t _ {2} - D _ {m _ {2}, j}\right) - \left(t _ {1} - D _ {m _ {1}, j}\right) \tag {9}
$$
where the characteristic timescale,  $\tau_{j}\in \mathbb{R}_{>0}$ , and the GP noise variance,  $\sigma_j^2\in (0,1)$ , are deterministic model parameters to be estimated from neural activity.  $\delta_{\Delta t}$  is the Kronecker delta, which is 1 for  $\Delta t = 0$  and 0 otherwise.
We also introduce two new parameters: the time delay to population  $m_1$ ,  $D_{m_1,j} \in \mathbb{R}$ , and the time delay to population  $m_2$ ,  $D_{m_2,j} \in \mathbb{R}$ . Notice that, when computing the auto-covariance for population  $m$  (i.e., when  $m_1 = m_2 = m$ ; Fig. 2c, top), the time delay parameters  $D_{m_1,j}$  and  $D_{m_2,j}$  are equal, and so  $\Delta t$  (equation 9) reduces simply to the time difference  $(t_2 - t_1)$ . Time delays are therefore only relevant when computing the cross-covariance between distinct populations  $m_1$  and  $m_2$ . The time delay to population  $m_1$ ,  $D_{m_1,j}$ , and the time delay to population  $m_2$ ,  $D_{m_2,j}$ , by themselves have no physically meaningful interpretation. Their difference  $D_{m_2,j} - D_{m_1,j}$ , however, represents a well-defined, continuous-valued time delay from population  $m_1$  to population  $m_2$  (Fig. 2c, bottom). The sign of the relative time delay indicates the directionality of the lead-lag relationship between populations captured by latent variable  $j$  (positive: population  $m_1$  leads population  $m_2$ ; negative: population  $m_2$  leads population  $m_1$ ), which we interpret as a description of signal flow. Note that time delays need not be integer multiples of the sampling period or spike count bin width of the neural activity.
Without loss of generality, we designate population  $m = 1$  as the reference area, and fix the delays for population 1 at 0 (i.e.,  $D_{1,j} = 0$  for all latent variables  $j = 1,\dots ,p$ ). We follow the same conventions as in [15, 21], and fix  $\sigma_j^2$  to a small value  $(10^{-3})$ . Furthermore, the GP is normalized so
that  $k_{m_1,m_2,j}(t_1,t_2) = 1$  if  $\Delta t = 0$ , thereby removing model redundancy in the scaling of the latent variables and loading matrices  $C^m$ .
Posterior inference and fitting the mDLAG model Let  $Y$  and  $X$  be collections of all observed neural activity and latent variables, respectively, across all time points and trials. Similarly, let  $\mathbf{d}$ ,  $\phi$ ,  $C$ ,  $\mathcal{A}$ , and  $D$  be collections of the mean parameters, noise precisions, loading matrices, ARD parameters, and time delays, respectively. From the neural activity, we seek to estimate posterior distributions over the probabilistic model components  $\theta = \{X, \mathbf{d}, \phi, C, \mathcal{A}\}$  and point estimates of the deterministic GP parameters  $\Omega = \{D, \{\tau_j\}_{j=1}^p\}$ .
We do so by employing a variational inference scheme [16, 22], in which we maximize the evidence lower bound (ELBO),  $L(Q,\Omega)$ , with respect to the approximate posterior distribution  $Q(\theta)$  and the deterministic parameters  $\Omega$ , where
$$
\log P (Y) \geq L (Q, \Omega) = \mathbb {E} _ {Q} [ \log P (Y, \theta | \Omega) ] - \mathbb {E} _ {Q} [ \log Q (\theta) ] \tag {10}
$$
We constrain  $Q(\theta)$  so that it factorizes over the elements of  $\theta$ :
$$
Q (\theta) = Q _ {x} (X) Q _ {d} (\mathbf {d}) Q _ {\phi} (\phi) Q _ {c} (C) Q _ {\mathcal {A}} (\mathcal {A}) \tag {11}
$$
This factorization enables closed-form updates during optimization. The ELBO can then be iteratively maximized via coordinate ascent of the factors of  $Q(\theta)$  and the deterministic parameters  $\Omega$ . Here all hyperparameters were fixed to a very small value,  $\beta, a_{\phi}, b_{\phi}, a_{\alpha}, b_{\alpha} = 10^{-12}$ , to produce noninformative priors [16]. Throughout this work, we take estimates of the latent variables and model parameters to be the corresponding means of the posterior distributions comprising equation 11. Full details are provided in Supplementary Section S2.
mDLAG special cases Finally, we consider some special cases of the mDLAG model that illustrate its relationship to other dimensionality reduction methods. In the case of two populations  $(M = 2)$ , mDLAG is equivalent to a Bayesian formulation of DLAG [15]. In the case of one population  $(M = 1)$ , and when all time delays are fixed to zero  $(D_{m,j} = 0)$ , mDLAG becomes equivalent to a Bayesian formulation of Gaussian process factor analysis [23]. By removing temporal smoothing (i.e., in the limit as all GP noise parameters  $\sigma_{j}$  approach 1) mDLAG becomes equivalent to GFA [16].
3 Validation in simulation and on neural recordings
Simulation 1: Uncovering directed interactions across multiple populations We first sought to demonstrate the ability of mDLAG to infer the structure of multi-population interactions. We therefore generated simulated neural activity from three populations  $(M = 3)$  according to the mDLAG generative model (equations 1-9). For the sake of illustration, we set 10 neurons in each population  $(q_{m} = 10)$ . Importantly, we designed the loading matrices  $C^m$  so that all types of interpopulation interactions were represented (Fig. 3a, left, ground truth): interactions shared globally, unique to each pair of populations, and local to one population. We also scaled the observation noise precision matrices  $\Phi^m$  so that noise levels were representative of realistic neural activity. Specifically, activity due to single-neuron observation noise was 10 times stronger than activity due to latents (the "signal-to-noise ratio"  $\mathrm{tr}(C^m C^{m^\top}) / \mathrm{tr}((\Phi^m)^{-1}) = 0.1$  for each population). We selected Gaussian process timescales and (relative) time delays that ranged between 20 ms to 150 ms and between 5 ms and 50 ms, respectively. With all model parameters specified, we then generated  $N = 100$  independent and identically distributed trials. Each trial was 500 ms in length, comprising  $T = 25$  time points with a sampling period of 20 ms, to mimic the 20 ms spike count time bins used to analyze the neural recordings in Section 4.
We then fit an mDLAG model to the simulated neural activity. We set the initial number of latents  $(p = 10)$  to be greater than the ground truth number  $(p = 7)$ , to verify that these additional latents would be pruned during fitting. In addition, to demonstrate the benefit of ARD in the mDLAG model, we also fit a modified mDLAG model that did not use ARD. Specifically, we fit (via an exact EM algorithm) a modified model with state model defined by equations 7-9 and observation model defined by equations 1 and 2, but with no prior distributions over the parameters  $C^m$ ,  $\mathbf{d}^m$ , and  $\Phi^m$  (i.e., we obtained point estimates). The number of latents for this modified model was selected via 4-fold cross-validation, considering model candidates with  $p = 1$  to  $p = 10$  latents. Cross-validation is not needed for the mDLAG model with ARD, a key computational advantage of the approach.
Figure 3: Automatic relevance determination encourages population-wise sparsity. (a) Loading matrix estimates. Left: Ground truth loading matrix. Center: mDLAG estimate. Right: mDLAG estimate where automatic relevance determination (ARD) was not used, i.e., no population-wise sparsity priors were placed on the loading matrix. Same conventions as in Fig. 2b. Note that the sign and ordering of each loading matrix column is, in general, arbitrary. We have therefore reordered and flipped the signs of the columns of the estimates to facilitate comparison with the ground truth. (b) Single-trial latent time course estimates. Each panel corresponds to the ground truth and estimated time course of a single latent variable. For concision, we show only latents corresponding to one population  $(\mathbf{x}_{n,j,:}^{m})$ ; the remaining latents are time-shifted versions of those shown here. Inset above each latent are the involved populations along with the signal flow and magnitude of time delays between populations (in ms). Magenta: mDLAG estimates; black: ground truth. a.u.: arbitrary units.
We were able to identify the total number of latent variables  $(p = 7)$  with both approaches. The mDLAG model with ARD additionally recovered the population-wise sparsity structure with high accuracy (Fig. 3a, center). The mDLAG model without ARD, however, produced an estimate of the loading matrix with mostly non-zero elements (Fig. 3a, right): had we not known the ground truth in advance, it would be difficult to interpret which population subsets are involved in which interactions. The mDLAG model with ARD also estimated the latent variable time courses and time delays (along with their implied signal flow) with high accuracy (Fig. 3b;  $R^2$  between ground truth and estimated time courses: 0.936; mean delay error: 1.14 ms).