Title: realSEUDO for real-time calcium imaging analysis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Calcium imaging

Real-time processing

Neuroimaging

Robust estimation

Sparse models
Abstract: Closed-loop neuroscience experimentation, where recorded neural activity is used to modify the experiment on-the-fly, is critical for deducing causal connections and optimizing experimental time. Thus while new optical methods permit on-line recording (via Multi-photon calcium imaging) and stimulation (via holographic stimulation) of large neural populations, a critical barrier in creating closed-loop experiments that can target and modulate single neurons is the real-time inference of neural activity from streaming recordings. In particular, while multi-photon calcium imaging (CI) is crucial in monitoring neural populations, extracting a single neuron's activity from the fluorescence videos often requires batch processing of the video data. Without batch processing, dimmer neurons and events are harder to identify and unrecognized neurons can create false positives when computing the activity of known neurons. We solve these issues by adapting a recently proposed robust time-trace estimator---Sparse Emulation of Unused Dictionary Objects (SEUDO) algorithm---as a basis for a new on-line processing algorithm that simultaneously identifies neurons in the fluorescence video and infers their time traces in a way that is robust to as-yet unidentified neurons. To achieve real-time SEUDO (realSEUDO), we introduce a combination of new algorithmic improvements, a fast C-based implementation, and a new cell finding loop to enable realSEUDO to identify new cells on-the-fly with no "warm-up" period. We demonstrate comparable performance to offline algorithms (e.g., CNMF), and improved performance over the current on-line approach (OnACID) at speeds of 120 Hz on average. This speed is faster than the typical 30 Hz framerate, leaving critical computation time for the computation of feedback in a closed-loop setting.

realSEUDO for real-time calcium imaging analysis

Abstract

Closed-loop neuroscience experimentation, where recorded neural activity is used to modify the experiment on-the-fly, is critical for deducing causal connections and optimizing experimental time. A critical step in creating a closed-loop experiment is real-time inference of neural activity from streaming recordings. One challenging modality for real-time processing is multi-photon calcium imaging (CI). CI enables the recording of activity in large populations of neurons however, often requires batch processing of the video data to extract single-neuron activity from the fluorescence videos. We use the recently proposed robust time-trace estimator-Sparse Emulation of Unused Dictionary Objects (SEUDO) algorithm-as a basis for a new on-line processing algorithm that simultaneously identifies neurons in the fluorescence video and infers their time traces in a way that is robust to as-yet unidentified neurons. To achieve real-time SEUDO (realSEUDO), we optimize the core estimator via both algorithmic improvements and an fast C-based implementation, and create a new cell finding loop to enable realSEUDO to also identify new cells. We demonstrate comparable performance to offline algorithms (e.g., CNMF), and improved performance over the current on-line approach (OnACID) at speeds of 120 Hz on average.

1 Introduction

Closed loop experiments enable neuroscientists to adapt presented stimuli or introduce perturbations (e.g., optogenetic stimulation) in real-time based on incoming observations of the neural activity. Such experiments are critical for both optimizing experimental time, e.g., by optimally selecting stimuli to fit neural response models [10], or by deducing causality by perturbing possible causeand-effect hypotheses. Despite this critical need, closed loop experiments at the level of populations of single neurons is incredibly difficult as they require real-time processing of neural data, which can be computationally intensive to process. In particular, population-level recordings using modern technologies often require significant computation to extract individual neuronal activity traces, e.g., spike sorting for high-density electrode electrophysiology or cell detection in fluorescence microscopy [6, 9].

Sergey Babkin

One particularly challenging recording technology is fluorescence microscopy, in particular multiphoton calcium imaging (CI). CI has progressed significantly since its inception with optical advances enabling access to larger fields of view, and therefore higher data throughput. While neuroscientists now have access to hundreds-to-thousands of neurons at a time, the neuronal time traces embedded in the video as fluorescing objects. To extract each neuron's activity, multiple methods have been developed, including matrix factorization approaches, deep learning approaches, and others (we refer to a recent review for a more complete coverage of available methods and their nuances [6]).

Almost all current calcium image processing methods uses batch processing: i.e., using a full video all at once to identify the neurons in the data and their time traces. For example, a common approach is to identify cells in a mean image (the image containing the average fluorescence per pixel over all time) and then to extract the time-trace from the video given the neuron's location, e.g., by averaging pixels. Real-time processing does not afford such luxury. Instead, frames must be processed as they are collected. Furthermore minimal data can be stored and used, as large image batches reduce algorithmic speed. Finally, the incomplete knowledge of the full set of cells in the video can cause unintended cross-talk. Unidentified cells may overlap with known cells, causing a well-documented effect of false transients when the unknown cells fluoresce [12].

We thus present an algorithm capable of demixing CI data frame-by-frame in real-time. Our design goals are to operate at > 30 Hz with minimal temporary data storage (e.g., no buffering or initialization period needed) while minimizing false transient activity. Our primary contributions are: 1) An optimized SEUDO algorithm for fast, robust time-trace computation, 2) A new feedback loop to identify cells in real-time, and 3) patch-based parallelization that enables high-throughput calcium trace estimation across larger fields of view.

2 Background

Traditionally, CI analysis has been performed on full imaging videos. The goal of these algorithms is to extract from a pixel-by-time data matrix Y ∈ R M × T , where M is the number of pixels in each frame and T is the number of frames, a set of neural profiles X ∈ R M × N (one for each of N neurons) and a corresponding set of time traces Φ ∈ R N × T . The former of these has, as each column of X , a single component profile depicting which pixels constitute that fluorescing object, and how strong that pixel is fluorescing. The latter has as each row the corresponding time traces that represent how bright that object was at each frame. These time-traces are particularly important for relating neural activity to each other (i.e., modeling population dynamics) or to stimuli and behavior.

In typical approaches, full videos are required to either 1) identify summary images (e.g., mean or max images [24, 11, 25, 19]) to identify cells in, 2) to create a dataset within which points are clustered into cells [17, 31, 22, 27, 1, 28, 3, 28, 18], or 3) to perform simultaneous cell identification and demixing [26, 23, 8, 15, 16, 20, 21, 29, 13] (e.g., via matrix factorization or dictionary learning). For example, in the latter of these classes of algorithms, the data decomposition is solved via a regularized optimization, e.g.,

where ∥ · ∥ 2 F is the Frobenius norm (sum of squares of all matrix elements), and R X ( X ) and R Φ ( Φ ) are regularization terms for the profiles and time-traces, respectively. While many regularization combinations exist, common terms include sparsity in the neural firing, minimal overlaps, nonnegativity, and spatial locality. Regardless, all methods require a large number of frames to identify the fluorescing components, with the exception of OnACID [14] and FIOLA [7].

OnACID and FIOLA operate in an on-line manner, utilizing the buffer of last l b residuals r t = y t -Xc t -Bf t where X and c represent the spatial and temporal profiles of already recognized cells and B and f represent the spatial and temporal profiles of the known background signal. Both methods use a local Constrained Non-negative Matrix Factorization (CNMF) [26] in the spatial and temporal vicinity of that point. CNMF is an off-line algorithm that repeatedly performs alternating optimizations on [ X , B ] and on [ c , f ] using the full dataset, until it converges to a designated precision. Both methods require initialization periods, and FIOLA further requires GPU and CPU optimizaiton, raising the computational infrastructure costs. We seek a solution that does not need any initialization data and can be run on simpler CPU machines for easier incorporation into user's workflows.

2.1 Sparse Emulation of Unknown Dictionary Objects

One primary challenge in fully on-line settings is the incomplete knowledge of all fluorescing components at the experiment onset. Even in off-line methods, incomplete identification of cells can create scientifically impactful cross talk-termed false transients -in inferred activity [12, 16]. Another challenge is identifying new components from few frames: ideally from individual frames to reduce memory usage. Recent work has provided an algorithm with the potential to solve both challenges: The Sparse Emulation of Unused Dictionary Objects (SEUDO) algorithm [12].

SEUDO is a robust time-trace estimator for neuronal time traces. Given a single fluorescence video frame y t , and a set of known profiles X , SEUDO models contamination from unknown profiles as Wc where W is a basis of small Gaussian bumps that linearly construct the interfering components, weighted by the sparse coefficients c (i.e., most c values are zero). SEUDO then solves the optimization

where λ and γ are model parameters and the internal min selects from the two internal expression that which has the minimal value. Since SEUDO operates per-frame, y t , Phi , c are all vectors. While SEUDO has demonstrated the ability to remove false transients [12], SEUDO's application has been limited to off-line post-processing due to: (1) slow computational speed, and (2) the need for pre-defined profiles X .

2.2 The FISTA Algorithm

The computational bottleneck in SEUDO is a weighted LASSO [32] optimization, which can be implemented with the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), which implements a momentum gradient descent [4]. FISTA optimizes

where f ( x ) is a smooth convex function with Lipschitz constant L > 0 , such that ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥∀ x, y ∈ R n , e.g. in SEUDO f ( x ) = ∥ y t -Xϕ t -Wc ∥ 2 2 , and g ( x ) is a continuous convex function that is typically non-smooth, e.g., g ( x ) = λ ∥ x ∥ 1 . Each descent step of FISTA consists of an ISTA descent step and a momentum step:

where the parameter η k gradually reduces with η k = t k -1 t k +1 , t k +1 = 1+ √ 1+4 t 2 k 2 , t 1 = 1 .

3 Real-time SEUDO

Here we develop Real-time SEUDO (realSEUDO) that resolves the primary limitations of SEUDO and extends the algorithm significantly from a time-trace estimator to a real-time cell identification method. Specifically we improve the computationally intensive momentum descent algorithm used to solve Equation (2) by reducing the number of steps of momentum descent, implementing parallelism, optimizing internal computations (e.g., of smoothness parameters), and reducing the complexity of the original fitting problem without a substantial loss of quality by manipulating its inputs. Moreover we add a new algorithm that automatically recognizes the neurons that have not previously been seen and adds them to the dictionary of known components. Finally, we implement our framework with a patch-based parallelism that avoids the computational scaling of LASSO in higher dimensions.

At a high level, the realSEUDO algorithm (Alg. 1, Fig. 1) operates as follows: realSEUDO is initialized with zero known components (an empty set). When fluorescence activity in a frame reaches threshold, an event is triggered that saves the activity profile of that event as a temporary candidate profile. Profiles are moved from the temporary profiles to the static set of profiles if they remain active for a sufficient number of frames . The static profile set is then used to identify the activity of those components in future frames, with unexplained components becoming candidate profiles and cycling back into the temporary profiles, followed by an update of the static profile set.

avg.

RealSEUDO Framework

Split

Add

Merge

Data

infer

adapt

Gaussian filter

(noise reduction)

run SEUDO on

current frame

motion corrected data

known component

activity

Short-term

Long-term

residuals

Update

component list

Data acquisition

Data modeling

Data modeling

SEUDO

model

A

typical

model

true

false

=

typical

model

single

profile

false

single SEUDO cell

fluor.

activity

Spatial profiles

Time traces

B

C

米

2

y

Atemp

Astab

3.1 SEUDO optimization

The first requirement of realSEUDO is a fast implementation of SEUDO that can operate at > 30 fps. We achieved this requirement through a combination of efficient implementations, algorithmic optimization, and updates to the base SEUDO model.

C++ implementation: The original SEUDO implementation used the TFOCS MATLAB library [5]. We thus first improved SEUDO's run-time by switching from MATLAB's interpreted programming language running TFOCS to a fast implementation of LASSO [32] via FISTA [4] in the C++ compiled language. To further improve performance, we optimized the C++ code with the use of templates to eliminate the function call overhead in tight loops, and also employ parallelism, based on the POSIX threads with TPOPP library wrapper [2]. To prevent a bottleneck from the passing of data through Matlab's OOP API at the MATLAB/C++ interface, we switched to the non-OOP version of the MATLAB-to-C API.

While beneficial, the C++ SEUDO implementation did not alone achieve the desired processing rate. We further improved runtimes by optimizing the cost function and derivative computations. The partial LASSO component of SEUDO that performs optimization at each frame y using FISTA can be written as arg min f ( ψ ) + λg ( ψ ) , where f = ∥ y -χψ ∥ 2 2 is the least-squares term and g = ∥ ψ ∥ 1 is the ℓ 1 penalty. In FISTA, the profile time traces and Gaussian kernels are unified in one vector, i.e., ψ is a concatenation of Φ and c , and χ = [ X , W ] . With M as the number of pixels per frame, N as the number of neurons, and K as the number of Gaussian kernels, the set of problem dimensions are y ∈ R M , χ ∈ R M × ( N + K ) , ψ ∈ R N + K , λ ∈ R N + K , with the first N elements of λ corresponding to Φ being equal to 0.

In FISTA, a number of internal computations become bottlenecks; in particular computing the gradients ∇ ψ f and ∇ ψ ( f + g ) , the Lipshitz smoothness estimation, and the momentum/stopping criteria.

Gradient computation: We reduce the burden of the gradient computations by both reducing the number of times the gradient must be used, and by improving the internal gradient computation. For the former, we note that naive implementations compute both a step in the direction of ∇ ψ f and then in the direction of ∇ ψ ( f + λg ) . Moreover, we note that these two steps in slightly different directions cause the gradient to dither around the optimum. We thus instead only take a step in the direction of ∇ ψ ( f + λg ) (similar to [4]). For the latter, computing ∇ ψ ( f + λg ) requires matrix vector multiplications with χ and its transpose. Since χ is sparse, we save memory and computation, by generating χ on-the-fly via convolutions instead of storing it in memory. Specifically, the gradient

∇ ψ f requires computing χ T χψ , which we reorganize to compute in two passes:

The first pass (Eqn. 6, left) computes a set of intermediary variables, and the second pass (Eqn. 6, right) uses these values to compute the gradient dimensions. The two pass approach factors out repeated computations, reducing the complexity from O ( n 3 ) to O ( n 2 ) . Moreover, the computation of each pass is highly parallelizable by partitioning of the first pass by j , the second by m , and efficiently skipping the iterations over the zero elements in the sparse matrix χ .

Lipshitz constant: To improve the efficiency of estimating the Lipschitz constant L , note that L = max( ∥∇ f ( x 1 ) -∇ f ( x 2 ) ∥ / ∥ x 1 -x 2 ∥ ) . For our cost function, we can approximate L with independent computations in each dimension. This estimation reduced the number of steps by as much as 30% over typical computations of L before each step based on the local gradient.

Momentum: The momentum descent central to solving the partial LASSO tends to spend many steps on stopping the momentum, especially with the large values of the Lipshitz constant L (i.e. the non-momentum steps are small). One such case is 'circling the drain' around the minimum, with the momentum causing the overshoot in one dimension while another dimension is stopping. Another case is when a dimension is moved past the boundary (e.g., x ≥ 0 for SEUDO), where the solution pushes past the boundary into negative values. This produces suboptimal solutions and increasing the number of steps necessary. FISTA includes a parameter η that progressively limits the top speed of descent to reduce such problems. We improved these cases by resetting the momentum to zero on a dimension when it either attempts to cross into the negative values or when its gradient changes sign. The dimensional momentum stopping stops abruptly at the right time, obviating the need for slowing and thus we can simplify FISTA by fixing η = 1 .

Our momentum stopping can further extend to broader optimization problems. We demonstrated this ability on momentum optimization in neural network training (see Supplement), where it provided a substantial improvement. Our modified FISTA produced the same error rate and squared mean error as gradient descent in about 10 times fewer training passes, or about 10 times lower error rate and 1.5 times lower mean square error in the same number of passes. The full modified FISTA algorithm is presented in Algorithm 2 (see Supplement).

SEUDO model adjustments: As a third step, we modified the SEUDO optimization program to achieve the final speedups. The original SEUDO spaced the Gaussian components in W by one pixel, which we found to be highly redundant. The kernels with radius r cover ( π ∗ r 2 ) pixels, and thus each pixel is covered by ( π ∗ r 2 ) kernels. This redundancy results in FISTA continuing to adjust the kernel coefficients c after the neural activations x converge. Reducing the number of kernels thus reduces both the number of gradient descent steps and the per-step cost, accelerating the computation more than quadratically. For a kernel with diameter of 30 pixels this improves the performance by a factor of over 100 without substantial degradation of false transients removal or the recognition of the interfering components' shape Wc .

Webenchmarked our speedups against the original SEUDO on 45000 frames across 50 cells from [12]. SEUDO ran at 5.8-6.9 s/cell on a Macintosh M1. The optimized C++ implementation without the MATLAB C++ API reduced the runtime to 0.9-1.1 s/cell (a 6-7x improvement). Sparse SEUDO provided further acceleration to a run time of 0.2 s, with 0.1 s for the computation and 0.1 s for the overhead of converting data between Matlab and native code; a total of a 29-34.5x speed-up, allowing SEUDO to run in real time.

3.2 Automatic cell recognition

We next developed the cell recognition feedback loop that completes the realSEUDO algorithm. To minimize data storage and compute, we designed realSEUDO to run on a frame-by-frame basis. At a high level, our automatic cell recognition first runs SEUDO on the current incoming (denoised) frame given the currently identified profiles X stab . The loop then identifies contiguous bright areas in the residual frames, i.e., the SEUDO cells Wc , and places them in a 'temporary profile' array X temp . The temporary profiles are then updated (via merging with new potential profiles) given new, incoming, frames until they are stable and moved to the stable, known profile list X stab that is updated less frequently by addition, merging and splitting of temporary profiles.

Procedurally, we first preprocess each incoming frame to reduce noise and improve profile recognition. Calcium imaging analysis often uses running averages in space and time for noise reduction. We thus implement both a spatial Gaussian filter, as well as a running average of several sequential frames. We keep the window length as a tunable parameter that can be set to one for frame-by-frame processing with minimize temporal blurring.

We estimate the activation level in the denoised frame for each of the stable profiles X stab using the our fast SEUDO implementation. SEUDO returns the activation level ϕ kt for the k th profile at time t along with a robust residual that contains the structured fluorescence not captured by X stab . We then run the residual through SEUDO a second time using the temporary profiles X temp to test if any temporary profile matches the frame's fluorescence and should be moved from X temp to X stab . The residual after the second SEUDO application represents completely unknown profiles and are analyzed separately to determine if a new member of X temp should be created.

The detection of new temporary profiles is based on finding the areas of the image above the noise level. The noise level is evaluated by noting that most of each video frame has no activity, indicating that the median pixel value will be very close to the median value of the pixels in an all-dark frame containing the same noise. The half-amplitude of the noise σ 1 / 2 can be estimated as:

In some cases different areas of the image may contain a different amount of background lighting (e.g., changes in neuropil), which can skew the noise estimate. To overcome this challenge, we split the larger image into sections, with each section computing a local median which is smoothly interpolated between section of the image. This can be thought as either a krigging procedure or a cheap approximation to a local median evaluated independently for each pixel in the image.

To merge new profiles into the exiting profiles when adding new profiles to X temp , we compute an overlap score. The scores aims to capture the following logic: If two temporal profiles are a close match in cross-section, they likely represent the same cell and should be merged. If they overlap only partially, they likely represent separate cells. If one cross-section is inside the other, look at relative brightness: if the smaller cross-section is also weaker, it's likely a weaker partial activation of the same cell and should be merged, if the smaller cross-section has a close or higher brightness, the larger cross-section likely represents an intertwining of two cells that has to be split.

The overlap computation thus is not a plain spatial overlap but includes a heuristics that identifies when one profile is mostly contained in another. The condition for merging two profiles in X temp is based on the comparison of numbers of common and unique pixels between profiles, where P 1 and P 2 are numbers of pixels in each of two profiles, U 1 and U 2 are the numbers of unique pixels in each profile, C is the number of common pixels, B 1 and B 2 are the perimeters of bounding boxes for each profile, and k temp is a constant with an empirically chosen value of 0.75:

After a profile is moved from X temp to X stab , a different score is computed pair-wise between the new profile and each existing profile, to decide whether they should be left separate, or merged, or one of profiles split. If a merge or split is performed, the original profiles are removed from X stab and the results entered recursively into X stab as new profiles. The score for two profiles A and B in X stab is computed based on the brightness and measures of least-squares fit of the cells into each other 1) as whole cells (i.e., α AB = ⟨ A,B ⟩ / ⟨ A,A ⟩ ) and 2) using only the overlapping region (i.e., β AB = ⟨ A ol , B ⟩ / ⟨ A ol , A ol ⟩ where A ol is the profile A restricted to the region overlapping with B ). β AB is used as a measure of difference in brightness, against which the fit of the whole cells α AB is compared as a measure of proximity in shape. Specifically we compute two ratios ρ AB and ρ BA are computed as

A higher value of ρ AB (which is always ≤ 1) means that cell A fits better inside cell B. The value of 1 means that it fits entirely inside cell B. The same principle applies symmetrically to ρ BA .

3.3 Patching and profile matching

realSEUDO, although highly efficient for smaller patches, is still based on the LASSO algorithm that reduces in efficiency with much larger frames. Thus we adopt a patching scheme that breaks each

frame into small patches that can be parallelized to maintain the high framerate by utilizing the multithreading in many modern processors. Patching, however, requires matching profiles across patches. Traditionally profiles discovered in data split into patches is to add overlap margins to the patches and to use profiles overlap in this region to determine matchings in neighboring patches. Additional margins, however, introduces redundant computation and decreases computational efficiency.

In realSEUDO we note that the logic behind the scoring we use to merge profiles in X stab and X temp within each patch can also be used to score the match of profiles across patches. Specifically, we extend the matching to include the profile temporal activity as an additional dimension to find matching cells in neighboring patches via consecutive gluing of the profiles. The highest score is assigned to the bidirectional match of both spatial and temporal dimensions, a lower score to symmetrical match of spatial dimensions and asymmetrical match of temporal dimension, a yet lower score to an asymmetrical match of both kinds of dimensions. We have observed successful matches even with zero margin, using the neighboring strips of pixels around the perimeters of the patches as the spatial dimension for matching.

4 Results

Validation metrics: To validate our approach we note that the main goal of realSEUDO, as with most functional imaging analyses, primarily aims to recover the time-traces of neural activity as accurately as possible [6, 12]. This means while the general location of neurons is important, metrics such as the the Intersection over Union (IoU) are too strong; i.e., the full set of pixels identified is not necessarily the important quantity. We instead compute the 'unique neurons found'. This metric aims to capture the need to know that the time-traces 1) correspond to real neurons in the data and 2) accurately reflect the temporal activity that will be used to study neural activity with respect to stimuli and behavior. The Unique Neurons Found (as defined in [30]) requires both that ROIs well align with known ROIs spatially (overlap of at least 50% of pixels) and that the time-trace correlation exceeds 0.5.

Simulated data experiments: We first applied all three algorithms to a simulated video created with Neural Anatomy and Optical Microscopy simulation (NAOMi) [30]. Specifically we simulated the neural activity over 20000 frames at 30Hz with fame size of 500x500 pixels. There are approximately 450 cells visible in this dataset (i.e. fluorescing cells intersecting the plane of imaging). We benchmarked the patch-based parallel processing of realSEUDO 80x80 pixel patches. For comparison we ran the off-line CNMF (a staple batch-based calcium imaging demixing algorithm) and OnACID, the computationally similar on-line method. realSEUDO found 201 true cells, identified as strongly correlated with ground-truth cells, while OnACID found 152 and CNMF (the offline method) found 308 (Fig. 2A-B). Furthermore, both realSEUDO and OnACID found many fewer false positives than CNMF, presumably because they cannot be fooled by small fluctuations integrated over the full recording (Fig. 2C-D). Note that to remove the confound of post-processing we followed prior work [30] in using the CNMF raw fluorescence traces instead of the model-based denoised traces. On average, realSEUDO processed 67.8 frames per second end-to-end, while OnACID ran at 9.2 fps.

Applications to in-vivo mouse CA1 recordings We applied realSEUDO to an in vivo calcium imaging recordings from mouse hippocampal area CA1 previously described in Gauthier et al. 2022 [12]. They consisted of 36 videos, each sized 90x90 pixels with 41750 frames sampled at 30 Hz. The outputs had previously been verified manually by Gauthier et al. 2022 [12] with human labeling of CNMF outputs. We applied realSEUDO to all videos and compared the outputs with the current online cell demixing algorithm OnACID [14], as well as a popular offline algorithm, CNMF [13], as an additional baseline.

We benchmarked realSEUDO against OnACID (an online analysis tool) and CNMF (an offline analysis tool) on real in-vivo calcium imaging movies (Fig. 3). Algorithmic performance was measured on an x86-64 computer with 48 CPU cores (Intel Xeon 6248R), 2 hyperthreads per core, 78 GB of memory, and without the use of a GPU. The initialization times were not included. On average, realSEUDO processed 162 frames per second compared to 26 processed by OnACID and 13 by CNMF: an improvement of 6.5x and 12.5x respectively (Fig. 3C). Quality-wise, we found that OnACID exhibited difficulties with adapting to larger ranges of pixel brightness, sometimes missing bright cells. Scaling pixel values improved OnACID results, but only mildly (one additional cell). Numerically OnACID and CNMF appear similar but they identified different components. SEUDO

0

0.2

CNMF

OnACID

Strong pairs

realSEUDO

Weak pairs

Un-paired

# profiles

Correlation with ground truth

# profiles

Fluorescence (AU)

6000

4000

2000

0

Profile 1

Frame number (simulated at 30 Hz)

Profile 2

Profile 3

CNMF (offline)

OnACID (9.4+/- 0.3 Hz)

realSEUDO (67.8+/- 24.1 Hz)

CNMF

OnACID

realSEUDO

Ground truth

A

C

B

D

100   m

100   m

100   m

100   m

100   m

100   m

100   m

100   m

100   m

Temporal correlations to ground truth cells

Example matched profiles to ground truth traces

Profiles found in the NAOMi data

100

NAOMi profiles

0

100

200

300

Hits

False

alarms

Ambig-

uous

Hits and false alarms

CNMF

OnACID

realSEUDO

100   m

20

40

60

80

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.1

C

results were most similar to CNMF, and with additional cells identified, and less false positives (Fig. 3B). Finally, the per-transient manual classification provided by [12] enabled us to assess if realSEUDO inherited the false transient removal properties of SEUDO. For a reasonable value of λ = 0 . 15 , realSEUDO had a true positive rate of 75% and a false positive rate of 24%. While these numbers are a bit lower than the numbers reported in [12], in that study the authors average N=3 frames to reduce noise, while maintained single-frame analysis. This can be evident by the fact that missed transients were very small: realSEUDO kept 98% of real fluorescence and only 15% of false fluorescence.

Additional in-vivo tests: As final test we applied all three algorithms (realSEUDO, OnACID, CNMF) to a 2000-frame mesoscope video example collected by the Yuste lab at Columbia University and provided with the OnACID github package as a demo. For this example we similarly saw improved cell detection and runtime improvement in terms of fps (Fig. 3B).

5 Discussion

We present here an online method for cell detection and fluorescence time-trace estimation from streaming CI data: realSEUDO. realSEUDO is based on the SEUDO robust time-trace estimator that reduces bias due to unknown cells while also providing approximate shapes of the unknown fluoresc-

Estimated Spatial Profiles

OnAcid demo dataset

real-SEUDO

OnACID

CNMF

Δ

F/F (AU)

Estimated Activity

real-SEUDO

OnACID

# components

7

9

4

2

4

2

21

19

24

True

False Positives

Detected ROIs

20

False Negatives

7

False Negatives

False Positives

10

True

5

8

1

0

0

7

# components

14

Detected ROIs

CPU seconds/frame

Frames/second

40

50

60

100

80

20

40

60

80

0

0.06

150

200

250

300

0.08

0.1

0.12

0.14

0.16

0.18

0.2

A

C

B

real-SEUDO

OnACID

CNMF

=

ing objects. To build realSEUDO we 1) improved SEUDO's runtime via significant modifications at the code, algorithmic, and model levels 2) built a new feedback loop that allowed SEUDO (that has no cell finding component currently) to identify cells in real-time. Overall, realSEUDO can achieve frame processing rates of 80-200 fps, depending on cell density. While our goal was to exceed the typical 30 Hz data collection rate common to many experiments, the high processing efficiency leaves additional time to compute feedback in future closed loop systems. Moreover, realSEUDO can scale with faster recording rates as calcium indicators become faster, e.g., GCaMP8 [33].

realSEUDO's implementation exhibits a higher degree of parallelism than OnACID, however both are likely constrained by the employed tools. Specifically, the measurements in Fig. 3C show very little fps fluctuation with respect to cell count for OnACID, which can be due to a bottleneck in a single thread. realSEUDO has a lower latency to the first events for a new cell, OnACID requires a history of 100 frames to recognize a cell, while realSEUDO would produce the first events starting with the first frame that passes the low brightness threshold. Alternatively, OnACID has a higher native scalability with respect to the frame area, conditioned on similar cell counts. This likely result from OnACID's algorithm restriction of processing to areas in the immediate vicinity of known cells. realSEUDO can still achieve a high processing efficiency with reasonable hardware with our parallelization. Future work may further improve realSEUDO runtime by blanking out entire patches until activity is detected via simpler detectors.

One strength of realSEUDO is that it can be initialized with either an empty profile set. This ability to start from nothing will be useful in mesoscope settings when the field-of-view can be changed on-the-fly. Not requiring an initialization step will reduce start-up overhead at new fields of view. Furthermore, many of our speed adaptions deviate from the traditional gradient descent approach. In particular the Lipshitz constant approximation and the changes in the momentum and stopping criteria. In other domains, in particular for training deep neural networks, these deviations may also provide significant speedups, the extent of which should be quantified across broader applications in future work.

Limitations: While our results achieve the design criteria we initially set out, there are some potential barriers. For one, as with many real-time systems, the compute environment is very important to configure correctly. We have found the importance of explicitly setting the Linux CPU manager to enable the performance mode. The default automatic adaptable mode does not react properly to CPU loads of less than 100% of the whole capacity, and significantly skews the benchmarking by running

the CPUs at low frequency. These challenges are unfortunately necessary to achieve high levels of throughput without specialized hardware. Future work should develop walkthroughs and automated tools to guide the installation of the tools.

While our core algorithm is written completely in C++, and thus open source, we have found MATLAB convenient and efficient as a wrapper for prototyping wrappers for our core functions. Further work will add Python wrappers to allow for seamless integration into both MATLAB and Python pipelines, enabling realSEUDO to be more widely used.

Finally, we focused here only on cell detection, assuming access to the on-line motion correction algorithm from OnAcid. This focus may require additional packages to be handled by users for motion correction. We will further aim in future iterations to extend the core package to include motion correction and delta-F over F computations in order to reduce communication overhead and ease adoption by users. Moreover, to fully optimize the package for speed, these steps should be more holistically incorporated in C++ as well.

References

Noah Apthorpe, Alexander Riordan, Robert Aguilar, Jan Homann, Yi Gu, David Tank, and H Sebastian Seung. Automatic neuron detection in calcium imaging data using convolutional networks. Advances in neural information processing systems , 29:3270-3278, 2016.

Sergey A Babkin. The practice of parallel programming . Createspace, 2010.

Yijun Bao, Somayyeh Soltanian-Zadeh, Sina Farsiu, and Yiyang Gong. Segmentation of neurons from fluorescence calcium recordings beyond real time. Nature Machine Intelligence , pages 1-11, 2021.

Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences , 2(1):183-202, 2009.

Stephen R Becker, Emmanuel J Candès, and Michael C Grant. Templates for convex cone problems with applications to sparse signal recovery. Mathematical programming computation , 3:165-218, 2011.

Hadas Benisty, Alexander Song, Gal Mishne, and Adam S Charles. Review of data processing of functional optical microscopy for neuroscience. Neurophotonics , 9(4):041402, 2022.

Changjia Cai, Cynthia Dong, Johannes Friedrich, Marton Rozsa, Eftychios A Pnevmatikakis, and Andrea Giovannucci. Fiola: an accelerated pipeline for fluorescence imaging online analysis. Nature Methods , 20(9):1417-1425, 2023.

Adam S. Charles, Nathan Cermak, Rifqi O. Affan, Benjamin B. Scott, Jackie Schiller, and Gal Mishne. Graft: Graph filtered temporal dictionary learning for functional neural imaging. IEEE Transactions on Image Processing , 31:3509-3524, 2022.

Adam S Charles, Benjamin Falk, Nicholas Turner, Talmo D Pereira, Daniel Tward, Benjamin D Pedigo, Jaewon Chung, Randal Burns, Satrajit S Ghosh, Justus M Kebschull, et al. Toward community-driven big open brain science: open big data and tools for structure, function, and genetics. Annual review of neuroscience , 43(1):441-464, 2020.

Adam S Charles, Mijung Park, J Patrick Weller, Gregory D Horwitz, and Jonathan W Pillow. Dethroning the fano factor: a flexible, model-based approach to partitioning neural variability. Neural computation , 30(4):1012-1045, 2018.

F. Diego and F. A. Hamprecht. Sparse space-time deconvolution for calcium image analysis. In NIPS , pages 64-72, 2014.

Jeffrey L Gauthier, Sue Ann Koay, Edward H Nieh, David W Tank, Jonathan W Pillow, and Adam S Charles. Detecting and correcting false transients in calcium imaging. Nature Methods , 19(4):470-478, 2022.

Andrea Giovannucci, Johannes Friedrich, Pat Gunn, Jérémie Kalfon, Brandon L Brown, Sue Ann Koay, Jiannis Taxidis, Farzaneh Najafi, Jeffrey L Gauthier, Pengcheng Zhou, Baljit S Khakh, David W Tank, Dmitri B Chklovskii1, and Eftychios A Pnevmatikakis. CaImAn an open source tool for scalable calcium imaging data analysis. Elife , 8:e38173, 2019.

Andrea Giovannucci, Johannes Friedrich, Matthew Kaufman, Anne K Churchland, Dmitri Chklovskii, Liam Paninski, and Eftychios A Pnevmatikakis. Onacid: online analysis of calcium imaging data in real time. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 2378-2388, 2017.

Benjamin D Haeffele and René Vidal. Structured low-rank matrix factorization: Global optimality, algorithms, and applications. IEEE transactions on pattern analysis and machine intelligence , 42(6):1468-1482, 2019.

Hakan Inan, Murat A Erdogdu, and Mark Schnitzer. Robust estimation of neural signals in calcium imaging. In NIPS , pages 2905-2914, 2017.

Patrick Kaifosh, Jeffrey D. Zaremba, Nathan B. Danielson, and Attila Losonczy. Sima: Python software for analysis of dynamic fluorescence imaging data. Frontiers in Neuroinformatics , 8, 2014.

Elke Kirschbaum, Alberto Bailoni, and Fred A Hamprecht. Disco: Deep learning, instance segmentation, and correlations for cell segmentation in calcium imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 151-162. Springer, 2020.

Aleksander Klibisz, Derek Rose, Matthew Eicholtz, Jay Blundon, and Stanislav Zakharenko. Fast, simple calcium imaging segmentation with fully convolutional networks. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support , pages 285-293. Springer, 2017.

Ryuichi Maruyama, Kazuma Maeda, Hajime Moroda, Ichiro Kato, Masashi Inoue, Hiroyoshi Miyakawa, and Toru Aonishi. Detecting cells using non-negative matrix factorization on calcium imaging data. Neural Networks , 55:11-19, 2014.

Gal Mishne and Adam S Charles. Learning spatially-correlated temporal dictionaries for calcium imaging. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-2019) , pages 1065-1069. IEEE, 2019.

Gal Mishne, Ronald R Coifman, Maria Lavzin, and Jackie Schiller. Automated cellular structure extraction in biological images with applications to calcium imaging data. bioRxiv , page 313981, 2018.

M. Pachitariu, A. M. Packer, N. Pettit, H. Dalgleish, M. Hausser, and M. Sahani. Suite2p: beyond 10,000 neurons with standard two-photon microscopy. bioRxiv , 2016.

Marius Pachitariu, Adam M Packer, Noah Pettit, Henry Dalgleish, Michael Hausser, and Maneesh Sahani. Extracting regions of interest from biological images with convolutional sparse block coding. Advances in Neural Information Processing Systems , pages 1745-1753, 2013.

Ashley Petersen, Noah Simon, and Daniela Witten. Scalpel: Extracting neurons from calcium imaging data. The annals of applied statistics , 12(4):2430, 2018.

E. A Pnevmatikakis, D. Soudry, Y. Gao, T. A. Machado, J. Merel, D. Pfau, T. Reardon, Y. Mu, C. Lacefield, W. Yang, Misha Ahrens, Randy Bruno, Thomas M. Jessell, DarcyÂ S. Peterka, Rafael Yuste, and Liam Paninski. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron , 89(2):285-299, 2016.

S. Reynolds, T. Abrahamsson, R. Schuck, P. Jesper Sjöström, S. R. Schultz, and P. L. Dragotti. ABLE: An activity-based level set segmentation algorithm for two-photon calcium imaging data. eNeuro , 2017.

Somayyeh Soltanian-Zadeh, Kaan Sahingur, Sarah Blau, Yiyang Gong, and Sina Farsiu. Fast and robust active neuron segmentation in two-photon calcium imaging using spatiotemporal deep learning. Proceedings of the National Academy of Sciences , 116(17):8554-8563, 2019.

Alexander Song, Adam S Charles, Sue Ann Koay, Jeff L Gauthier, Stephan Y Thiberge, Jonathan W Pillow, and David W Tank. Volumetric two-photon imaging of neurons using stereoscopy (vTwINS). Nature methods , 14(4):420, 2017.

Alexander Song, Jeff L Gauthier, Jonathan W Pillow, David W Tank, and Adam S Charles. Neural anatomy and optical microscopy (naomi) simulation for evaluating calcium imaging methods. Journal of neuroscience methods , 358:109173, 2021.

Quico Spaen, Roberto Asín-Achá, Selmaan N Chettih, Matthias Minderer, Christopher Harvey, and Dorit S Hochbaum. HNCcorr: a novel combinatorial approach for cell identification in calcium-imaging movies. Eneuro , 6(2), 2019.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996.

Yan Zhang, Márton Rózsa, Yajie Liang, Daniel Bushey, Ziqiang Wei, Jihong Zheng, Daniel Reep, Gerard Joey Broussard, Arthur Tsang, Getahun Tsegaye, et al. Fast and sensitive gcamp calcium indicators for imaging neural populations. Nature , pages 1-8, 2023.

A Appendix

Algorithm 1 realSEUDO Algorithm

A.1 Application of modified FISTA to neural network optimization

We further tested the modified FISTA momentum descent algorithm to problems outside of neuroscience-the training of neural networks-to evaluate the scope of applicability of our improvements. We used the problem of recognition of handwritten digits on a data set from AT&T Research available at https://hastie.su.domains/StatLearnSparsity_files/ DATA/zipcode.html , reduced to 8x8 pixels, with a training set of 7291 images. The neural network (NN) model used the Leaky ReLU activation, with layer sizes 64, 64, 32, 10.

The common approach to training NNs uses stochastic gradient descent, including stochastic momentum methods. Thus to compare to a non-stochastic momentum method we established a non-stochastic baseline.

The dynamic estimation of the Lipshitz constant L from TFOCS cannot be applied to the neural network optimization because the optimization cost is highly non-linear. The multi-dimensional estimation of L is also computed only for the specific SEUDO function. The common practice is to use a fixed descent rate, which serves as an analog of 1 2 L . Estimating the highest descent rate is still not a fully solved problem. Algorithms for dynamic evaluation of the descent rate do exist (e.g., the ADAM algorithm in Kingma & Ba, 2015), however they rely on a constant to be picked for a particular problem.

The advantage of stochastic methods is that they can use a higher descent rate without diverging, as seen by observing the dependency of logarithm of mean square error from the number of training passes for various descent rates (Fig 6). In these results we ensure that we start from the same fixed randomized initial state since different initial states can produce wildly different results.

We observe that for the same descent rate (0.05 per pass), the stochastic and non-stochastic methods produce very similar error values, however the graph for the stochastic method is more smooth. The roughness of the graph represents the small divergences that manage to converge again over time, and shows that the descent rate is close to the maximum. However the stochastic method can accommodate a 100 times higher descent rate without diverging, and even a 1000 higher descent

Algorithm 2 Modified FISTA algorithm

̸

rate becomes rough but still converges. The non-stochastic method is able to make the passes faster, because it performs the same accumulation of partial gradients, but saves the overhead of updating the weights after each training case (or batch). However even adjusted for time, the stochastic method performs faster. It is possible to compute the non-stochastic gradient in parallel by multiple threads but we have not implemented this. We used this example of the non-stochastic descent as a baseline for the FISTA-based momentum methods.

The summary of training errors in the momentum methods can be found in Figure 7A, and the mean square errors after 10,000 training passes are listed in the Table 1.

The unmodified FISTA algoithm with λ = 0 performed on this task out of its domain worse than the non-momentum baseline. Adding the momentum stop in the dimenstions with gradient sign change produced a substantial improvement over the baseline. Fixing the parameter η = 1 produced a close result to not fixing η , but with a less rough curve. We tested if the smoothness indicated that setting

η = 1 could accommodate a substantial increase in descent rate by re-running the algorithm at a 4x rate (0.2 instead of 0.05), and while this run did not diverge, we did observe a higher error rate.

Finally, we attempted to devise an algorithm that acts similar to the TFOCS dynamic evaluation of L but using the ratios of mean square values of gradient dimensions that change or not change sign as an indication of roughness. The rapid growth of gradient dimensions after sign change is seen as a beginning of a divergence, that causes the reduction of descent rate. This algorithm allowed training at a substantially higher rate in the first few thousands of passes but then flattened out. The automatically determined rate is close to the empirically found 0.05, and is higher in the initial passes where it reaches higher values, but then drops to the lower values (Fig. 7B). It is possible that the chosen criteria were not aggressive enough, and can be improved.

While event with the momentum descent the stochastic methods specialized for NN training can still achieve faster speeds (Fig. 7C-D), we have demonstrated that our more general optimization still represent a major improvement over both simple gradient descent and plain FISTA in a different domain.

Cell ID

realSEUDO

ON-ACID

CNMF

6

忆

3

3

15

5

9

9

5

乙

3

4

3

乙

realSEUDO output:

0

1

2

3

4

Time (frames at 30 Hz)

Cell ID

x10 4

3

5

6

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

0

200

400

600

800

1000

1200

1400

Seconds

-4.5

-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

Log(mean square error)

Training error by time: stochastic vs non-stochastic method

stochastic

non-stochastic

stochastic 10x rate

stochastic 100x rate

stochastic 1000x rate

0

2000

4000

6000

8000

10000

Training passes

-4.5

-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

Log(mean square error)

Training error: stochastic vs non-stochastic method

stochastic

non-stochastic

stochastic 10x rate

stochastic 100x rate

stochastic 1000x rate

A

B

C

D

Fluorescence (AU)

50

100

150

200

Profile number

realSEUDO

100   m

100   m

20000

Frame number (simulated at 30 Hz)

15000

10000

5000

0

100   m

Fluorescence (AU)

Fluorescence (AU)

20000

Frame number (simulated at 30 Hz)

15000

10000

5000

0

20000

Frame number (simulated at 30 Hz)

15000

10000

5000

0

50

100

150

Profile number

Profile number

100

300

200

OnACID

CNMF

C

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] ,

Justification: This work claims to provide a new algorithm for real-time processing of calcium imaging data. The results show processing speeds of 80-200 Hz, far exceeding the >30 Hz minimum for real-time processing.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a discussion in the discussion section that discusses explicitly our limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer:[NA]

Justification: Our paper is based on developing a new algorithm, and we do not include theoretical claims.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The data from this paper is open and available and full pseudo-code and all parameter selections are provided in the main text or supplementary.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: While the data is all freely available, the implementation still requires some fine-tuning to be easy to use for the general public. We plan to release before the conference itself, and, provide full algorithmic and parameter selection details in the paper.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our method provides all parameter selections. Note that these parameters were not fit to data in the optimization sense but rather selected based on rules of thumb for the classes of algorithms used (sparse inference).

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Where applicable (e.g., timing assessment etc.) error bars are provided.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: A number of compute infrastructures were used to compute the results, and no special hardware was needed (e.g., GPUs).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper does not use human subjects, private or protected data, or provide any capability in that might compromise safety or security. This work is squarely in the scientific microscopy analysis domain for non-human experiments.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work describes the broader scientific impacts, but the advances are squarely in the imaging and optimization domain.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This algorithm does not need safeguards as it is made for microscopy for non-human applications.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Everyone who contributed is properly referenced.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are presented.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing was used.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No IRB was needed.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors extend an available algorithm called SEUDO for extracting time traces from two-photon imaging data robustly to the real-time close-loop setting. The algorithm can simultaneously extract ROIs and time traces. The algorithm performs on par with state-of-the-art offline algorithms and better than state-of-the-art online algorithms.\n\nSTRENGTHS:\n* The paper is well written and the motivation is clear and the summary of the literature concise and compelling.\n* While the algorithmic components of the work are not new in themselves, they are combined in a convincing way to solve the problem laid out.\n* The speed of the algorithm is impressive, at satisfying performance compared to the SOTA.\n\nWEAKNESSES:\n* The main improvement come from engineering the implementation of SEUDO such as to run in close to real time (is that a weakness or strength?).\n* Lines 228: The merging procedure seems quite ad-hoc and heuristic. The logic is only laid out at the end (lines 247), maybe it would help to discuss this earlier.\n* Only two algorithms are chosen for comparison. \n* Fig. 2B is very hard to read and understand. Does this mean, OnACID produces no false alarms and no ambiguous units?\n\nQUESTIONS:\n* A lot of engineering work has gone into the fast implementation of the online algorithm using C++ and various bells and whistles. Have the authors considered a GPU based implementation e.g. in jax and how would that compare in performance?\n* Where does equation 7 come from? Min operations are quite sensitive to outliers. The details won’t matter, but in the spike sorting literature, a threshold of n*0.6745(median (abs(x))) is often used for detecting spikes, which is indeed a multiple of a robust estimate of the noise sd without the spikes.\n* Why was the tool not compared to the other online algorithm FIOLA?\n* Why was the plain CNMF algorithm used and not one of the more advanced implementations/methods or an additional algorithm?\n* Doesn’t Fig. 2C speak very clearly for the results of the OnACID algorithm?\n* Lines 348: Why do the authors say that a c++ implementation makes the code open source?\n\nLIMITATIONS:\nAdequately adressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this study, the authors created a real-time version of the “Sparse Emulation of Unused Dictionary Objects” (SEUDO) algorithm termed realSEUDO, that can process multi-photon calcium imaging (CI) data in real time. The goal of this approach is to overcome the limitations of batch processing methods commonly employed in CI analysis by processing brain activity data in real-time, thus enabling closed-loop neuroscience investigations. The authors identified several places that the original SEUDO algorithm warranted improvements, implemented the changes and compared their method to other current online methods, with both simulated and real datasets.\n\nSTRENGTHS:\nThis research presents a novel real-time CI data analysis method that is important for closed-loop investigations.  The realSEUDO algorithm as presented here, incorporates a significant number of optimizations over the original SEUDO, including a faster C++ implementation, parallel processing, and an efficient feedback loop for real-time cell identification that improves the computational load. In this way they can achieve much faster performance compared to other online methods (OnACID). Another important strength is its ability to match sources across non-overlapping parches, which reduces the computational load. The authors use both simulated and real datasets to evaluate the performance of their algorithm. This method could also become very useful in online-analysis of voltage traces that higher speeds can have even more benefits.\n\nWEAKNESSES:\nWhile the performance with a small number of neurons is impressive there are two drawbacks. \nFirst, the testing datasets only include a very limited number of neurons and the scalability with much larger datasets are not presented. For example, current methods such as with a mesoscope  (Sofroniew et al. 2016), allow for much larger recording samples in the of thousands of neurons. From the figure 3c we can see that the algorithm drops in performance quite fast and when much larger samples are included, it could potentially perform worse than existing methods, something that remains to be tested. \nLastly, calcium sensors already limit our ability to extract with fine temporal precision the neural activity thus the usefulness of a method that far exceeds that temporal resolution remains limited. \nAnother potential issue is to validate the model with more typical cortical recordings that have higher level of background fluorescence, and how it deals with overlapping sources of fluorescent signal.\n\nQUESTIONS:\nBy facilitating the real-time processing of CI data, the paper shows the potential to improve the accuracy and efficiency of closed-loop studies. The claims of broad applicability would be strengthened by more extensive testing under various experimental situations and with diverse kinds of brain data. Additionally, it would be great if they could characterize and show the performance with different patch overlaps. Finally, while the paper discusses the integration with motion correction algorithms from OnACID, integration of all preprocessing steps such as motion correction within the realSEUDO pipeline would further widen its potential usability.\n\nLIMITATIONS:\nThe authors have a limitation section that addresses potential issues with their algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an online method for neuron profile and trace estimation in multi-photon calcium imaging. Their approach enhances the SEUDO algorithm by running it on a frame-by-frame basis and periodically updating profiles using a patching mechanism across space and time. The implementation leverages different degrees of stability, namely temporary and stable. The proposed method is benchmarked against both an online method and an offline method using simulated and real data.\n\nSTRENGTHS:\n- The manuscript provides extensive details about the implementation.\n- The impact of the proposed solution is clearly stated and is highly relevant for neuroscientists working with CI acquisitions, addressing several real issues such as empty initialization of profiles and scalability.\n\nWEAKNESSES:\n- The work largely relies on previously described algorithms like FISTA and SEUDO. Since a significant portion of the method describes these algorithms in detail, it is unclear what the key contributions of this work are.\n- The evaluation was conducted on synthetic data (Naomi) and a real dataset (36 videos). Including additional sources like 2p movies from the Allen Brain Observatory would have enhanced the credibility of the results.\n- The given metrics should be improved. While the images are good, providing tables with quantitative metrics can enhance the overall presentation and support the claims made. In addition to correlation, metrics like IoU could be evaluated on neuron profiles.\n- The experimental setting lacks an ablation study on the main components of the work, such as automatic cell recognition (e.g. with and w/o X_temp) and patching.\n\nQUESTIONS:\n1. What are the key methodological differences between SEUDO and realSEUDO?\n2. How did you choose the benchmark datasets?\n3. Why correlation was selected as the main (and only) metric?\n4. How do the authors justify the lack of an ablation study on the main components of the work? Can the authors describe the quantitative improvement of their method compared to existing methods they rely on?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel online method for cell detection and fluorescence estimation from streaming calcium imaging (CI) data, called realSEUDO. Building upon the SEUDO algorithm, realSEUDO introduces significant advancements in real-time cell identification and fluorescence estimation. The authors effectively address the challenge of high-speed processing by leveraging parallel computing and profile matching techniques across patches and temporal dimensions.\n\nSTRENGTHS:\nThe paper introduces realSEUDO, a groundbreaking online method for real-time cell detection and fluorescence estimation in calcium imaging data. It significantly outperforms existing methods like OnACID and CNMF in both processing speed and accuracy, achieving frame rates of 80-200 fps while maintaining high detection precision. The innovative integration of temporal and spatial profile matching, combined with efficient patch-based parallel processing, marks a substantial advancement in the field. Detailed experimental validation further underscores realSEUDO’s superior performance, adaptability, and potential for scaling with faster imaging technologies.\n\nWEAKNESSES:\nWhile the paper presents a novel and efficient approach with realSEUDO, several limitations are apparent. The reliance on specific hardware configurations, such as Linux CPU performance modes, could limit accessibility and generalizability. Additionally, the implementation primarily in C++ with MATLAB wrappers, while effective, could benefit from broader integration into Python pipelines to enhance usability. The paper also assumes access to external motion correction algorithms and does not address the full pipeline holistically, which may impact its applicability in scenarios without these additional tools.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe paper presents notable advancements with realSEUDO but has some limitations. Firstly, the performance of the system is highly dependent on specific hardware configurations and the proper setting of Linux CPU manager modes, which may not be universally applicable. Additionally, the reliance on MATLAB for prototyping, combined with limited Python integration, may restrict its accessibility and ease of use for a broader audience. Moreover, the paper focuses on cell detection without fully integrating or addressing the motion correction and complete processing pipeline, which could limit its applicability in real-world scenarios where such components are not readily available.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-22"}