realSEUDO for real-time calcium imaging analysis
Authors: Iuliia Dmitrieva
Sergey Babkin
Adam Shabti Charles
Venue: NeurIPS 2024 poster (2024)
Keywords: Calcium imaging
Real-time processing
Neuroimaging
Robust estimation
Sparse models
Primary area: neuroscience_and_cognitive_science
TL;DR: This paper presents a new real-time calcium imaging processing algorithm that is fast, robust, and flexible.

Abstract: Closed-loop neuroscience experimentation, where recorded neural activity is used to modify the experiment on-the-fly, is critical for deducing causal connections and optimizing experimental time. Thus while new optical methods permit on-line recording (via Multi-photon calcium imaging) and stimulation (via holographic stimulation) of large neural populations, a critical barrier in creating closed-loop experiments that can target and modulate single neurons is the real-time inference of neural activity from streaming recordings. In particular, while multi-photon calcium imaging (CI) is crucial in monitoring neural populations, extracting a single neuron's activity from the fluorescence videos often requires batch processing of the video data. Without batch processing, dimmer neurons and events are harder to identify and unrecognized neurons can create false positives when computing the activity of known neurons. We solve these issues by adapting a recently proposed robust time-trace estimator---Sparse Emulation of Unused Dictionary Objects (SEUDO) algorithm---as a basis for a new on-line processing algorithm that simultaneously identifies neurons in the fluorescence video and infers their time traces in a way that is robust to as-yet unidentified neurons. To achieve real-time SEUDO (realSEUDO), we introduce a combination of new algorithmic improvements, a fast C-based implementation, and a new cell finding loop to enable realSEUDO to identify new cells on-the-fly with no "warm-up" period. We demonstrate comparable performance to offline algorithms (e.g., CNMF), and improved performance over the current on-line approach (OnACID) at speeds of 120 Hz on average. This speed is faster than the typical 30 Hz framerate, leaving critical computation time for the computation of feedback in a closed-loop setting.

realSEUDO for real-time calcium imaging analysis
Iuliia Dmitrieva
Sergey Babkin Microsoft Research Redmond, WA, USA
Applied Math and Statistics Johns Hopkins University Baltimore, MD 21218, USA
Adam S. Charles
Biomedical Data Science
Center for Imaging Science
Mathematical Institute for Data Science Kavli Neuroscience Discovery Institute
Johns Hopkins University Baltimore, MD 21218, USA adamsc@ hu. edu
Abstract
Closed-loop neuroscience experimentation, where recorded neural activity is used to modify the experiment on-the-fly, critical for deducing causal connections and optimizing experimental time. A critical step in creating closed-loop experiment is real-time inference of neural activity from streaming recordings. One challenging modality for real-time processing is multi-photon calcium imaging (CI). CI enables the recording of activity large populations of neurons however, often requires batch processing of the video data to extract single-neuron activity from the fluo- rescence videos. We use recently proposed robust time-trace estimator-Sparse Emulation of Unused Dictionary Objects (SEUDO) algorithm new on-line processing algorithm that simultaneously identifies neurons the fluorescence video and infers their time traces in way that is robust to ys- unidentified neurons. To achieve real-time SEUDO (realSEUDO), we optimize the estimator via both algorithmic improvements and fast C-based implementa- tion, and create a new cell finding loop to enable rEalSEUDO to also identify new cells. We demonstrate comparable performance to offline algorithms (e.g., CNMF), and improved performance over the current on-line approach (OnACID) at speeds of 120 Hz average.
basis for
Introduction
Closed loop experiments enable neuroscientists to adapt presented stimuli or introduce perturbations (c.g., optogenetic stimulation) in real-time based on incoming observations of the neural activity. Such experiments are critical for both optimizing experimental time, e.g., by optimally selecting stimuli to fit neural response models [TO], or by deducing causality by perturbing possible cause- and-effect hypotheses. Despite this critical need, closed loop experiments at the level of populations of single neurons is incredibly difficult as they require real-time processing of neural data, which can be computationally intensive to process. In particular, population-level recordings using modern technologies often require significant computation to extract individual neuronal activity traces. e.g., spike sorting for high-density electrode electrophysiology or cell detection in fluorescence microscopy 6
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

One particularly challenging recording technology is fluorescence microscopy, in particular multi- photon calcium imaging (CI). CI has progressed significantly since its inception with optical advances enabling access to larger fields of view, and therefore higher data throughput. While neuroscientists now have access to in the video as fluorescing objects. To extract each neuron's 's activity, multiple methods have been developed, including matrix factorization approaches, deep learning approaches, and others (we refer to recent review for more complete coverage of available methods and their nuances [6].
of neurons at time, the neuronal time traces embedded
Almost all current calcium image processing methods uses batch processing: i.e., using full video all at once to identify the neurons in the data and their time traces. For example, common approach to identify cells in a mean image (the image containing the average fluorescence per pixel over all time) and then to extract the time-trace from the video given the neuron's 's location, e.g., by averaging pixels. Real-time processing does not afford such luxury. Instead, frames must be processed as they are collected. Furthermore minimal data can be stored and used, as large image batches reduce algorithmic speed. Finally, the incomplete knowledge of the full set of cells in the video can cause unintended cross-talk. Unidentified cells may overlap with known cells, causing well-documented effect of false transients when the unknown cells fluoresce [2].
We thus present an algorithm capable of demixing CI data frame-by-frame in real-time. Our design goals are operate at >> 30 Hz with minimal temporary data storage (e.g.. no buffering or initialization period needed) while minimizing false transient activity. Our primary contributions are: 1) An optimized SEUDO algorithm for fast, robust time-trace computation, 2) A new feedback loop to identify cells in real-time, and 3) patch-based parallelization that enables high-throughput calcium trace estimation across larger fields of view.
2 Background
Traditionally. CI analysis has been performed on full imaging videos. The goal of these algorithms to extract from a pixel-by-time data matrix Y E MRT, where M is the number of pixels in each frame and T is the number of frames, a set of neural profiles X E R MxN (one for each of N neurons) and a corresponding set of time traces & E NxT The former of these has, as each column of X, single component profile depicting which pixels constitute that fluorescing object, and how strong that pixel is fluorescing. The latter has as each row the corresponding time traces that represent how bright that object was at each frame. These time-traces are particularly important for relating neural activity to each other (i.e., modeling population dynamics) or to stimuli and behavior.
In typical approaches. full videos are required to either 1) identify summary images (e.g., mean 19) to identify cells in, 2) to create dataset within which points are or max images [24 IT 25
clustered into cells [7 31 22, 27 1 28 3 28 18 or 3) to perform simultaneous cell identification 29 (e.g., via matrix factorization or dictionary learning). For example, in the latter of these classes of algorithms, the data decomposition is solved via regularized optimization, e.g.,
and demixing 26 23 15 16 20
X - arg min X
Y + RX()) + (T))
+
where 11 is the Frobenius norm (sum of squares of all matrix elements), and Rx (X) and R. ( ()) are regularization terms for the profiles and time-traces, respectively. While many regularization combinations exist, common terms include sparsity in the neural firing, minimal overlaps, non- negativity, and spatial locality. Regardless, all methods require a large number of frames to identify the fluorescing components, with the exception of OnACID [4] and FIOLA 77.
OnACID and FIOLA operate in an on-line manner, utilizing the buffer of last 4 residuals r; y: - Xc; f where X and represent the spatial and temporal profiles of already recognized cells and B and f represent the spatial and temporal profiles of the known background signal. Both methods use a local Constrained
Matrix Factorization (CNMF) (26] in the spatial and temporal vicinity of that point. CNMF is an off-line algorithm that repeatedly performs alternating optimizations on [X, B and on [c. fI using the full dataset, until it converges to designated precision. Both methods require initialization periods, and FIOLA further requires GPU and CPU optimizaiton, raising the computational infrastructure costs. We seek a solution that does not need any initialization data and can be run on simpler CPU machines for easier incorporation into user's workflows.

---

2.1 Sparse Emulation of Unknown Dictionary Objects
One primary challenge in fully on-line settings the incomplete knowledge of all fluorescing components at the experiment onset. Even in off-line methods, incomplete identification of cells can create scientifically impactful cross talk talk-termed false transients- inferred activity [6]. Another challenge is identifying new components from few frames: ideally from individual frames to reduce memory usage. Recent work has provided an algorithm with the potential to solve both challenges: The Sparse Emulation of Unused Dictionary Objects (SEUDO) algorithm [12.
SEUDO is robust time-trace estimator for neuronal time traces. Given a single fluorescence video frame yt. and a set of known profiles X, SEUDO models contamination from unknown profiles We where W is basis of small Gaussian bumps that linearly construct the interfering components, weighted by the sparse coefficients (i.e., most values are zero). SEUDO then solves the optimization
3. arg min min 0.c20
X
Φb
y XΦ. Well + A||c|| + +
(2)
where À and 7 are model parameters and the internal min selects from the two internal expression that which has the minimal value. Since SEUDO operates per-frame, yr. Phvi, are all vectors. While SEUDO has demonstrated the ability to remove false transients [12). SEUDO's application has been limited to off-line post-processing due to: (1) slow computational speed, and (2) the need for pre-defined profiles X.
2.2 The FISTA Algorithm
The computational bottleneck in SEUDO is a weighted LASSO 32 optimization, which can be implemented with the Fast Iterative Thrikkag--hrresholdinng Algorithm (FISTA), which implements a momentum gradient descent 4. FISTA optimizes
min [F(x) f(x) + (()) e R=]
where f(x) is smooth convex function with Lipschitz constant L > 0, such that [V f((i)l S L||x - y|[Vz, y E R", egg. in SEUDO f(x) = ly: - X Φ. W Wells. and g(z) is
continuous convex function that is typically non-smooth, e.g., g(x) A||=||||. Each descent step of FISTA consists of an ISTA descent step and momentum step:
L
Fk arg min g(x)
yk f(((k)
(4)
2
yk++ = x + nh (x 11)
(5)
1+ / 1+4r
where the parameter уi gradually reduces with 7/x
s==1
1R+1
1.
Ex+1
3 Real-time SEUDO
Here we develop Real-time SEUDO (realSEUDO) that resolves the primary limitations of SEUDO and extends the algorithm significantly from time-trace trace estimator to a real-time cell identification method. Specifically we improve the computationally intensive momentum descent algorithm used to solve Equation (2) by reducing the number of steps of momentum descent, implementing parallelism, optimizing internal computations (e.g., of smoothness parameters), and reducing the complexity of the original fitting problem without substantial loss of quality by manipulating its inputs. Moreover we add a new algorithm that automatically recognizes the neurons that have not previously been seen and adds them to the dictionary of known components. Finally, we implement our framework with patch-based parallelism that avoids the computational scaling of LASSO in higher dimensions.
At high level, the realSEUDO algorithm (Alg. Fig. T) operates as follows: realSEUDO is initialized with zero known components (an empty set). When fluorescence activity in a frame reaches threshold, an event is triggered that saves the activity profile of that event as temporary candidate profile. Profiles are moved from the temporary profiles to the static set profiles if they remain active for sufficient number of frames The static profile set is then used to identify the activity of those components future frames, with unexplained components becoming candidate profiles and cycling back into the temporary profiles, followed by an update of the static profile set.
3

---

A
infer
motion
Data Y
Sparial profiiss
Time macos
adigt
(nosa reduction)
known component activity
residuall
current, frame
single profile
Wc
typical model
Updata companent list
Short
wobelly
True
SEUDO
Nerepp
X
model
single SEUDO
Ang. Fubk
Figure 1: The realSEUDO algorithm. A: Real-time inference of cells and their activity from calcium imaging crucial to closed-loop experiments, however, Typical CI demixing requires batch processing, e.g., via matrix factorization. B: realSEUDO builds the robust SEUDO algorithm that prevents activity in missing or unknown cells from creating false activity in known cells by explicitly modeling contamination
sparse sum of small Gaussian blobs (right). The sum of the estimated Guassians further provides an approximation of shape of the unknown cells, which can be used to known cells. C: We propose method based around the SEUDO estimation algorithm that can identify cells in real time by robustly removing known cells and using the residuals to identify new cells in the data.
3.1 SEUDO optimization
The first requirement of realSEUDO is a fast implementation of SEUDO that can operate at > 30 fps.
We achieved this requirement through a combination of efficient implementations, algorithmic optimization, and updates to the base SEUDO model.
C++ implementation: The original SEUDO implementation used the TFOCS MATLAB library S We thus first improved SEUDO's run-time by switching from MATLAB'S interpreted programming language running TFOCS fast implementation of LASSO 32 via FISTA 4 in the C++ compiled language. To further improve performance, we optimized the C++ code with the use of templates eliminate the function call overhead in tight loops. and also employ parallelism, based on the POSIX threads with TPOPP library wrapper (2). To prevent bottleneck from the passing of data through Matlab's OOP API the MATLAB/C+ interface, we switched the non-OOP version of the MATLAB-to- API.
While beneficial, the C++ SEUDO implementation did not alone achieve the desired processing rate. We further improved runtimes by optimizing the cost function and derivative computations. The partial LASSO component of SEUDO that performs optimization at each frame y using FISTA can be written as arg min f(bb) + )g(zb), where f = y x3513 is the least-squares term and ||*/]| is the & penalty. In FISTA, the profile time traces and Gaussian kernels are unified in vector, i.e., ab is concatenation of & and c, and X xXx, W.. With A as the number of pixels per frame, N as the number of neurons, and K as the number of Gaussian kernels, the set of problem dimensions are RM, X R (NN+R) 2b N+K X N+K with the first N elements of À corresponding to & being equal to 0.
In FISTA, number of internal computations become bottlenecks; in particular computing the gradients Vf and (f + g), the Lipshitz smoothness estimation, and the momentum/stopping criteria.
Gradient computation: We reduce the burden of the gradient computations by both reducing the number of times the gradient must be used, and by improving the internal gradient computation. For the former, we note that naive implementations compute both step the direction of V and then in the direction of V (f Åg). Moreover, we note that these two steps in slightly different directions cause the gradient to dither around the optimum. We thus instead only take step in the direction of (s >g) (similar to (4). For the latter, computing (a( + >g) requires matrix vector multiplications with X and its transpose. Since X sparse, we save memory and computation, by generating X on-the-fly via convolutions instead of storing it in memory. Specifically, the gradient

---

Vol requires computing x' X2b, which we reorganize to compute in two passes:
df
yj - X,ith..
=
Σ
Xim Un.
(6)
dep...
ISiNN+
IS/SM
The first pass (Eqn. 6 left) computes set of intermediary variables, and the second pass (Eqn. right) uses these values to compute the gradient dimensions. The two pass approach factors out repeated computations, reducing the complexity from Onn)) to (n Moreover, the computation of each pass is highly parallelizable by partitioning of the first pass by j. the second by m, and efficiently skipping the iterations over the zero elements in the sparse matrix X.
Lipshitz constant: To improve the efficiency of estimating the Lipschitz constant L, note that == max (I f V f=(x)))//| xy |I). For our cost function, we can approximate L with independent computations in each dimension. This estimation reduced the number of steps by as much as 30% over typical computations of L before each step based on the local gradient.
The momentum descent central to solving the partial LASSO tends to spend many steps on stopping the momentum, especially with the large values of the Lipshitz constant L (i.e. the non-momentum steps are small). One such case is "circling the drain" around the minimum, with the momentum causing the overshoot in one dimension while another dimension is stopping. Another case is when a dimension is moved past the boundary (e.g., æ 2 0 for SEUDO), where the solution pushes past the boundary into negative values. This produces suboptimal solutions and increasing the number of steps necessary. FISTA includes a parameter 1 that progressively limits the top speed of descent to reduce such problems. We improved these cases by resetting the momentum to zero dimension when it either attempts to cross into the negative values or when its gradient changes sign. The dimensional momentum stopping stops abruptly at the right time, obviating the need for slowing and thus we can simplify FISTA by fixing 1 = 1.
Momentum:
Our momentum stopping can further extend to broader optimization problems. We demonstrated this ability on momentum optimization in neural network training (see Supplement), where it provided a substantial improvement. Our modified FISTA produced the same error rate and squared mean error as gradient descent about 10 times fewer training passes, or about 10 times lower error rate and 1.5 times lower mean square error in the same number of passes. The full modified FISTA algorithm is presented in Algorithm 2 (see Supplement).
SEUDO model adjustments: As third step. we modified the SEUDO optimization program to achieve the final speedups. The original SEUDO spaced the Gaussian components in W by one pixel, which we found to be highly redundant. The kernels with radius cover (æ * x pixels, and thus each pixel is covered by (IT r²) kernels. This redundancy results in FISTA continuing to adjust the kernel coefficients e after the neural activations æ converge. Reducing the number of kernels thus reduces both the number of gradient descent steps and the per-step cost, accelerating the computation more than quadratically. For kernel with diameter of 30 pixels this improves the performance by factor of over 100 without substantial degradation of false transients removal or the recognition of the interfering components' shape Wc.
We benchmarked our speedups against the original SEUDO on 45000 frames across 50 cells from 12 SEUDO ran at 5.8-6.9 s/cell on Macintosh M1. The optimized C++ implementation without the MATLAB C++ API reduced the runtime 0.9-1.1 s/cell (a 6-7x improvement). Sparse SEUDO provided further acceleration to run time of 0.2 s. with 0.1 for the computation and 0.1 for the overhead of converting data between Matlab and native code; a total of a 29-34.5x speed-up, allowing SEUDO to run in real time.
3.2 Automatic cell recognition
We next developed the cell recognition feedback loop that completes the realSEUDO algorithm. To minimize data storage and compute, we designed realSEUDO to run on a frame-by-frame basis. At a high level, our automatic cell recognition first runs SEUDO on the current incoming (denoised) frame given the currently identified profiles X Xstat. The loop then identifies contiguous bright areas in the residual frames, i.e., the SEUDO cells We, and places them in a 'temporary profile' array Xtemp- The temporary profiles are then updated (via merging with new potential profiles) given new, incoming, frames until they are stable and moved to the stable, known profile list X stab that updated less frequently by addition, merging and splitting of temporary profiles.
5

---

Procedurally, we first preprocess each incoming frame to reduce noise and improve profile recognition. Calcium imaging analysis often uses running averages in space and time for noise reduction. We thus implement both a spatial Gaussian filter, as well as running average of several sequential frames. We keep the window length as a tunable parameter that can be set to one for frame-by- by-frrme processing with minimize temporal blurring.
We estimate the activation level in the denoised frame for each of the stable profiles Xatab using the our fast SEUDO implementation. SEUDO returns the activation level Φu for the bth profile at time t along with a robust residual that contains the structured fluorescence not captured by X staab. We then run the residual through SEUDO a second time using the temporary profiles X Xtemp to test if any temporary profile matches the frame's fluorescence and should be moved from Xtemp to stabb The residual after the second SEUDO application represents completely unknown profiles and are analyzed separately to determine if a new member of X temp should be created.
The detection of new temporary profiles is based on finding the areas of the image above the noise level. The noise level evaluated by noting that most of each video frame has no activity, indicating that the median pixel value will be very close to the median value of the pixels in an all-dark frame containing the same noise. The half-amplitude of the noise 01/2 can be estimated as:
012 = median yy) - (it.))
In some cases different areas of the image may contain a different amount of background lighting (e-g., changes in neuropil), which can skew the noise estimate. To overcome this challenge, we split the larger image into sections, with each section computing a local median which is smoothly interpolated between section of the image. This can be thought as either krigging procedure cheap approximation to local median evaluated independently for each pixel in the image.
To merge new profiles into the exiting profiles when adding new profiles to X tempp we compute an overlap score. The scores aims to capture the following logic: If two temporal profiles are a close match in cross-section. they likely represent the same cell and should be merged. If they overlap only partially, they likely represent separate cells. If one cross-section is inside the other, look at relative brightness: if the smaller cross-section is also weaker, it's likely a weaker partial activation of the same cell and should be merged, if the smaller cross-section has a close or higher brightness, the larger cross-section likely represents an intertwining of two cells that has be split.
The overlap computation thus not a plain spatial overlap but includes a heuristics that identifies when one profile is mostly contained in another. The condition for merging two profiles in Xtemp is based on the comparison of numbers of common and unique pixels between profiles, where P1 and P2 are numbers of pixels in each of two profiles, U, and U2 are the numbers of unique pixels in each profile, C' is the number of common pixels, B1 and B2 are the perimeters of bounding boxes for each profile, and ktemp is a constant with an empirically chosen value of 0.75:
U1 S B1 0.5
U2 S B2 0.5
C 2 Ktemp min(P1,
(8)
After a profile is moved from X Xtemp to stabb, different score is computed pair-wise between the new profile and each existing profile, to decide whether they should be left separate, or merged, or one of profiles split. If a merge or split is performed, the original profiles removed from X stab and the results entered recursively into stab as new profiles. The score for two profiles A and B in Xstab is computed based on the brightness and measures of least-squares fit of the cells into each other 1) as whole cells (i.c., QAB = (A, B) (A, A)) and 2) using only the overlapping region (i.c., BAB (Aat, BB (Aot, Aot) where Anr is the profile A restricted to the region overlapping with B). 9AB is used as a measure of difference in brightness, against which the fit of the whole cells a is compared as a measure of proximity in shape. Specifically we compute two ratios PAB and PBA are computed as
QAB
PBA = OBA
PAB
99)
AAB"
A higher value of PAB (which is always < 1) means that cell A fits better inside cell B. The value of I means that it fits entirely inside cell B. The same principle applies symmetrically to PBA-
3.3 Patching and profile matching
realSEUDO, although highly efficient for smaller patches, is still based on the LASSO algorithm that reduces in efficiency with much larger frames. Thus adopt a patching scheme that breaks each

---

frame into small patches that can be parallelized to maintain the high framerate by utilizing the multi- threading many modern processors. Patching, however, requires matching profiles across patches. Traditionally profiles discovered in data split into patches is to add overlap margins to the patches and to use profiles overlap in this region to determine matchings in neighboring patches. Additional margins, however, introduces redundant computation and decreases computational efficiency.
In realSEUDO we note that the logic behind the scoring we use to merge profiles in stab and Xtemp within each patch can also be used to score the match of profiles across patches. Specifically, we extend the matching to include the profile temporal activity as an additional dimension to find matching cells in neighboring patches via consecutive gluing of the profiles. The highest score assigned to the bidirectional match of both spatial and temporal dimensions, a lower score to symmetrical match of spatial dimensions and asymmetrical match of temporal dimension, a yet lower score to an asymmetrical match of both kinds of dimensions. We have observed successful matches even with zero margin, using the neighboring strips of pixels around the perimeters of the patches as the spatial dimension for matching.
4 Results
To validate our approach we note that the main goal of realSEUDO, as with most functional imaging analyses. primarily aims recover the time-traces of neural activity as accurately as possible [6] [2]. This means while the general location of neurons is important, metries such as the the Intersection over Union (loU) are too strong; i.e., the full set of pixels identified not necessarily the important quantity. We instead compute the "unique neurons found". This metric aims to capture the need to know that the time-traces 1) correspond to real neurons in the data and 2) accurately reflect the temporal activity that will be used to study neural activity with respect to stimuli and behavior. The Unique Neurons Found (as defined in (30) requires both that ROIs well align with known ROIs spatially (overlap of at least 50% of pixels) and that the time-trace correlation exceeds 0.5.
Validation metries:
Simulated data experiments: We first applied all three algorithms to simulated video created with Neural Anatomy and Optical Microscopy simulation (NAOMi) (0) Specifically we simulated the neural activity over 20000 frames at 30Hz with fame size of 500x500 pixels. There are approximately 450 cells visible in this dataset (i.e. fluorescing cells intersecting the plane of imaging). We benchmarked the patch-based parallel processing of realSEUDO 80x80 pixel patches. For comparison we ran the off-line CNMF (a staple batch-based calcium imaging demixing algorithm) and OnACID, the computationally similar on- on-line method. realSEUDO found 201 true cells, identified as strongly correlated with ground-truth cells, while OnACID found 152 and CNMF (the offline method) found 308 (Fig. -B) Furthermore, both realSEUDO and OnACID found many fewer false positives than CNMF, presumably because they cannot be fooled by small fluctuations integrated over the full recording (Fig. (Fig.2C-D), Note that to remove the confound of post-processe post-processing we followed prior work [30) in using the CNMF raw fluorescence traces instead of the model-based denoised traces. On average, realSEUDO processed 67.8 frames per second end-to-end, while OnACID ran at 9.2 fps.
Applications to in-vivo mouse CA1 recordings We applied realSEUDO to an in vivo calcium imaging recordings from mouse hippocampal area CA1 previously described in Gauthier et al. 2022 [1). They consisted of 36 videos, each sized 90x90 pixels with 41750 frames sampled at 30 Hz. The outputs had previously been verified manually by Gauthier et al. 2022 [12] with human labeling of CNMF outputs. We applied realSEUDO to all videos and compared the outputs with the current online cell demixing algorithm OnACID [1), as well as a popular offline algorithm, CNMF [3. as an additional baseline.
We benchmarked realSEUDO against OnACID (an online analysis tool) and CNMF (an offline analysis tool) on real in-vivo calcium imaging movies (Fig. 3 Algorithmic performance was measured on an x86-64 computer with 48 CPU cores (Intel Xeon 6248R). 2 hyperthreads per core, 78 GB of memory, and without the use of GPU. The initialization times were not included. On average, realSEUDO processed 162 frames per second compared to 26 processed by OnACID and 13 by CNMF: an improvement of 6.5x and 12.5x respectively (Fig. 3C). Quality-wise, we found that OnACID exhibited difficulties with adapting to larger ranges of pixel brightness, sometimes missing bright cells. Scaling pixel values improved OnACID results, but only mildly (one additional cell). Numerically OnACID and CNMF appear similar but they identified different components. SEUDO
7

---

A
Profiles found in the NAOMi data
B
Hits and false alarms
CNMF
OnACID
realSEUDO
NAOMI profiles
CNMF
OnACID
realSEUDO
Strong pairs
profies
300
200
100
100
100
Hits
False
Ambig-
alarms
uous
Weak pairs
C
Temporal correlations to ground truth cells
CNMF (offline)
OnACID (9.4+)- 0.3 Hz)
realSEUDO (67.8+/- 24.1 Hz)
100 pm
100 pm
130 jum
protlees
80
60
Un-paired
20
0
100 jm
100
100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Correlation with ground truth
D Example matched profiles to ground truth traces
Ground truth
CNMF
OnACID
realSEUDO
Profile 1
Fluorescence (AU)
Profile 2
Profile 3
2000
4000
6000
Frame number (simulated at 30 Hz)
Figure 2: NAOMi results: A) Found cells in NAOMi for CNMF, OnACID and realSEUDO separated
into Hits (strong or weakly correlated) and false alarms (uncorrelated). B) realSEUDO finds more cells than OnACID with minimal false positives. C) Temporal correlations for found "hits". D) Examples time-traces show correlation to ground truth.
results were most similar CNMF, and with additional cells identified, and less false positives
(Fig. 3). Finally, the per-transient manual classification provided by [12] enabled us to assess
realSEUDO inherited the false transient removal properties of SEUDO. For reasonable value of
A = 0.15, realSEUDO had true positive rate of 75% and a false positive rate of 24%. While these numbers are a bit lower than the numbers reported in [12], in that study the authors average N=3 frames to reduce noise, while maintained single-frame analysis. This can be evident by the fact that missed transients were very small: realSEUDO kept 98% of real fluorescence and only 15% of false fluorescence.
Additional in-vivo tests: As final test applied all three algorithms (realSEUDO, OnACID, CNMF) to 2000-frame mesoscope video example collected by the Yuste lab at Columbia University and provided with the OnACID github package as demo. For this example similarly saw improved cell detection and runtime improvement in terms of fps (Fig. 33)
Discussion
We present here an online method for cell detection and fluorescence time-trace estimation from streaming CI data: realSEUDO. realSEUDO is based on the SEUDO robust time-trace estimator that reduces bias due to unknown cells while also providing approximate shapes of the unknown fluoresc-

---

A
Estimated Spatial Profiles
Estimated Activity
300
roal SEUDO
250
Framos/eecood
(FF)
200
150
100
Detected ROls
0.2
24
OnAcid demo dataset
Mouse CA1 dataset
CPU secordsfiramms
0:18
real-SEUDO
14
components
19
OnACID
components
0.12
7
7
0.1
2
0
0.08
20
40
60
80
Posiiives
Negalloss
Falis
Nageliino
Detected ROIs
Figure 3: (A) Selected cell profiles and time-traces generated by realSEUDO, OnACID, and CNMF respectively on a subset of 2000 frames from a single image patch. (B) Counts of true positive, false positive and false negative cells found by each algorithm for two different recordings: all 41,750 frames data from one video from Gauthier et al. 12] (right) and from the OnACID demo (left). (C) Top: Total computational performance
function of the number of detected cells for realSEUDO and OnACID, evaluated on the full set of 36 movies from Mouse CA1. Bottom: CPU in CPU seconds per frame function of the number of detected cells for realSEUDO and OnACID, evaluated on the same 36 recordings.
ing objects. To build realSEUDO we 1) improved SEUDO's runtime via significant modifications at code, algorithmic, and model levels 2) built new feedback loop that allowed SEUDO (that has no cell finding component currently) to identify cells in real-time. Overall, realSEUDO can achieve frame processing rates of 80-200 fps, depending on cell density. While our goal was to exceed the typical 30 Hz data collection rate common to many experiments, the high processing efficiency leaves additional time to compute feedback in future closed loop systems. Moreover, realSEUDO can scale with faster recording rates as calcium indicators become faster, e.g., GCaMP8 33)
realSEUDO's implementation exhibits a higher degree of parallelism than OnACID, however both are likely constrained by the employed tools. Specifically. the measurements in Fig. 3C show very little fps fluctuation with respect to cell count for OnACID, which can be due to bottleneck in single thread. realSEUDO has lower latency to the first events for new cell, OnACID requires history of 100 frames to recognize cell, while realSEUDO would produce the first events starting with the first frame that passes the low brightness threshold. Alternatively, OnACID has higher native scalability with respect to the frame area, conditioned on similar cell counts. This likely result from OnACID's algorithm restriction of processing to areas in the immediate vicinity of known cells. realSEUDO can still achieve a high processing efficiency with reasonable hardware with our parallelization. Future work may further improve realSEUDO runtime by blanking out entire patches until activity is detected via simpler detectors.
One strength of realSEUDO is that it can be initialized with either an empty profile set. This ability to start from nothing will be useful in mesoscope settings when the field-of-view can be changed on-the-fly. Not requiring Furthermore, many of our speed adaptions deviate from the traditional gradient descent approach. In particular the Lipshitz constant approximation and the changes in the momentum and stopping criteria. In other domains, in particular for training deep neural networks, these deviations may also provide significant speedups, the extent of which should be quantified across broader applications in future work.
initialization step will reduce start-up overhead at new fields of view.
While our results achieve the design criteria we initially set out, there are some potential barriers. For one, as with many real-time systems, the compute environment very important to configure correctly. We have found the importance of explicitly setting the Linux CPU manager to enable the performance mode. The default automatic adaptable mode does not react properly to CPU loads of less than 100% of the whole capacity, and significantly skews the benchmarking by running
Limitations:

---

the CPUs at low frequency. These challenges are unfortunately necessary to achieve high levels of throughput without specialized hardware. Future work should develop walkthroughs and automated tools to guide the installation of the tools.
While core algorithm is written completely in C++, and thus open source, have found MATLAB convenient and efficient as a wrapper for prototyping wrappers for our core functions. Further work will add Python wrappers allow for seamless integration into both MATLAB and Python pipelines, enabling realSEUDO to be more widely used.
Finally, focused here only cell detection, assuming access to the on-line motion correction algorithm from OnAcid. This focus may require additional packages to be handled by users for motion correction. We will further aim in future iterations to extend the package include motion correction and delta-F over F computations in order to reduce communication overhead and ease adoption by users. Moreover, to fully optimize the package for speed, these steps should be more holistically incorporated in C++ as well.
References
[1] Noah Apthorpe, Alexander Riordan, Robert Aguilar, Jan Homann, Yi Gu, David Tank, and H Sebastian Seung. Automatic neuron detection in calcium imaging data using convolutional networks. Advances neural information processing systems, 29:3270-3278, 2016.
[2] Sergey A Babkin. The practice of parallel programming. Createspace, 2010.
[3] Yijun Bao, Somayyeh Soltanian-Zadeh, Sina Farsiu, and Yiyang Gong. Segmentation of neurons from fluorescence calcium recordings beyond real time. Nature Machine Intelligence, pages 1-11,201 2021.
[4] Amir Beck and Mare Teboulle. A fast iterative shrinkagg-thressholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.
[5] Stephen R Becker, Emmanuel J Candès, and Michael C Grant. Templates for convex cone problems with applications to sparse signal recovery. Mathematical programming computation, 3:165 218, 2011.
[6] Hadas Benisty, Alexander Song, Gal Mishne, and Adam S Charles. Review data processing of functional optical microscopy for neuroscience. Neurophotonics, 9(4)004140, 2022.
[7] Changjia Cai, Cynthia Dong. Johannes Friedrich, Marton Rozsa, Eftychios A Pnevmatikakis, and Andrea Giovannucei. Fiola: an accelerated pipeline for fluorescence imaging online analysis. Nature Methwds, 20(9): 2023.
[8] Adam S. Charles, Nathan Cermak, Rifqi O. Affan, Benjamin B. Scott, Jackie Schiller, and Gal Mishne. Graft: Graph filtered temporal dictionary learning for functional neural imaging. IEEE Transactions on Image Processing, 31:3509-3524 -5244, 2022.
[9] Adam S Charles, Benjamin Falk, Nicholas Turner, Talmo D Percira, Daniel Tward, Benjamin D Pedigo, Jaewon Chung, Randal Burns, Satrajit S Ghosh, Justus M Kebschull, et al. Toward community-driven big open brain science: open big data and tools for structure, function, and genetics. Annual review of neuroscience, 43(1):441 464, 2020.
[10] Adam S Charles, Mijung Park, J Patrick Weller, Gregory D Horwitz, and Jonathan W Pillow. Dethroning the fano factor: flexible, model-based approach to partitioning neural variability. Neural computation, 1012-1045111 2018.
[11] Diego and A. Hamprecht. Sparse space-time time deconvolution for calcium image analysis. In NIPS, pages 64-72, 2014.
[12] Jeffrey I. Gauthier, Sue Ann Koay, Edward H Nieh, David W Tank, Jonathan W Pillow, and Adam Charles. Detecting and correcting false transients in calcium imaging. Nature Methods, 19(4):4704477 2022.
10

---

[13] Andrea Giovannucci, Johannes Friedrich, Pat Gunn, Jérémie Kalfon, Brandon L Brown, Sue Ann Koay, Jiannis Taxidis, Farzaneh Najafi, Jeffrey L Gauthier, Pengcheng Zhou, Baljit Khakh, David W Tank, Dmitri B Chklovskiil, and Eftychios A Pnevmatikakis.. CalmAn an open source tool for scalable calcium imaging data analysis. Elife, 8:e38173, 2019.
[14] Andrea Giovannucei, Johannes Friedrich, Matthew Kaufman, Anne K Churchland, Dmitri Chklovskii, Liam Paninski, and Eftychios A Pnevmatikakis. Onacid: online analysis of calcium imaging data in real time. In Proceedings of the 31 International Conference Neural Information Processing Systems, pages 2378-2388, 2017.
[15] Benjamin D Hacffele and René Vidal. Structured low-rank matrix factorization: Global optimality, algorithms, and applications. IEEE transactions on pattern analysis and machine intelligence, 42(6):1468-1482 2019.
[16] Hakan Inan, Murat A Erdogdu, and Mark Schnitzer. Robust estimation of neural signals in calcium imaging. In NIPS, pages 2905-2914, 2017.
[17] Patrick Kaifosh, Jeffrey D. Zaremba, Nathan B. Danielson, and Attila Losonezy. Sima: Python software for analysis of dynamic fluorescence imaging data. Frontiers in Neuroinformaties 8, 2014.
[18] Elke Kirschbaum, Alberto Bailoni, and Fred A Hamprecht. Disco: Deep learning, instance segmentation, and correlations for cell segmentation in calcium imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 151-162. Springer, 2020.
[19] Alcksander Klibisz, Derek Rose, Matthew Eicholtz, Jay Blundon, and Stanislav Zakharenko. Fast, simple calcium imaging segmentation with fully convolutional networks. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages Springer, 2017.
[20] Ryuichi Maruyama, Kazuma Maeda, Hajime Moroda, Ichiro Kato, Masashi Inoue, Hiroyoshi Miyakawa, and Toru Aonishi. Detecting cells using non-negative matrix factorization on calcium imaging data. Neural Networks, 55:11-19, 2014.
[21] Gal Mishne and Adam S Charles. Learning spatially-correlace temporal dictionaries for calcium imaging. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-2019), pages 1065-1069, IEEE, 2019.
[22] Gal Mishne, Ronald Coifman, Maria Lavzin, and Jackie Schiller. Automated cellular structure extraction in biological images with applications to calcium imaging data. bioRxiv, page 313981, 2018.
[23] M. Pachitariu, A. M. Packer, N. Pettit, H. Dalgleish, M. Hausser, and M. Sahani. Suite2p: beyond 10,000 neurons with standard two-photon microscopy. bioRxiv, 2016.
[24] Marius Pachitariu, Adam M Packer, Noah Pettit, Henry Dalgleish, Michael Hausser, and Maneesh Sahani. Extracting regions of interest from biological images with convolutional sparse block coding. Advances in Neural Information Processing Systems, pages 1745-1753, 2013.
[25] Ashley Petersen, Noah Simon, and Daniela Witten Scalpel: Extracting neurons from calcium imaging data. The annals of applied statistics, 12(4):2430. 2018.
[26] E. A Pnevmatikakis, D. Soudry. Y. Gao, T. A. Machado, J. Merel, D. Pfau, T. Reardon, Y. Mu, C. Lacefield,
W. Yang, Misha Ahrens, Randy Bruno, Thomas M. Jessell, Darcy S. Peterka, Rafael Yuste, and Liam Paninski. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron, 89(2):285-299 2016.
[27] S. Reynolds, T. Abrahamsson, R. Schuck, P. Jesper Sjöström, S. R. Schultz, and P. L. Dragotti. ABLE: An activity-based level set segmentation algorithm for two-photon calcium imaging data. eNeuro, 2017.
11

---

[28] Somayyeh Zadani, Kaan Sahingur, Sarah Blau, Yiyang Gong, and Sina Farsiu. Fast and robust active neuron segmentation in two-photon calcium imaging using spatiotemporal deep learning. Proceedings of the National Academy of Sciences, 116(1))88544 8563, 2019.
[29] Alexander Song, Adam S Charles, Sue Ann Koay, Jeff L Gauthier, Stephan Y Thiberge, Jonathan W Pillow, and David W Tank. Volumetric two-photon imaging of neurons using stereoscopy (vTwINS). Nature methods, 14(4):420, 2017.
[30] Alexander Song, Jeff L Gauthier, Jonathan W Pillow, David W Tank, and Adam S Charles. Neural anatomy and optical microscopy (naomi) simulation for evaluating calcium imaging methods. Journal of neuroscience methods, 358: 10917,, 2021.
[31] Quico Spaen, Roberto Asín-Achá, Selmaan N Chettih, Matthias Minderer, Christopher Harvey, and Dorit S Hochbaum. HNCcorT: novel combinatorial approach for cell identification in malcium-imaging movies. Eneuro, 6(2), 2019.
[32] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288) 1996.
[33] Yan Zhang, Márton Rózsa, Yajie Liang, Daniel Bushey, Ziqiang Wei, Jihong Zheng, Daniel Reep, Gerard Joey Broussard, Arthur Tsang, Getahun Tsegaye, et al. Fast and sensitive gcamp calcium indicators for imaging neural populations. Nature, pages 1-8, 2023.
12

---

A
Appendix
Algorithm 1 realSEUDO Algorithm
Initialize: X temp <- []: stab <-
2: for each frame Y/ do
3: Denoise Yt
4: Identify the profiles in the current frame
5: for all new profiles do
6: if current profile overlaps any of temp then
7:
Merge new Xtemp profiles into the currentXtemp else
8:
9:
Add the current profile to Xtemp end if
10:
11:
end for
12:
for all profiles in X temp that have not been updated in the last few frames do
13:
Move them from X teep to Xtrab
Merge the moved profiles with existing stah profiles
15:
end for
Se. Tt t SEUDO stab)
17:
Report Φ; as detections
Φ SEUDO Xtemp)
19:
for each profile l in X Xtemp do
if the d/ > ? or this profile was previously active then
21:
Report this activation as early detection
22: end if
23:
end for
24:
end for
A.1 Application of modified FISTA to neural network optimization
We further tested the modified FISTA momentum descent algorithm to problems outside of neuroscience-ttee the training of neural networks-to evaluate the scope of applicability of our improvements. We used the problem of recognition of handwritten digits on a data set from AT&T Research available at hhttts:s//hstttt DATA/Zipcode. html reduced to 8x8 pixels, with a training set of 7291 images. The neural network (NN) model used the Leaky ReLU activation, with layer sizes 64, 64, 32, 10.
su.
The common approach training NNs stochastic gradient descent, including stochastic momen- tum methods. Thus to compare to a non-stochastic momentum method we established a non-stochastic baseline.
The dynamic estimation of the Lipshitz constant L from TFOCS cannot be applied to the neural network optimization because the optimization cost is highly non-linear. The multi-dimensionn estimation of L is also computed only for the specific SEUDO function. The common practice is to use fixed descent rate, which serves still not a fully solved problem. Algorithms for dynamic evaluation of the descent rate do exist (e.g., the ADAM algorithm in Kingma & Ba, 2015), however they rely constant be picked for particular problem.
analog of श.. Estimating the highest descent rate is
The advantage of stochastic methods is that they use higher descent rate without diverging, as seen by observing the dependency of logarithm of mean square error from the number of training passes for various descent rates 6 In these results ensure that start from the same fixed randomized initial state since different initial states can produce wildly different results.
We observe that for the same descent rate (0.05 per pass), the stochastic and non-stochastic methods produce very similar error values, however the graph for stochastic method more smooth. The roughness of the graph represents the small divergences that manage to converge again over time, and shows that the descent rate is close the maximum. However the stochastic method can accommodate 100 times higher descent rate without diverging, and even a 1000 higher descent
13

---

Algorithm 2 Modified FISTA algorithm
1: Initialize 1 1; =1 (initial values); diff'] [0]: gradient_ last
[0]
2: for slep
mazslep do
3: Ineat 1+v1+222
4: "
1-1
Inez!
5:
if step 7 then
6:
1= Ineat
7:
end if
8:
=0 - 1) diff]
9:
for each dimensions of do
10:
if
<=| < 0 then
diff =
12:
end if
end for
14:
gradient[] = compute compute_gradient f(x]]
=1 =0 gradient[l
16:
for each in dimensions of x do
if [ 0 then
18:
=[i]] 0; di = O: gradient(i] = 0
else
20:
if gradient[i] * gradient last[i] < 0 then diff
22:
else
diff diffle] gradient(()
23:
24:
end if 25: end if
26: end for
27:
gradient_last] last[ gradient[]
28:
end for
Method
error
log(error)
stochastic
0.0956
-2.3476
baseline non-stochastic
0.0952
-2.3515
FISTA
0.1662
-1.7946
momentum+stop on gradient sign change
0.0699
-2.6607
momentum+stop on gradient sign change
0.0701
-2.6578
momentum+stop gradient sign change + at 4x rate
0.0751
-2.5889
auto-adjusted rate for momentum-stop on gradient sign change
0.0631
-2.7630
auto-adjusted rate for momentum+stop on gradient sign change +
0.0654
-2.7272
Table I: Performance of algorithms on handwritten dataset
rate becomes rough but still converges. The non-stochastic method is able to make the passes faster, because performs the same accumulation of partial gradients, but saves the overhead of updating the weights after each training case (or batch). However even adjusted for time, the stochastic method performs faster. possible to compute the non-stochastic gradient in parallel by multiple threads but we have not implemented this. We used this example of the non-stochastic descent as baseline for the FISTA-based momentum methods.
The summary of training errors in the momentum methods can be found in Figure 7 and the mean square errors after 10,000 training passes are listed in the Table
The unmodified FISTA algoithm with non-momentum baseline. Adding the momentum stop in the dimenstions with gradient sign change produced a substantial improvement over the baseline. Fixing the parameter v produced close result to not fixing 7, but with less rough curve. We tested if the smoothness indicated that setting
performed on this task out its domain worse than the
14

---

1 could accommodate substantial increase in descent rate by re-rum re-running the algorithm at 4x rate (0.2 instead of 0.05), and while this run did not diverge, did observe higher error rate.
Finally, we attempted to devise an algorithm that acts similar to the TFOCS dynamic evaluation of L but using the ratios of mean square values of gradient dimensions that change not change sign as an indication of roughness. The rapid growth of gradient dimensions after sign change is seen beginning of a divergence, that causes the reduction of descent rate. This algorithm allowed training at a substantially higher rate in the first few thousands of passes but then flattened out. The automatically determined rate is close the empirically found 0.05, and higher the initial passes where it reaches higher values, but then drops to the lower values (Fig. 7B). It is possible that the chosen criteria were not aggressive enough, and can be improved.
While event with the momentum descent the stochastic methods specialized for NN training can still achieve faster speeds (Fig. CCD)) we have demonstrated that more general optimization still represent a major improvement over both simple gradient descent and plain FISTA different domain.
15

---

CNMF
ON-ACID
realSEUDO
24
30
36
15
12
22
35
20
37
Cell ID
Figure 4: The whole set of cells detected by realSEUDO (bluc), OnACID (red), and CNMF (green)
in one movie, with selected time traces of matching cells.
16

---

realSEUDO output:
Cell ID
11
14
15
10
21
22
25
20
29
32
33
35
36
39
40
42
43
46
3
x10
Time (frames at 30 Hz)
Figure 5: Example realSEUDO cells and traces from a single patch, ordered by the discovery time.
17

---

Training by non-stochastic
Training eror: non
0.5
0.5
richelic
tixcwell
ren-tscheett
ren-trcdeatc
ticeatic 10s rate
racheeds
tichaaiic 1000x
stxchaids 1000
Loggowann squai aro))
Log recan sqgarr
355
4.5
4.5
Secords
Tramiin passees
Figure 6: Comparison of different algorithm's 's learning curves on handwritten datasets. Left: mean- squared error (MSE) as function of optimization time. Right: MSE as function of training passes.
A
Training error: non-stochastic momentum metheds
B
Automatically adjusted training rate
actorals
bacelie man-etochadi:
nomeeemm-ttiaee 1 rate
Logincan square aron
eiforale
murreaan Anp
Loginwea square amror
0.07
0.05
-3.5
0.05
0.03
0.02
C
2000
4000
Training passes
6000
5000
10000
D
2000
4:00
Training passes
6000
2000
10000
Training error: stochastic vs best non-stochastic momentum
Training error time: stochastic vs best non-stochestic momentum
momerfun--tiphne auttrttttee reehaatic 160s rate
autoradee.111 stocherl 200s nate
Logimcan square arren
335
-1.
-2
Logimeann square
3.5
2000
4200
6000
5000 10000
200
400
600
300
1000
1200
1400
Training passes
Seconds
Figure 7: Comaprison of training curves for different algorithms. A: Training MSE as function of training passes for different variants of the improved FISTA algorithm. B: Training MSE improvement when setting 1 = 1. C: Training MSE as function of training passes for the best tested non-stochastic methods vs. momentum-improvee FISTA D: Training MSE as function of optimization time for the best tested non-stochastic methods vs. momentun-improved FISTA
18

---

realSEUDO
OnACID
CNMF
1000
400
000
(477)
Profile number
Profile number
Profile number
200
150
100
150
15
100
00
50
006
200
100
5000
5000
5000
10000 Frame number (simulated at 30 Hz)
10000 Frame number (simulated at 30 Hz)
10000 Frame number (simulated at 30 Hz)
15000
15000
15000
20000
20000
20000
Fluoooo Eluorescancc (AU)
weeesssaunn (AU)
Conresscccnnn AAU
padd---aliee paired traces from CNMF, OnACID und SealSECa
Figure 8: Full sets of strongly
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes].
Justification: This work claims to provide new algorithm for real-time processing of calcium imaging data. The results show processing speeds of 80-200 Hz, far exceeding the >30 Hz minimum real-time processing.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We provide discussion in the discussion section that discusses explicitly our limitations.
Justification:
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer[[NA]
Justification: Our paper based on developing new algorithm, and we do not include theoretical claims.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The data from this paper is open and available and full pseudo-code and all parameter selections are provided in the main text supplementary.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: While the data all freely available, the implementation still requires some fine-tuning to be easy to for the general public. We plan release before the conference itself, and, provide full algorithmic and parameter selection details the paper.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our method provides all parameter selections. Note that these parameters were not fit to data in the optimization sense but rather selected based on rules thumb for the classes of algorithms used (sparse inference).
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Where applicable (e.g., timing assessment etc.) error bars provided.
8. Experiments Compute Resources
20

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: A number of compute infrastructures were used to compute the results, and no special hardware was needed (e.g.. GPUs).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics hhine//rrc/////errr....cc///actp
Answer: [Yes]
Justification: This paper does not use human subjects, private or protected data, provide any capability in that might compromise safety or security. This work is squarely in the scientific microscopy analysis domain for non-human experiments.
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work describes the broader scientific impacts, but the advances are squarely in the imaging and optimization domain.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This algorithm does not need safeguards as it is made for microscopy for non-human applications.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Everyone who contributed is properly referenced.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets presented.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing used.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No IRB was needed.
21


Reviews:
[{"review_id": "ubD4YUlE81", "reviewer": "Reviewer_86dC", "summary": "The paper presents a novel online method for cell detection and fluorescence estimation from streaming calcium imaging (CI) data, called realSEUDO. Building upon the SEUDO algorithm, realSEUDO introduces significant advancements in real-time cell identification and fluorescence estimation. The authors effectively address the challenge of high-speed processing by leveraging parallel computing and profile matching techniques across patches and temporal dimensions.", "questions": "N/A", "limitations": "The paper presents notable advancements with realSEUDO but has some limitations. Firstly, the performance of the system is highly dependent on specific hardware configurations and the proper setting of Linux CPU manager modes, which may not be universally applicable. Additionally, the reliance on MATLAB for prototyping, combined with limited Python integration, may restrict its accessibility and ease of use for a broader audience. Moreover, the paper focuses on cell detection without fully integrating or addressing the motion correction and complete processing pipeline, which could limit its applicability in real-world scenarios where such components are not readily available.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper introduces realSEUDO, a groundbreaking online method for real-time cell detection and fluorescence estimation in calcium imaging data. It significantly outperforms existing methods like OnACID and CNMF in both processing speed and accuracy, achieving frame rates of 80-200 fps while maintaining high detection precision. The innovative integration of temporal and spatial profile matching, combined with efficient patch-based parallel processing, marks a substantial advancement in the field. Detailed experimental validation further underscores realSEUDO’s superior performance, adaptability, and potential for scaling with faster imaging technologies.", "weaknesses": "While the paper presents a novel and efficient approach with realSEUDO, several limitations are apparent. The reliance on specific hardware configurations, such as Linux CPU performance modes, could limit accessibility and generalizability. Additionally, the implementation primarily in C++ with MATLAB wrappers, while effective, could benefit from broader integration into Python pipelines to enhance usability. The paper also assumes access to external motion correction algorithms and does not address the full pipeline holistically, which may impact its applicability in scenarios without these additional tools.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses: While the paper presents a novel and efficient approach with realSEUDO, several limitations are apparent. The reliance on specific hardware configurations, such as Linux CPU performance modes, could limit accessibility and generalizability. Additionally, the implementation primarily in C++ with MATLAB wrappers, while effective, could benefit from broader integration into Python pipelines to enhance usability. The paper also assumes access to external motion correction algorithms and does not address the full pipeline holistically, which may impact its applicability in scenarios without these additional tools.**\n\n**Our response:** We appreciate the reviewer’s positive comments and feedback. We agree that the additional limitations of the current infrastructure should be more explicit. Some drawbacks, such as additional wrappers in Python (as well as pure C++ integration into imaging pipelines) are a primary focus of future development in this project. We are also testing the exact limitations of the hardware impact on performance. We do believe that in the long-run CPU integration will allow for more generalizability than, e.g., GPU or FPGA approaches, and will add more text in the discussion to discuss the pros and cons of each. Finally, as online motion correction was well addressed by others (e.g., in the OnACID work) we instead focused on the less-considered cell finding. It is of course important, as the reviewer indicates, that full pipelines work well together and so integration of prior methods are a primary consideration as incorporate these pipelines into full closed-loop systems.  \n\n**Limitations: The paper presents notable advancements with realSEUDO but has some limitations. Firstly, the performance of the system is highly dependent on specific hardware configurations and the proper setting of Linux CPU manager modes, which may not be universally applicable. Additionally, the reliance on MATLAB for prototyping, combined with limited Python integration, may restrict its accessibility and ease of use for a broader audience. Moreover, the paper focuses on cell detection without fully integrating or addressing the motion correction and complete processing pipeline, which could limit its applicability in real-world scenarios where such components are not readily available.**\n\n**Our response:** We thank the reviewer for noting our advancements and seeing the potential in our work. As the reviewer mentioned, the primary goal of this work is to address a critical step in the pipeline for real-time cell detection. We agree that with every advance there will be limitations and that these limitations should be clearly delineated. Moreover we agree that the stage of the pipeline we address does not live in a vacuum and should be mentioned in the context of the other stages of the pipeline to best inform readers of when/how they can use the method. We will thus discuss in more detail the limitations raised by the reviewer and the steps we are actively taking to expand the ease of others to adopt and implement realSEUDO. These include additional integration tools into Python on the user’s end, as well as building in prior motion correction algorithms that will be a part of the complete pipeline development.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xrV0YDXCVA", "reviewer": "Reviewer_Xdei", "summary": "The authors present an online method for neuron profile and trace estimation in multi-photon calcium imaging. Their approach enhances the SEUDO algorithm by running it on a frame-by-frame basis and periodically updating profiles using a patching mechanism across space and time. The implementation leverages different degrees of stability, namely temporary and stable. The proposed method is benchmarked against both an online method and an offline method using simulated and real data.", "questions": "1. What are the key methodological differences between SEUDO and realSEUDO?\n2. How did you choose the benchmark datasets?\n3. Why correlation was selected as the main (and only) metric?\n4. How do the authors justify the lack of an ablation study on the main components of the work? Can the authors describe the quantitative improvement of their method compared to existing methods they rely on?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The manuscript provides extensive details about the implementation.\n- The impact of the proposed solution is clearly stated and is highly relevant for neuroscientists working with CI acquisitions, addressing several real issues such as empty initialization of profiles and scalability.", "weaknesses": "- The work largely relies on previously described algorithms like FISTA and SEUDO. Since a significant portion of the method describes these algorithms in detail, it is unclear what the key contributions of this work are.\n- The evaluation was conducted on synthetic data (Naomi) and a real dataset (36 videos). Including additional sources like 2p movies from the Allen Brain Observatory would have enhanced the credibility of the results.\n- The given metrics should be improved. While the images are good, providing tables with quantitative metrics can enhance the overall presentation and support the claims made. In addition to correlation, metrics like IoU could be evaluated on neuron profiles.\n- The experimental setting lacks an ablation study on the main components of the work, such as automatic cell recognition (e.g. with and w/o X_temp) and patching.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n1. **The work largely relies on previously described algorithms like FISTA and SEUDO. Since a significant portion of the method describes these algorithms in detail, it is unclear what the key contributions of this work are.**\n\n**Our response:** While we leveraged prior developments in the field, we would like to note that the ability to create a method that met the criteria for real-time demixing was not a trivial combination of these tools. Several significant changes, from efficient computation, implementation, and model modification had to work together in conjunction just to have SEUDO run in realtime. Moreover we developed a feedback loop for generating new ROIs (note that SEUDO only identifies time-traces given a-priori known ROIs) in an online manner.\n\n2. **The evaluation was conducted on synthetic data (Naomi) and a real dataset (36 videos). Including additional sources like 2p movies from the Allen Brain Observatory would have enhanced the credibility of the results.**\n\n**Our response:** While more results are always better, we carefully selected datasets that had reliable ground truth, either through data generation (e.g., the NAOMi simulation) or through rigorous prior assessment (e.g., the data from Gauthier et al. 2022). Other datasets do not have very reliable ground truth and thus would not serve as well to validate our approach. Moreover data from the Allen Brain Observatory is only disseminated as traces and ROIs, not as raw data. This is especially problematic as the SEUDO paper (Gauthier et al. 2022) previously showed that these are not reliable decompositions of the data.   \n\n3. **The given metrics should be improved. While the images are good, providing tables with quantitative metrics can enhance the overall presentation and support the claims made. In addition to correlation, metrics like IoU could be evaluated on neuron profiles.**\n\n**Our response:** We appreciate the suggestion. As the IoU implies that the full set of pixels identified is the important quantity, we instead compute the “unique neurons found”. This is since in functional imaging we instead typically need to know that the time-traces 1) correspond to real neurons in the data and 2) accurately reflect the temporal activity that will be used to study neural activity with respect to stimuli and behavior. The Unique Neurons Found requires both that ROIs well align with known ROIs spatially without the strict pixel counting that might not accurately reflect what a neuroscientist deems “success” in regards to cell finding. We will clarify our criteria for selecting these metrics and how the metrics used reflect these key requirements for validation in the results and discussion. We also selected to place the comparison as bar plots to situate them visually next to the images of the identified components and example time-traces, compartmentalizing that example. \n\n4. **The experimental setting lacks an ablation study on the main components of the work, such as automatic cell recognition (e.g. with and w/o X_temp) and patching.**\n\n**Our response:** We agree with the importance of ablation studies in algorithmic development, however the overall algorithm is quite simple. In fact directly adding ROIs to the dictionary instead of using X_temp is, as the reviewer saw, really the only section that can be ablated without effectively breaking the ROI selection loop. We have previously had ROIs added directly to the dictionary, which has resulted in a lot of noise being treated as signal. We did directly test an ablation study focused on the speed of computation, slowly adding every improvement. We noted the increasing framerate on the last paragraph before Section 3.2 on Page 5, which shows the necessity of each improvement to the overall performance.\n\n**Questions:**\n1. **What are the key methodological differences between SEUDO and realSEUDO?**\n\n**Our response:** There are differences in SEUDO and realSEUDO on both the computation and algorithmic levels. On the algorithmic level we introduced a full feedback loop that identifies ROIs. SEUDO, as derived and presented in Gauthier et al. (2022) is a time-trace estimation method only. It relies on pre-defined ROIs and focuses on robust estimation of the activity for each ROI. We add the ability to use the residuals from SEUDO to detect in an online manner new ROIs to iteratively build a dictionary of ROIs from nothing.  \n\n2. **How did you choose the benchmark datasets?** \nWe chose the benchmark datasets as datasets with extensive information (either through simulation in NAOMi data or manual expert annotation in the Gauthier et al. dataset) on both the ROIs and also the time-traces (e.g., our false positive comparisons). These properties gave us the best targets to validate our data quantitatively. \n\n3. **Why correlation was selected as the main (and only) metric?**\n \n**Our response:** Correlation as a metric was also predicated on ROI overlap. Essentially our “Unique cells detected” number combined both temporal correlation and spatial information to claim that we detected a cell well. We isolated the correlations into a separate histogram to highlight the success in identifying the time-traces; a quantity that is vital to the fitting of computational neuroscience models. \n\n4. **How do the authors justify the lack of an ablation study on the main components of the work? Can the authors describe the quantitative improvement of their method compared to existing methods they rely on?** \n\nIn terms of completely removing parts of our algorithm, the main parts are SEUDO and the feedback for specifying ROIs. Removing either would break the loop and not allow for the algorithm to be run at all. In terms of speed, we do show how each change we made improves speed (a type of ablation) showing that all different improvements we made are necessary to achieve the full speed-up. These numbers are detailed on the last paragraph before Section 3.2 on Page 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6fErZPbBY2", "reviewer": "Reviewer_uJzY", "summary": "In this study, the authors created a real-time version of the “Sparse Emulation of Unused Dictionary Objects” (SEUDO) algorithm termed realSEUDO, that can process multi-photon calcium imaging (CI) data in real time. The goal of this approach is to overcome the limitations of batch processing methods commonly employed in CI analysis by processing brain activity data in real-time, thus enabling closed-loop neuroscience investigations. The authors identified several places that the original SEUDO algorithm warranted improvements, implemented the changes and compared their method to other current online methods, with both simulated and real datasets.", "questions": "By facilitating the real-time processing of CI data, the paper shows the potential to improve the accuracy and efficiency of closed-loop studies. The claims of broad applicability would be strengthened by more extensive testing under various experimental situations and with diverse kinds of brain data. Additionally, it would be great if they could characterize and show the performance with different patch overlaps. Finally, while the paper discusses the integration with motion correction algorithms from OnACID, integration of all preprocessing steps such as motion correction within the realSEUDO pipeline would further widen its potential usability.", "limitations": "The authors have a limitation section that addresses potential issues with their algorithm.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This research presents a novel real-time CI data analysis method that is important for closed-loop investigations.  The realSEUDO algorithm as presented here, incorporates a significant number of optimizations over the original SEUDO, including a faster C++ implementation, parallel processing, and an efficient feedback loop for real-time cell identification that improves the computational load. In this way they can achieve much faster performance compared to other online methods (OnACID). Another important strength is its ability to match sources across non-overlapping parches, which reduces the computational load. The authors use both simulated and real datasets to evaluate the performance of their algorithm. This method could also become very useful in online-analysis of voltage traces that higher speeds can have even more benefits.", "weaknesses": "While the performance with a small number of neurons is impressive there are two drawbacks. \nFirst, the testing datasets only include a very limited number of neurons and the scalability with much larger datasets are not presented. For example, current methods such as with a mesoscope  (Sofroniew et al. 2016), allow for much larger recording samples in the of thousands of neurons. From the figure 3c we can see that the algorithm drops in performance quite fast and when much larger samples are included, it could potentially perform worse than existing methods, something that remains to be tested. \nLastly, calcium sensors already limit our ability to extract with fine temporal precision the neural activity thus the usefulness of a method that far exceeds that temporal resolution remains limited. \nAnother potential issue is to validate the model with more typical cortical recordings that have higher level of background fluorescence, and how it deals with overlapping sources of fluorescent signal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses: While the performance with a small number of neurons is impressive there are two drawbacks. First, the testing datasets only include a very limited number of neurons and the scalability with much larger datasets are not presented. For example, current methods such as with a mesoscope (Sofroniew et al. 2016), allow for much larger recording samples in the of thousands of neurons. From the figure 3c we can see that the algorithm drops in performance quite fast and when much larger samples are included, it could potentially perform worse than existing methods, something that remains to be tested. Lastly, calcium sensors already limit our ability to extract with fine temporal precision the neural activity thus the usefulness of a method that far exceeds that temporal resolution remains limited. Another potential issue is to validate the model with more typical cortical recordings that have higher level of background fluorescence, and how it deals with overlapping sources of fluorescent signal.**\n\n**Our response:** We thank the reviewer for noticing the positive aspects of our work and the novelty of our contribution. We would like to address the three main weaknesses that concern this reviewer: 1) scaling to very large imaging, 2) speed of calcium sensors and the need for even faster processing, and 3) the potential for overlapping sources of fluorescence.\n1. We agree that newer methods have been developed that aim to improve the number of sources that can be detected simultaneously. It is also the case that 500um-by-500um fields of view are still the standard for most labs and our patching approach easily extends to these sizes. We of course are interested in testing the boundaries of the capabilities of our method, and we will mention this limitation presented by the reviewer in the discussion and limitations section.\n2. The reviewer is right in mentioning the speeds of most calcium sensors. However we believe that two main points justify even faster processing. For one, new indicators such as GCaMP8f are even faster and may require more than 30Hz imaging to properly capture (and thus even faster processing). Second, the cell finding step is only one part of a closed loop system. The idea is to enable on-line experimental perturbations, which will also require online model fitting and the computation of the right perturbation (e.g., stimulation or experimental change) after each cell’s activity is demixed by realSEUDO. Thus if realSEUDO takes even less time, there is more time for later analysis steps that will enable true closed-loop experiments\n3. In terms of overlapping sources of fluorescence, the reviewer is correct that this is a critical issue, especially in on-line demixing where sources of fluorescence are unknown. Thus, a key design choice in selecting SEUDO as the core of our approach was its ability to be robust to overlapping sources of fluorescence even when one source was unknown. The original SEUDO paper, Gauthier et al. 2022, demonstrated this ability, which we further demonstrate remains present in realSEUDO by counting the number of ``false transients’’ (i.e., bursts of activity due to bleedthrough from other sources rather than the cell itself) in Section 4, bottom of Page 7. Specifically, realSEUDO can continue to remove false bleedthrough activity despite the many overlapping sources of fluorescence in the CA1 tissue that we tested realSEUDO on. \n\n**Questions: By facilitating the real-time processing of CI data, the paper shows the potential to improve the accuracy and efficiency of closed-loop studies. The claims of broad applicability would be strengthened by more extensive testing under various experimental situations and with diverse kinds of brain data. Additionally, it would be great if they could characterize and show the performance with different patch overlaps. Finally, while the paper discusses the integration with motion correction algorithms from OnACID, integration of all preprocessing steps such as motion correction within the realSEUDO pipeline would further widen its potential usability.**\n\n**Our response:** Again we appreciate the reviewer’s positive outlook on this work. We aimed to focus our validation on datasets with rigorous ground truth, which included real data from Hippocampus and Visual cortex, along with the realistic simulations. As the Hippocampus data had much richer validation data (i.e., it was previously annotated by the authors of the study not just in terms of cells but also time-traces), we focused on this dataset for our work. We will mention these aspects of our study in more detail in the discussion to clarify. Moreover we will make mention of the need to integrate out advance, which focused on one critical step of the pipeline, with other steps such as motion correction that have been addressed by other researchers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dz9ecd9JR6", "reviewer": "Reviewer_2uGa", "summary": "The authors extend an available algorithm called SEUDO for extracting time traces from two-photon imaging data robustly to the real-time close-loop setting. The algorithm can simultaneously extract ROIs and time traces. The algorithm performs on par with state-of-the-art offline algorithms and better than state-of-the-art online algorithms.", "questions": "* A lot of engineering work has gone into the fast implementation of the online algorithm using C++ and various bells and whistles. Have the authors considered a GPU based implementation e.g. in jax and how would that compare in performance?\n* Where does equation 7 come from? Min operations are quite sensitive to outliers. The details won’t matter, but in the spike sorting literature, a threshold of n*0.6745(median (abs(x))) is often used for detecting spikes, which is indeed a multiple of a robust estimate of the noise sd without the spikes.\n* Why was the tool not compared to the other online algorithm FIOLA?\n* Why was the plain CNMF algorithm used and not one of the more advanced implementations/methods or an additional algorithm?\n* Doesn’t Fig. 2C speak very clearly for the results of the OnACID algorithm?\n* Lines 348: Why do the authors say that a c++ implementation makes the code open source?", "limitations": "Adequately adressed.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written and the motivation is clear and the summary of the literature concise and compelling.\n* While the algorithmic components of the work are not new in themselves, they are combined in a convincing way to solve the problem laid out.\n* The speed of the algorithm is impressive, at satisfying performance compared to the SOTA.", "weaknesses": "* The main improvement come from engineering the implementation of SEUDO such as to run in close to real time (is that a weakness or strength?).\n* Lines 228: The merging procedure seems quite ad-hoc and heuristic. The logic is only laid out at the end (lines 247), maybe it would help to discuss this earlier.\n* Only two algorithms are chosen for comparison. \n* Fig. 2B is very hard to read and understand. Does this mean, OnACID produces no false alarms and no ambiguous units?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n1. **The main improvement come from engineering the implementation of SEUDO such as to run in close to real time (is that a weakness or strength?).**\n\n**Our response:** Thank you, and we believe that our improvements represent a strength of our approach as we are able to successfully improve and implement a previously slow and unyieldly algorithm in real-time. \n\n2. **Lines 228: The merging procedure seems quite ad-hoc and heuristic. The logic is only laid out at the end (lines 247), maybe it would help to discuss this earlier.**\n\n**Our response:** We appreciate this suggestion and agree that this section can be improved. In the revision we will begin this section with the logic of our merging procedure to clarify the steps we take. \n\n3. **Only two algorithms are chosen for comparison.**\n\n**Our response:** In our work we selected the most relevant algorithms. As a baseline we selected CNMF as a standard, SOTA, well-maintained method representative of off-line methods. Comparing to other off-line methods we did not believe to be relevant given that we expect them all to perform at or below the level of CNMF (see, e.g., Song et al. 2021 and Charles et al. 2022 for comparisons between offline methods). OnACID currently represented the only CPU-based online method in existence and so our comparisons had to be to it.\n   \n4. **Fig. 2B is very hard to read and understand. Does this mean, OnACID produces no false alarms and no ambiguous units?**\n\n**Our response:** Yes, that is correct. In the NAOMi data, under the definition from Song et al. 2021 of a found cell (spatial overlap of over 50% and temporal correlation >0.5) OnACID performed very conservatively in the best parameter set we could find for it. This meant that it actually reduced quite a bit of noise as well as true transients. Thus the correlations are overall higher, however the total number of cells was reduced. Additionally, OnACID is still very slow (<10 Hz) meaning that it is not truly \"real-time\" for this data. \n \n**Questions:**\n\n1. **A lot of engineering work has gone into the fast implementation of the online algorithm using C++ and various bells and whistles. Have the authors considered a GPU based implementation e.g. in jax and how would that compare in performance?**\n\n**Our response:** We had considered both GPU and FPGAs as alternative approaches that could have been viable. While potentially beneficial in some ways, our ultimate goal was integration of our online cell-finding into a larger closed-loop pipeline. In this case bringing the pipeline closer to the imaging is necessary, and most imaging groups don’t have high performance GPUs or the expertise to install and make use of FPGAs. We thus decided that a purely CPU approach would reduce the hardware complexity in the long term as we work with experimental groups to implement this method in closed loop systems.\n\n2. **Where does equation 7 come from? Min operations are quite sensitive to outliers. The details won’t matter, but in the spike sorting literature, a threshold of n*0.6745(median (abs(x))) is often used for detecting spikes, which is indeed a multiple of a robust estimate of the noise sd without the spikes.**\n\n**Our response:** Equation (7) simply comes from the desire to assess the width of the pixel histogram mode (the lower half of the histogram as there is usually an asymmetric tail due to active pixels). It is true that the min is usually sensitive to noise, however we actually compute this min on the denoised data which provides a more stable minimum value. We agree that this is unclear in the paper and will clarify this detail in the text. \n\n3. **Why was the tool not compared to the other online algorithm FIOLA?**\n\n**Our response:** FIOLA is an online algorithm for demixing, and not cell identification. Specifically it is more of an optimized pipeline that relies on pre-computed ROIs to extract time-traces from online. The authors mention this in the FIOLA paper (page 1420). Thus we did not feel it is an appropriate comparison point. In fact we can envision parts of the realSEUDO concept being integrated into their very excellently engineered pipeline that solves other important problems, such as motion correction and spike extraction.\n\n4. **Why was the plain CNMF algorithm used and not one of the more advanced implementations/methods or an additional algorithm?**\n\n**Our response:** We specified CNMF, however used the most recent version available in the CaImAn package provided by the Simons institute. We felt that it represented one of the state-of-the-art methods and was related to OnACID which gave a good complementary point of reference of what the same algorithm could look like on-line and off-line.   \n\n5. **Doesn’t Fig. 2C speak very clearly for the results of the OnACID algorithm?**\n\n**Our response:** OnACID had varying results, performing better in the NAOMi simulated data, for example, than the CA1 real dataset. Specifically we found that OnACID performed quite well in the simulated NOAMi data under conservative parameters. This is primarily due to the high thresholding value that both reduced noise in the estimated time-traces. However this higher threshold also removed smaller transients (see e.g., Profile 2 in Fig. 2D). Moreover OnACID was considerably slower and unable to run at real-time speeds (<10 Hz as opposed to realSEUDO's >65 Hz, see Fig. 2C).\n  \n6. **Lines 348: Why do the authors say that a c++ implementation makes the code open source?**\n\n**Our response:** We simply wanted to make sure the community understood that the code was not written completely in MATLAB, which is not technically open source even when released. The C++ code has a wrapper in MATLAB, and we are working on similar wrappers for Python. We will clarify this point in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time and effort in reviewing our manuscript. We are encouraged that the reviewers recognized the relevance, capabilities, and potential impact of our work. We take the time here to respond to the primary concerns the reviewers raised, with more detailed responses to each reviewer.
1. **Novelty of approach:** Reviewers were mixed in their assessment of novelty of the approach. We would like to take the time to clarify that while we did leverage prior developments in the field, specifically FISTA and SEUDO, we significantly improved these algorithms at multiple levels to create a new method that met the criteria for real-time demixing. In addition, none of these methods has a cell-finding feature, and so the individual improvements (efficient computation, implementation, and model modification), the combination, and the creation of the feedback loop to turn fast time-trace estimation into fast cell finding we believe was not a trivial combination of these tools.
2. **Integration into more general pipelines:** Reviewers 86dC and uJzY asked about further integration into current pipelines (e.g., including the motion correction and having additional wrappers). We agree, and are working on complete integration. As online motion correction already exists, this work focuses on a different critical piece of the real-time analysis puzzle: cell detection. While some engineering work will be required to connect these pieces, we felt that the creation of a non-existent piece should be the main focus of this paper. Similarly, we created a wrapper for our C++ code in MATLAB for easy testing, and will be developing similar Python scripts (or even moving the whole pipeline into C++) to best reach a wider audience. We will discuss these additional pieces and the future steps to integrate realSEUDO into full pipelines in the limitations and discussion sections.     
3. **Choice of datasets:** Reviewer Xdei wondered about our choice in validation data and if this presented a limitation in our work. Our primary goal in dataset selection was to choose datasets that can validate the estimated time-traces which are vital for to model neural activity and relate it to behavior. Specifically, time-traces are also the most prone to errors in real-time cell finding as unknown cells can bleed into known cells, creating false activity. Both the NAOMi simulation (Song et al. 2021) and the SEUDO dataset  (Gauthier et al. 2022) have both spatial and temporal annotation. NAOMi provides visual-cortex like data with ground truth traces and the SEUDO data provides dense CA1 data with true/false annotations for individual activity bursts. We further analyzed the third dataset included in the OnACID package, for which we could only assess spatial ROIs.
4. **Quantification and metrics:** Reviewer Xdei mentioned other potential metrics, such as the intersection-over-union metric. We agree with the reviewer that assessing spatial matches is important and we wish to clarify that while we did not use the IoU to validate our approach, we did use spatial information in our “unique cells detected” (Fig. 2B) which considers both spatial overlap and temporal correlations of the time-traces to identify if a found component is a true positive. We find that integrating the spatial and temporal information together into a hit is overall more informative in functional fluorescence microscopy as the exact pixel count is less important than cell ID (i.e., overall location) and time-trace accuracy. 
5. **Choice of algorithm comparisons:** Reviewer 2uGa wonders about the selection of comparison algorithms. There are very few comparison points for online processing of calcium imaging data. OnACID represents the primary method available to the community. Other methods, such as FIOLA are more of a pipeline and actually rely on pre-computed ROIs for fast time-trace estimation. We thus compared to OnACID, which while accurate at times can be quite slow. We also compared realSEUDO to a state-of-the-art off-line method CNMF (using the latest version available through Simons) to show how close (or far) on-line methods are as compared to off-line methods. Many off-line methods exist, and other works compare off-line methods (e.g., Song et al. 2021 and Charles et al. 2022) which informed our decision of selecting one of the consistently highest performing off-line algorithms.

Meta-review: This paper extends a previously developed algorithm called SEUDO for extracting time traces from two-photon imaging data robustly also in challenging real-time close-loop setting. The algorithm can simultaneously extract ROIs and time traces. The algorithm performs competitively with state-of-the-art offline algorithms and better than state-of-the-art online algorithms. The results and concepts are well explained, broadly accessible and potentially very impactful for closed-loop neuroscience experiments.
Decision: Accept (poster)