ActSort: An active-learning accelerated cell sorting algorithm for large-scale calcium imaging datasets
Authors: Yiqi Jiang
Hakki Orhun Akengin
Ji Zhou
Mehmet Anil Aslihak
Yang Li
Radoslaw Chrapkiewicz
Oscar Hernandez
Sadegh Ebrahimi
Omar Jaidar
Yanping Zhang
Hakan Inan
Christopher Miranda
Fatih Dinc
Marta Blanco-Pozo
Mark Schnitzer
Venue: NeurIPS 2024 poster (2024)
Keywords: neuroscience
active learning
calcium imaging
cell sorting
Primary area: neuroscience_and_cognitive_science

Abstract: Recent advances in calcium imaging enable simultaneous recordings of up to a million neurons in behaving animals, producing datasets of unprecedented scales. Although individual neurons and their activity traces can be extracted from these videos with automated algorithms, the results often require human curation to remove false positives, a laborious process called \emph{cell sorting}. To address this challenge, we introduce ActSort, an active-learning algorithm for sorting large-scale datasets that integrates features engineered by domain experts together with data formats with minimal memory requirements. By strategically bringing outlier cell candidates near the decision boundary up for annotation, ActSort reduces human labor to about 1–3\% of cell candidates and improves curation accuracy by mitigating annotator bias. To facilitate the algorithm's widespread adoption among experimental neuroscientists, we created a user-friendly software and conducted a first-of-its-kind benchmarking study involving about 160,000 annotations.  Our tests validated ActSort's performance across different experimental conditions and datasets from multiple animals.  Overall, ActSort addresses a crucial bottleneck in processing large-scale calcium videos of neural activity and thereby facilitates systems neuroscience experiments at previously inaccessible scales. (\url{https://github.com/schnitzer-lab/ActSort-public})

ActSort: An active-learning accelerated cell sorting algorithm for large-scale calcium imaging datasets
Yiqi Jiang" Stanford
Hakki O. Akengin
Zhou', Johns Hopkins
Stanford
Mehmet A. Aslihak
Yang Li Stanford
Radostaw Chrapkiewicz Stanford
Oscar Hernandez
UC San Diego
Stanford
Sadegh Ebrahimi
Omar Jaidar
Yanping Zhang HHMI, Stanford
Hakan Inan
Stanford
Stanford
Meta
Christopher Miranda
Fatih Dine
Marta Blanco-Pozo" Stanford
Mark J. Schnitzer HHMI, Stanford
Stanford
Stanford
Abstract
Recent advances in calcium imaging enable simultaneous recordings up to a million neurons in behaving animals, producing datasets of unprecedented scales. Although individual neurons and their activity traces can be extracted from these videos with automated algorithms, the results often require human curation to remove false positives, laborious process called cell sorting. To address this challenge, we introduce ActSort, an active active-learning algorithm for sorting large- scale datasets, which integrates features engineered by domain experts together with data formats with minimal memory requirements. By strategically bringing outlier cell candidates near the decision boundary up for annotation, ActSort reduces human labor to about 1-3% of cell candidates and improves curation accuracy by mitigating annotator bias. To facilitate the algorithm's 's widespread adoption among created user-friendly software and conducted
experimental neuroscientists, a first-of-its- tind benchmarking study involving about 160,000 annotations. Our tests validated ActSort's performance across different experimental conditions and datasets from multiple animals. Overall, ActSort addresses a crucial bottleneck in processing large-scale calcium videos of neural activity and thereby facilitates systems neuroscience experiments at previously inaccessible scales. (Link to our Github: https:://// https:://githbb cotm///ttniizzerraaaaAAffffoorpp
Introduction
Systems neuroscience research has been revolutionized by large-scale Ca imaging techniques, which now regularly produce terabyte scale datasets 2
g 9 This rapid increase in neural data has necessitated automated methods for identifying neuronal sources, leading to the development of cell extraction algorithms
[IO
13 These algorithms facilitate large-
scale analysis by detecting even to a million neurons
g 14 15). However, they can sometimes
misidentify non-neuronal elements neuronal structures, potentially leading erroneous biological
interpretations in downstream analyses [16] 17 [8].
To mitigate errors arising in automated cell extraction pipelines, experimental studies often perform quality control processes, known as cell sorting, to validate and refine cell extraction outputs [10 |1
"These authors contributed equally to this work.
These authors co- ro-supervised this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

12 13 19.. In practice, researchers either laboriously annotate all cell candidates using specialized software (See CIATAH, (20) or employ "cell classifiers" that are trained on subset of annotated datasets to label the rest I00
20 However, manual annotation is no longer practical given the large data sizes modern neuroscience [8] 14
15 21 22 As we show in this work, existing automated methods still require substantial human labor to achieve desirable accuracy. Yet, human annotations are known to develop undesirable biases ([23) 24 25 26 (27) due to, e,g., getting tired after long hours of repetitive work. Therefore, there is an urgent need for a scalable and robust solution that minimizes human labor without compromising the accuracy of the cell sorting.
A potential solution may lie in the active learning paradigm, a branch of machine learning that optimizes model performance while minimizing the need for annotated samples (28 29
30 31 32
33 34
In this paradigm, so-called query algorithms select the most informative and sparse samples to be labeled by human annotators [36 37 38) To date, active learning has improved several scientific pipelines such as classifying pathological images (39) 40), detecting intracranial hemorrhage [41], and segmenting melanoma [42]. Inspired by these successes, we set out to leverage the active learning framework to accelerate the cell sorting process.
35
In this work, we introduce ActSort: an active learning accelerated cell sorting algorithm to process large-scale Ca imaging datasets. As first step, we developed a memory-efficien and user-friendly cell annotation software. Using this software, we designed an extensive cell annotation benchmark. This benchmark contains roughly 160,000 annotations of 40,000 cells collected using one- and two-photon mesoscopes across 5 mice, which we used to test and validate ActSort. Supported with novel features engineered by expert neuroscientists and a novel query algorithm tailored for cell sorting, ActSort matched human-level accuracies while reducing the required human labor to a negligible fraction of the full samples (down to 3% when training from seratch, and 1% if ActSort is initialized with previously annotated samples). In large-scale mesoscope movies, ActSort achieved convergence by sorting few hundreds of cells (c.a. hour of human labor). In contrast, to achieve the same mark, labeling random subsets of representative datasets required sorting an order of magnitude larger number of cells. Notably, our analysis also revealed significant disagreements among human annotators, yet autimautomated classifiers mitigated this large variability across experimenters. Overall, our findings suggest that ActSort can save experimentalists significant time, and improve the reliability, robustness, standardization, and reproducibility of systems neuroscience research.
2 Results
Our cell sorting pipeline (ActSort) consists of three main components (Fig. 1 the preprocessing module, the graphical user interface (GUT), and the active learning module. The full pipeline is available to end-users with a point-click installation in our GitHub repository as an application or in Matlab-online without a license. In this section, we introduce ActSort and validate it with several experiments on large-scale annotation benchmarks.
4 and can be used
2.1 A large scale benchmark of 160,000 annotations sorted with a custom software
Only five years ago, Ca imaging movie with 100 GB size was considered large [12). Today, neuroscientists routinely collect datasets several terabytes in size [4]. Given this rapid increase, we start our work by designing a scalable software with primary focus on memory efficiency. Without keeping the full movie in RAM, the software needs to include essential information used by human annotators to determine whether a sample a cell: i) the shapes of cells' extracted spatial footprints, ii) their C activity traces over time, and iii) the movie snapshots at the time of Ca²+ events (Fig. ST. Appendix BB). While earlier work often retained only the first two elements for memory efficiency 10.. we found that the spatiotemporal information provided by the Ca²+ movie snapshots was crucial for human annotators, especially in edge cases, which are particularly important for the active learning paradigm (see below). This design choice also aligns well with the newer Ca²+ imaging processing pipelines (see the cell sorting module in CIATAH (20), and can be incorporated with an efficient data compression procedure without increasing dataset sizes significantly.
To achieve this design goal, as part of the preprocessing module (Fig. SSp. we opted to compress the movie and the cell extraction results jointly. We saved the variables in sparse formats whenever applicable and retained only the relevant parts of the Ca imaging movies where the cells had Ca events (up to pre-defined maximum; see Appendix [. for details). This approach allowed us to compress the large inputs of the traditional cell sorting pipelines (Fig. [. resulting in approximately 270 + 90 MB/1,000 cells (mean + std over 5 mice) data sizes. Thanks to the extreme compression

---

Inputs
Active learning accelerated cell sorting
Outputs Labeled cells
Ca" imaging movie
Selection GUI
mm
Preprocessing Module
Cell candidates cell extraction algorithi
Active Learning
An active learning accelerated framework for rapid quality control of cell extraction results. Left. A typical processing of Ca imaging movies involves identification of cell candidates by automated cell extraction algorithms. Middle. ActSort's preprocessing module jointly compresses the cell extraction results and the Ca imaging movie to allow memory efficient cell sorting in large-scale Ca imaging movies, and computes quality metries for each cell candidate. The selection GUI allows human annotators to visually inspect the cell extraction results and annotate the true and false positive cell candidates. The active learning module, working in closed-loop feedback with the selection GUI, strategically selects next cell candidates to be annotated by the experimenter in order to optimally reduce the human effort. Right. The resulting outputs from ActSort are probabilities and binary labels (cell not cell) for each cell candidate.
Figure I:
of the movie and cell extraction data, we were able to develop a memory efficient and user-frient -rieenlyyl graphical user interface for cell sorting (Figs. ST] S2 and [S3). This software does not require any coding knowledge, utilizes buttons and sliders to operate, and can be used on laptops.
Most public Ca imaging datasets share cells' spatial profiles and Ca2+ activity traces, but not the Yet, motion artifacts and/or neuropil contamination can often not be identified by simply looking at Ca activity traces, but can alter biological conclusions [13 18). Thanks to our memory memory-effiiient data format, open-sourced datasets can now include relevant movie snapshots without astronomical increases in data sizes, and the quality of the Ca activity traces can be probed with our software.
movies themselves due to large data sizes 45 46
After coding the cell sorting software, we first searched for public datasets to validate the use of cell classifiers and active learning paradigm as a viable method for automated cell sorting. Although there are publicly available Ca2+ imaging movies with annotated cells, notably NeuroFinder challenge [7]. these datasets are often region specific and contain few hundreds of cells. Yet, we have designed ActSort to process large-scale mesoscope movies. These modern movies simultaneously record large populations of neurons across multiple brain regions, and are now regularly collected by neuroscience labs 14 44. To the best of our knowledge, no annotation dataset is publicly available with brain-wide wide mesoscope movies.
To bridge this gap. we curated a total five mesoscope movies: i) three one-photon Ca2+ imaging movies spanning the hemisphere through 7mm window (Table S1b, , ii) one Ca²+ imaging movie from previous work (144)) spanning several neocortical regions (Table S2). and iii) one two-photon Ca imaging movie capturing layer 2/3 cortical pyramidal neurons in mice primary visual cortex (Table (S3). We extracted an approximate total of 40,000 cells from these movies using the EXTRACT cell extraction algorithm [ We then had 6 annotators independently perform cell sorting these movies, with 4 annotators per movie, totaling up to roughly 160,000 annotations (Methods; Appendix C.. We used this benchmark to perform validation experiments throughout this work.
2.2 Automated cell sorting with linear classifiers utilizing newly engineered features
Automated cell sorting pipelines rely on training cell classifiers annotate the unlabeled cell candidates [10 20 44 These classifiers often use features derived from the cell extraction outputs and, in some cases, from Ca imaging movies (2). The central hypothesis behind this automation is that these derived features- such as the stereotypical shapes of cells' spatial profiles croflls--ootain sufficient information to distinguish false positives from real cells. However, relying on a select few features overlooks crucial aspect of cell sorting: human annotators incorporate information from broad sources their decisions. Consequently, we hypothesized that cell classifiers would benefit from broader class of features than those traditionally implemented in the literature.
contain
3

---

Figure 2: Feature engineering improves the accuracy of automated cell sorting with linear classifiers. A The newly engineered features increased classifiers' ability to reject false- possse-ossvvees and led to improved area under the receiver operat- ing characteristic curve (Wilcoxon signed- rank tests ( (""p P < 10- B We com- puted the discriminability indices (d') that quantify how well each feature can sepa- rate out cells from false- paise-positives. C Many features exhibited middle or strong effects (d' 2 0.5 d' 2 0.8, respectively), whereas few features were not discrimi- native for this particular dataset. Boxes span the 25th to 75th percentiles, whiskers 1.5 times the inter-quartile range.
Ap
Accuracy of matching annotator's cell rajection
Area under the receivar operating curve
Call candidate density (a.u.)
Cells False positives
proference 0.5
2a =o²
Au
O
Traditional features only All engineered features
Feature value (a.u.)
Absolute discriminaabllyy index (ld%)
Temporal features
Spatial foatures Spatiotemporal features
To test this hypothesis, we first designed traditional feature set (Methods; Appendix B.I). which aims to be a strong baseline representing the set of features used by existing cell extraction pipelines [O. 12 13 20 44). With these features, we processed hemisphere Ca imaging movies from three mice, encompassing 28,010 cell candidates and 112000 annotations (Table ST For each mouse, we trained cell classifiers using traditional features on half of the cell candidates and predicted the identity of the other half. While the classifiers performed superbly in identifying true positives, correctly accepting 97% of the cells (data not shown), they were suboptimal in rejecting false positives, accurately rejecting only 67% of the candidates rejected by human annotators (Fig. AA). Part of this low accuracy may be due to the fact that only a small fraction of the samples (often <10%) are false positives, and human annotators may struggle to remain consistent with their strategy for duplicate selection and/or when deciding edge cases. Nevertheless, this observation was in line with our prediction that there may be room for improvements in the feature sets used by cell classifiers.
To bridge this gap, we engineered 76 new features for the cell classifiers by mimicking the spa- tiotemporal information human annotators pay attention to during cell sorting (see Appendix [B.] for the full list). This set includes traditional temporal features (c.g.. number of spikes in a trace, signal-to-note ratio) and spatial features (c.g., mean pixel intensity, circumference), as well as novel spatiotemporal features (c.g., severity of the non-Gaussian noise contamination). To ensure -ffforr-effiiient computation, we derived these features solely from the compressed data format, allowing feature extraction on demand during the cell extraction routine. Re-analyzing the dataset from Fig. 2A we found that the new features increased the rejection accuracy to 72% (Fig. ( without decreasing the accuracy of accepting true true-positives (data not shown, 97%), leading to more effective separation between cell and not cell samples in our benchmarks (Fig. BCC). We also confirmed that this newly engineered set of features were more efficient and effective than features Appendix ....
extracted in an unsupervised manner from a ResNet-50 model (Fig. S4
The question remains, however, how can we improve the cell classifiers further? We conjectured that one of the reason behind the relatively low rejection accuracy (72% with the new features) may be the inconsistency of human annotators, and thereby the imperfect "ground truth" evaluators. We studied this question extensively in Appendix A.2 which revealed that individual human annotators were indeed quite inconsistent. Though, cell classifiers trained on their annotations ended up being more consistent (Fig. S5) Yet. if human annotators are inconsistent, how can we validate the accuracy of ActSort, or in general, automated cell sorting? To address this, we designed a cross-validated evaluation process that incorporated the f'ct that humans often disagree with each other (Fig. [S6). Specifically, to evaluate the active learning routines trained with a specific annotator, we created ground truth evaluator by taking the majority vote of the three held-out annotators. Fortunately, the majority votes among the held-out human annotators had a good consistency quantified with the intra class correlation (ICC) analysis (JCC- = 0.79 + 0.05, mean + std, two-way mixed, average measures [48). far outperforming the individual annotators (ICC 0.56 + 0.06, mean + std, two-way mixed, single rater ICC). Thus, throughout this work, the annotations performed by either the classifiers and/or the experimenters are evaluated on these majority-voted held-out labels.

---

2.3 Disriinnnatie-confidnee active learning query algorithm for cell sorting
So far, we introduced our custom software for cell sorting, discussed how to utilize feature engineering and cell classifiers to annotate unlabelled cell candidates, and established how to reliably and consistently evaluate the cell sorting results via majority votes. Now, we introduce the final piece: integrating the cell classifiers into the active learning framework with "query algorithms."
In ActSort, the active learning query algorithm prompts human annotators to label specifically selected samples. These samples are expected to improve the automated cell sorting accuracy, which creates an online feedback loop between the annotator and classifier (Fig. I. But how should the samples be chosen for annotation? A straightforward approach is to annotate random subset and Yet, as we discuss below, better
label the rest, as has been done in the literature to date 10 20 alternatives exist.
When designing alternative query algorithms, we focused primarily on speed and scalability, as sorting is performed on the spot and within seconds. Therefore, we considered query algorithms that can train and predict rapidly. Two such strategies are confidence-
active learning (CAL) 49 (5) and discriminative active learning (DAL) [51]. These query algorithms aim mitigate two distinct types of uncertainties: (i) uncertainty regarding the sample's position relative to the decision boundary of the current model (CAL), and (īi) uncertainty regarding whether labeled samples faithfully represent the full dataset (DAL), Both query algorithms achieve these aims with rapidly trainable linear classifiers (see Methods for details; Appendix B.3
Briefly, CAL trains a cell classifier on the currently labeled samples and picks the unlabeled sample closest to the decision boundary for annotation. DAL trains a secondary "label" classifier to discrim- inate labeled cells from the unlabeled data and selects a sample most different from those labeled subset. Evidently, an optimal solution should bring samples up for annotation that are close to the final decision boundary (CAL). However, the query algorithm should approximate where the final boundary is with sparse samples, which requires collecting diverse samples that represent, roughly, the full dataset (DAL). Therefore, both approaches have their unique advantages.
To amplify the strengths of both approaches, we introduce new query algorithm: discriminative- confidence active learning (DCAL). DCAL selects samples for human annotation by considering both the classifier's uncertainty and the representation of samples in the unlabeled dataset. This achieved by training both cell and label classifiers and adaptively combining their scores with a user-initialized weight w (Methods: Appendix B..)). As the number of samples increases, the weight is adjusted based on the accuracy of the label classifiers. Specifically, DCAL gradually approximates CAL, since the label classifiers tend to become random predictors due to the growing diversity of annotated samples, Meanwhile, the DAL component is crucial in the early stages to identify outlier data points (Fig. S7A) Thanks to the adaptive estimation process, DCAL depends minimally on the initial choice of the trade-off weight (Fig. B), and is thereby practically parameter free.
2.4 Geometrical interpretation of discriminatiiv--oniidee
queries
The query algorithms we consider in this work - random, CAI., DAL., and DCAI, - have simple geometric interpretations, which we discuss
using example movie from our benchmark (Figs. 3 and (8). To illustrate the sample selection preferences by these query algorithms, we reduced the dimension of the feature space from 76 down to 2 using partial-least squares analysis and plotted in Figs. and S8
Both random and DAL queries comprised of diverse samples covering the full spectrum, with the latter showing increased diversity as expected. As a consequence, since there are for more true positive samples in the full dataset, both of these approaches picked mostly real cells up for annotation (Fig. A and random and DAI.). In contrast, CAL primarily picked cell candidates that are close to the decision boundary, Icading to more balanced selection of true and false positive annotations (Fig. 3 SS) CAL). Yet, especially for sparse annotations, CAL had a preference to sample from higher density regions of the boundary (Fig. S8) CAL with 1%). DCAL, on the other hand, was practically equivalent to CAL in large annotation limit, but also mitigated the dense sampling with sparse annotation (Fig. AA and and DCAL). In other words, DCAL picked outlier boundary cells, providing a comprehensive coverage across the boundary.
Next, we quantified these observations by focusing on subset of the dataset, which we termed "boundary samples" (Methods; Appendix [D.4). Here, we confirmed that both CAL and DCAL selected a high percentage of boundary samples (Fig. 3 B). These selections preferentially included
5

---

A ground truth
random
CAL
DAL
DCAL-0.3
DCAL-0.5
DCAL-0.7
DCAL-0.9
PLS reduced dimensions
not cell
cell
Average fraction of selected
candidates on boundary
0.8
selected agreement type
0.2
Average cosine distance
between features of the
annotated boundary cells
Average fraction of
random
CAL
DAL
DCAL-0.3
DCAL-0.5
DCAL-0.7
DCAL-0.9
randomm
CAL
DAL
DCAL 0.3
DCAL- 0.5
DCAL-0. 0.7
DCAL-0.9
random
CAL
DAL DCAL-03
DCAL-C 0.5
DCAL-0.7
DCAL-0.9
4-0 vote
3-1 vote
2-2 vote
active learning (DCAL) algorithm selects outlier boundary samples. A Dimension reduction via supervised partial least squares (PLS) was applied to the 9,983 cell candidates from representative annotation of one hemisphere dataset. For each query algorithm, we sorted up to 3% cell candidates and visualized them in the PISSreduccd dimensions computed from the majority- positives; dashed black lines, (approximate) decision boundaries the reduced feature space. The fraction next DCAL indicates its user-defined initial weight. B The average percentage of the selected samples near the decision boundary (probabilities within [0.25, 0.75]. Method; Appendix D.4) when sorting 3% of cell candidates with each algorithm. C The fraction of cell candidate types, based on annotators' votes (0:4/4:0, 1:3/3:1, and 2:2), selected by each query algorithm normalized with respect the total number in that particular confidence pool. The average cosine distance between standardized features of boundary samples, the same samples that lie within the decision boundary (B). In (B-D), each dot represents a single annotation instance. Error bars: s.c.m. over 12 annotations. All tests are two-sided Wilcoxon signed-rank tests with Bonferroni-Holm corrections ("*"p < 10-3 "*p < 10 10-2, < 0.05).
Figure 3: Discriminative-confii
held-out labels. Blue dots, cells (as annotated by human); red dots, false
samples that were challenging for human annotators to agree on (Fig. BC). In selecting these samples, DCAL has shown significant improvements over DAL (Fig. 3 B,C), which was expected since the latter has no relevant incentive. Notably, DCAL also matched the performance of CAL in picking boundary samples, an algorithm solely designed to do so, and had improved diversity selected boundary cells (Fig- FA, D). Thereby, these results jointly confirmed that DCAL preferentially picked samples lying on the decision boundary and those further from previously picked samples, leading to broader coverage and better representation of the boundary (Figs. and S8).
2.5 Performance evaluations of ActSort on the human annotated benchmarks
Though automated cell sorting is necessity, work to date has not performed a clear validation of such approaches large scale benchmarks. Similarly, though active learning query algorithms may provide improvements, existing approaches to date have focused on simple random sampling strategies. To perform the first test of this paradigm, we ran several experiments on our benchmarks using the cross-vale validated evaluation process (Fig. (56A). Specifically, learning query algorithms (random, CAL, DAL, and DCAL) with closed-loop simulated experiments, which mimicked the real-time annotation that ActSort aims to facilitate (Fig. 4A).
tested the cell classifiers and the active
As first test, we processed individual 2+ imaging movies covering hemisphere from three distinct mice (Table and negative rates (Fig. 4D), whereas all algorithms except DAL and random sampling had an above human level true positive rate of > 95% with < 1% annotated samples (Fig. 4C). As expected, DCAL's performance was mostly independent from the chosen weight due to the adaptive estimation
and Fig. B-D). With as little as 5% annotations, DCAL achieved human level true
process (Appendix B.3) but also distinct (and better) compared to either DAL or CAL. Notably,

---

A
Labelod and or unlabeled cell data
Unlabeled cell data
Feature Computing
Select oell next labeling
Query Algorithm
Automated cell labeling
Spatial Features
Temporal Features
Spatiotemporal Features!
Closed-loop
enforce stopping
Evaluation
stop
GUI Computing
Spatial Dala
Temporal Data
Individual
Cell Classifier
Cell
Snapahots Data
Train L
Active learning on individual datasets
B
0.9
C
D
0.8
Balanced Accuracy
True Positive Rate
True Negative Rate
0.8
0.85
0.9
0.97
0.7
0.75
0.6
0.6
0.8
0.8
0.92
0
0.6
0.5
0.5
50
Percentage of human annotated cells
Active learning combined dataset
G
0.8
Balanced Accuracy
True Positive Rate
True Negative Rate
0.8
0.6
0.7
0.7
0.8
0.1
0.6
0.6
0.7
0.4
Percentage of human annotated cells
random
CAL
DAL
DCAL-0.3
DCAL-0.5
DCAL-0.7
DCAL-0.9
human
Figure 4: ActSort converges rapidly (at 10% data) and outperforms human annotators (at 1 3%). We simulated online annotation scenarios using the hemisphere datasets from three mice with 12 associated human annotations (Table sorting up to 50% of cell candidates. An illustration of the ActSort workflow. (B) balanced accuracies, (C) true positive rates, and (D) true negative rates as a function of the annotation percentages. Solid lines: means. Shaded areas: s.e.m. over 12 annotators. Dashed line and gray area: mean and s.c.m. from 12 human annotations. See Table and Fig. S13B for additional details. E-G We re-ran the experiments in (B-D), but processed multiple imaging datasets simultaneously. Dashed line and gray area: mean and s.c.m. from 64 augmented human annotations. See Table S7 and Fig. ST3 for additional details.
ActSort with DCAL reached human-level balanced accuracy with annotating roughly 3% of the full dataset and converged within < 10% beyond these levels, even though the classifiers were trained on human outputs (Fig. 4B). In comparison, random sampling reached the same marks with roughly 10% and 50% annotated samples, respectively (Fig. 4B-D).
Next, we considered processing cell candidates from multiple animals in single combined batch (Fig. GEG To test this scenario, we augmented 64 new annotators and corresponding evaluators by selecting one annotator from each dataset (Fig. B) Consequently, each combined dataset contained slightly more than 28,000 samples across three mice. this experiment, with < 1% annotated instances, DCAL reached human-level balanced accuracy, and had minimal improvements beyond roughly 3% annotated samples (Fig. CC. Surprisingly, both DAL and CAL massively underperformed DCAL, offering little improvement over random sampling (Fig. B-G).. These results underscore the
7

---

A
Labeled and / or unlabeled cell data
Unlabeled
Current dataset
Labeled cell data
Automated cell labeling
Human annotator
Pre-annotated datasets
movic
Active learning module
If If
Closed-loop
movie
GUI
Pre-labeled cell data
Oversampling
B
C
D
0.8
Balanced Accuracy
True Positive Rate
True Negative Rate
0.7
0.95
0.9
0.62
0.7
0.8
0.5
20
20
20
Percentage of human annotated cells
random
CAL
DAL
DCAL-0.3
DCAL-0.5
DCAL-0.7
DCAL-0.9 human
Figure ActSort pre-trained with previously annotated datasets converges faster on animals. A To test ActSort's generalization capacity, we trained cell classifiers on half of the cell candidates from one mouse (representing pre-am annotated datasets) and fine-tuned on samples from another mouse (representing the new dataset that needs to be annotated) using the hemisphere dataset. The GUT displays the current dataset, whereas the active learning module trains the cell classifiers using both newly- and pre-annotated data in the background. The query algorithm then selects the next candidates for human annotation, facilitating fine-tuning to the dataset of interest (Method; Appendix D.5). B D We created 6 pairs of mice, involving 16 augmented annotations per pair, totaling 56,020 cell candidates and 224,080 annotations. The first dataset is sorted up to 50% of cell candidates. Using the same query algorithm, the second dataset is sorted to 20%. We evaluated various query strategies random, CAL, DAL, and DCAL by computing (B) balanced accuracies, (C) true positive rates, and (D) true negative rates by averaging Solid lines: means. Shaded areas: s.e.m. over 96 augmented annotations pairs. Dashed line and gray area: mean and s.c.m. from human annotations. See Table SB and Fig. S for additional details.
the 96
pairs.
importance of broader coverage and better representation of the decision space, especially when the annotation dataset contains diverse cell candidates sampled from multiple imaging sessions.
The hemisphere-ue wide movies we considered so far can be considered as high quality cell extraction results, with false-positive rates already as low as 5-10% per movie and high quality imaging and experimental conditions. Next, we asked whether ActSort would be able to accurately process lower quality imaging datasets. To test this, we processed one of the cortex-wide movies from 44 with less stringent quality parameters in EXTRACT such that the resulting cell extraction output had many halse-positives, around 35%, by design (Fig. S9 We also tested ActSort a two-photon Ca²+ imaging movie with residual motion (Fig. STOP.
both datasets, DCAL outperformed all other approaches. Hence, the low false positive rate in the cell candidates or the optimal imaging/processen conditions cannot explain the success of ActSort in converging rapidly with few annotated samples. We further confirmed that these results were robust to changes in hyperparameters, such as the regularization parameter (Fig. S11 and classifier thresholds (Fig. S12).
As a test of ActSort's generalization ability, we conducted additional experiments with a set of pre-labeled samples using the three Ca imaging movies of the hemisphere dataset. Specifically, we pre-trained the cell classifier on one dataset using 50% of the pre-labeled samples. This classifier was later fine-tuned using the real-time annotations on new dataset from different mouse. To test this pre-labeling approach, we paired movies with each other, creating six dataset pairs. Each pair had 16 augmented annotations, pre-labels came from of four annotators of the first movie,

---

fine-tuning was performed using one of four annotators of the second movie (Fig. 5 The details of the fine-tuning process can be found in Algorithm |S2 and Appendix D..]
Our results, illustrated in Fig. S demonstrate the effectiveness of combining pre-labeled samples with DCAL. Notably, with pre-labels, DCAL achieved a human-lever balanced accuracy with as little as 1% annotated samples and converged with roughly < 6% annotations (Fig. (). In contrast, random sampling did not achieve human-level performance until c.a. 8% annotations and had not converged even with the 20% samples (Fig. BB. Similar to before, we found that DAL had similar performance to random sampling in picking true positives (Fig. SCC) for which CAL was the better algorithm. However, DAL outperformed CAL in the true negative rates (Fig. [P)) As their combination, DCAL often outperformed or matched the accuracies of both of these approaches, highlighting its robustness and efffcctiveness in sorting cells under varying conditions (Fig. 5
In Fig. S13 we provide additional metrics from our analyses performed in the main text. Finally, in Appendix AA.3 and Fig. [S14] we showcase how to apply ActSort to datasets processed with another cell extraction algorithm utilizing independent component analysis [52] and that cell sorting is necessary to cull out false positive signals that may correlate with animals' behavior.
3 Discussion
ActSort is as a standalone quality control software, which can be used to probe the outputs of cell extraction pipelines such as EXTRACT [13, Suite2p [10]. ICA [52), CAIMAN [12,, and
others [ 19 53 54). These cell extraction algorithms take the raw Ca 2+ movie as their inputs, correct the brain motion, perform simple transformations to standardize the Ca²+ imaging movies, identify putative cells' spatial profiles and temporal activities, and often perform primitive quality controls to output the final set of neural activities. Historically. additional quality controls on the cell extraction outputs would be performed with manual annotation, which was feasible for the small neural recordings with up to thousands of neurons [20 23 24 25 26 27 Yet, with the advent of large scale Ca²+ imaging techniques, now recording up to one million cells (9) manual review is no longer realistic. Instead, the field of experimental neuroscience direly needs automated quality control mechanisms that would correctly identify the true cell candidates while rejecting false positives
misidentified by the extraction algorithms.
ActSort is the first scalable and generalizable solution in this direction. Yet, previous works (as parts of existing cell extraction pipelines [10 12 [3) had tackled this problem in specific instances: Suite2p
designed cell classifiers based on some basic features to increase the precision of the algorithm 10
CAIMAN pre-trained deep classifier for two-photon movies [12] (though not applicable to 1p Ca imaging movies (55)). whereas EXTRACT performed thresholding on a set of quality metrics [13] Notably, these existing automated methods with pre- pre-trained cell classifiers often found success only for high quality 2p Ca imaging movies [IO [2). and even then underperformed human annotators 12 One-photon Ca imaging datasets, on the other hand, are quite diverse in their imaging (miniscope vs mesoscope) and experimental (head-fixed vs freely behaving) conditions and face additional challenges due low resolution and high background noise. With ActSort, we sought a generalizable solution that does not target a specific modality or require substantial re- re-training, but uses interpretable features that are robust across various behavioral experiments, Ca indicators, imaging conditions, and techniques.
To provide baseline for existing methods in our benchmarks, we designed a feature set called "traditional features" (Fig. 2 including features used by classifiers in prior works (See Appendix [..]). These methods did not use active learning, instead annotating random subsets. In our experiments, these prior methods are represented as the random sampling query algorithm, using the full feature set for fair comparisons with active learning approaches. In all benchmarking experiments, active learning (specifically DCAL) clearly outperformed random sampling, as is traditionally done in the literature 20 Notably, classifier predictions showed significantly lower variability compared to human annotations despite being trained on them (see, e.g., the differences between gray and colored shaded regions in Figs. 4 and 5). These observations suggest that automation not only reduces human labor but also standardizes it.
Though we performed extensive analysis to highlight the efficiency and effectiveness of ActSort, our work is merely first step for what may hopefully become a fruitful collaborative subfield comprising of experimental neuroscientists and active learning researchers. There are several future directions that future work could improve upon, which we will briefly summarize below.

---

Firstly, in this work, we used linear classifiers for rapid online training during cell sorting and reproducibility of annotation results. This was mainly rooted in the fact that pre-training deep networks required substantial data and standardization across various Ca imaging movies. We believe that with additional public datasets that may follow our lead, this direction can become reality (as was the case for a different, yet relevant, problem of spike extraction [56]). Our results in this work have set a strong baseline for such future deep-learning approaches.
One limitation of ActSort is that it functions as a quality control pipeline to existing cell extraction approaches. Therefore, any mistakes in the cell extraction step would automatically propagate to cell sorting. Yet, some of these mistakes can be mitigated or highlighted post hoc. For instance, future ActSort versions could involve options for merging segmented or duplicated cells, or identifying motion in activity traces and thereby notifying the user to improve their preprocessing steps.
Another important aspect requiring further exploration is the expertise level of the annotators. To date, each Ca imaging movie is often annotated by single annotator, who unfortunately can tire after long hours and become inconsistent as we have discussed above. This was the reason behind our choice to have the movies in our benchmark be annotated by multiple researchers. Yet. the human annotators had different levels of expertise in working with Ca imaging datasets. Hence, potential moderation relationship can exist depending the expertise of the annotator. To fully explore this relationship requires further research with more annotators per dataset, by having the same annotators sort the same cells in different times, and/or testing sorting effectiveness before and after a standardized sorting training.
Finally, we validated ActSort for one-photon and two-photon Ca imaging datasets and binary class sorting. Yet, our framework is generally applicable to other modalities, barring additional feature engineering performed by domain experts in those fields, or with pre-trained deep network. For instance, by replacing the logistic regression with a multinomial version and approximating CAL scores with entropy instead of decoder scores, our work readily applies to multi-class datasets that may jointly include, e.g., dendritic and somatic activities. With the right features, our framework could potentially support emerging voltage imaging recordings, especially as the number of cells will inevitably increase in such movies with the technological advances.
Overall, while human annotation was possible in datasets discussed in this work due to the manageable number of cell candidates, data analysis and hypothesis testing at scale with millions of neurons will require robust and automated cell sorting tools. In such cases, ActSort, and tools that will follow its lead, are essential for filtering out non-neuronal artifacts that could otherwise compromise the accuracy and reproducibility of neuroscience research.
4 Conclusion
In this work, we introduced ActSort, a quality control algorithm for processing large scale 2a imaging datasets. We utilized real-time human feedback via a closed-loop active learning paradigm to decrease the need for human annotation by two-orders of magnitude. We achieved this by first engineering features that generalize across experimental and imaging conditions, and by developing a simple, yet effective, active learning query algorithm. Finally, to support the development of active learning algorithms in systems neuroscience, we introduced the first publicly available benchmark for cell extraction quality control on both one- and two-photon large-scale Ca imaging, comprising five datasets: four one-photon and one two-photon Ca imaging datasets, with approximately 40,000 cells and 160,000 annotations.
The potential impact of this work is two-fold: i) decreasing the number of hours humans spend annotating modern Ca²+ imaging data sets, and ii) increasing the accuracy of cell classification by mitigating the annotator biases. paradigm requiring that merely 1-5% of cells be annotated by a human can reduce cell sorting from several hours per 2a imaging data set, to merely tens of minutes. Through ActSort, researchers can reclaim tens of hours that would have otherwise been spent labeling modern massive data sets. Importantly, increasing cell detection accuracy can be largely beneficial to modern systems neuroscience, which relies on minimizing false positive cell identification to avoid spurious results. Thus, removing the human element from repetitive tasks that can be prone to errors can ultimately improve the reliability, robustness, and reproducibility of systems neuroscience research.
10

---

Acknowledgements
We would like to thank Beyzanur Arican Dine for suggesting the intraclass correlation analysis to quantify the consistency among annotations. HOA acknowledges funding support from Ozyegin University. JZ. thanks Matlab's Toolbox Trainee program for fostering a connecting to the EXTRACT team at Schnitzerlab. CM acknowledges funding support from the Stanford University Wu Tsai Neurosciences Institute Interdisciplinary Scholar Award. FD, YL, and YJ received funding from Stanford University's Mind, Brain, Computation and Technology program, which is supported by the Stanford Wu Tsai Neuroscience Institute. MJS gratefully acknowledges funding from the Howard Hughes Medical Institute, NIH Brain Initiative, Simons Collaboration the Global Brain, and the Vannevar Bush Faculty Fellowship Program of the U.S. Department of Defense.
Data and code availability
ActSort software and the benchmark used in this study are publicly at our Github repository: hutps//ggttbboooommmmmmtteee///d
References
[1] Christine Grienberger and Arthur Konnerth. Imaging calcium in neurons. Neuron, 73(5):822-
885, 2012.
[2] Adam Packer, Lloyd E Russell, Henry WP Dalgleish, and Michael Häusser. Simultaneous all-optical manipulation and recording of neural circuit activity with cellular resolution in vivo. Nature methods, 12(2):140-146, 2015.
[3] Nicholas James Sofroniew, Daniel Flickinger, Jonathan King, and Karel Svoboda. A large field of view two-photon mesoscope with subcellular resolution for in vivo imaging. elife, 5:e14472, 2016.
[4] Malavika Murugan, Hee Jae Jang, Michelle Park, Ellia Miller, Julia Cox, Joshua P Taliaferro, Nathan F Parker, Varun Bhave, Hong Hur, Yupu Liang, et al. Combined social and spatial coding in a descending projection from the prefrontal cortex. Cell, 171(7):1663-1677, 2017.
[5] Krishna V Shenoy and Jonathan C Kao. Measurement, manipulation and modeling of brain-wide neural population dynamics. Nature communicationss 12(1):633, 2021.
[6] Francesco Gobbo, Rufus Mitchell-Heggs, Dorothy Tse, Meera Al Omrani, Patrick A. Spooner, Simon R. Schultz, and Richard G. M. Morris. Neuronal signature of spatial decision-making during navigation by freely moving rats by using calcium imaging. Proceedings of the National Academy of Sciences, ( 2022.
[7] Lynne Chantranupong, Celia C Beron, Joshua A Zimmer, Michelle J Wen, Wengang Wang, and Bernardo L Sabatini. Dopamine and glutamate regulate striatal acetylcholine in decision-makee mmaiin.. Nature, 621(7979):577
2023.
[8] Tony Hyun Kim and Mark Schnitzer. Fluorescence imaging of large-scale neural ensemble dynamics. Cell, 185(1):9-41, 2022.
[9] Jason Manley, Sihao Lu, Kevin Barber, Jeffrey Demas, Hyewon Kim, David Meyer, Fran- cisca Martínez Traub, and Alipasha Vaziri. Simultaneous, cortex-wide dynamics of up to million neurons reveal unbounded scaling of dimensionality with neuron number. Neuron, 2024.
[10] Marius Pachitariu. Carsen Stringer, Sylvia Schröder, Mario Dipoppa, L Federico Rossi, Matteo Carandini, and Kenneth D Harris. Suite2p: beyond 10,000 neurons with standard two-photon microscopy. BioRxiv, page 061507, 2016.
[11] Pengeheng Zhou, Shanna L Resendez, Jose -orgguez--RRauuuraaa Jessica C Jimenez, Shay Q Neufeld, Andrea Giovannucci, Johannes Friedrich, Eftychios A Pnevmatikakis, Garret D Stuber, Rene Hen, al. Efficient and accurate extraction of in vivo calcium signals from microendoscopic video data. elife, 7:228728, 2018.
[12] Andrea Giovannucci, Johannes Friedrich, Pat Gunn, Jérémie Kalfon, Brandon L Brown, Sue Ann Koay, Jiannis Taxidis, Farzanch Najafi, Jeffrey L Gauthier, Pengcheng Zhou, et al. Caiman an open source tool for scalable calcium imaging data analysis. elife, 8:e38173, 2019.
[13] Fatih Dine, Hakan Inan, Oscar Hernandez, Claudia Schmuckermair, Omer Hazon, Tugce Tasci, Biafra O. Ahanonu, Yanping Zhang, Jerome Lecoq. Simon Haziza, Mark J. Wagner, Murat A.
11

---

Erdogdu, and Mark J. Schnitzer. Fast, scalable, and statistically robust cell extraction from large-scale neural calcium imaging datasets. bioRxiv, 2024.
[14] Atika Syeda, Lin Zhong, Renee Tung, Will Long. Marius Pachitariu, and Carsen Stringer. Facemap: a framework for modeling neural activity based on orofacial tracking. Nature Neuroscience, pages 1-9, 2023.
[15] Aniruddha Das. Sarah Holden, Julie Borovicka, Jacob Icardi, Abigail O'Nicl, Ariel Chaklai, Davina Patel, Rushik Patel, Stefanie Kaech Petrie, Jacob Raber, et al. Large-scale recording of neuronal activity in freely-moving mice at cellular resolution. Nature Communications, 14(1):6399, 2023.
[16] Ying Ma, Mohammed A Shaik, Sharon H Kim, Mariel G Kozberg, David N Thibodeaux, Hanzhi T Zhao, Hang Yu, and Elizabeth MC Hillman. Wide-field optical mapping of neu- ral activity and brain haemodynamics: considerations and novel approaches. Philosophical Trunsuctions of the Royal Society B: Biological Sciences, 371 (71
2016.
[17] Jack Waters. Sources of widefield fluorescence from the brain. Elife, 9te59841, 2020.
[18] Jeffrey L Gauthier, Sue Ann Koay, Edward H Nieh, David W Tank, Jonathan W Pillow, and Adam S Charles. Detecting and correcting false transients in calcium imaging. Nature Methods, pages 1-9, 2022.
[19] Eftychios A Pnevmatikakis, Daniel Soudry, Yuanjun Gao, Timothy A Machado, Josh Merel, David Pfau, Thomas Reardon, Yu Mu, Clay Lacefield, Weijian Yang, et al. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron, 89(2):285-299 2016.
[20] Gregory Corder, Biafra Ahanonu, Benjamin F Grewe, Dong Wang, Mark J Schnitzer, and Grégory Scherrer. An amygdalar neural ensemble that encodes the unpleasantness of pain. Science, 363(6424))276-2811 2019.
[21] Anne E Urai, Brent Doiron, Andrew M Leifer, and Anne K Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature neuroscience, 25(1):11-19, 2022.
[22] Jeffrey Demas, Jason Manley, Frank Tejera, Kevin Barber, Hyewon Kim, Francisca Martínez Traub, Brandon Chen, and Alipasha Vaziri. High-speed, cortex-wide volumetric recording of neuroactivity at cellular resolution using light beads microscopy. Nature Methods, 18(9):1103-
1111, 2021.
[23] Christian Marzahl, Christof A Bertram, Mare Aubreville, Anne Petrick, Kristina Weiler, Agnes Gläsel, Marco Fragoso, Sophie Merz, Florian Bartenschlager, Judith Hoppe, et al. Are fast labeling methods reliable? a case study of computer-aided expert annotations on microscopy slides. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part I 23, pages 24-32. Springer, 2020.
[24] Massimo Salvi, Valentina Cerrato, Annalisa Buffo, and Filippo Molinari. Automated seg- mentation of brain cells for clonal analyses in fluorescence microscopy images. Journal of Neuroscience Methods, 325: 108348, 2019.
[25] Jacopo Amidei, Paul Piwek, and Alistair Willis. Identifying annotator bias: A new irt-based method for bias identification. In Proceedings of the 28th International Conference on Compu- tational Linguistics, pages 4787-4797, 2020.
[26] Shanshan Wang, Cheng Li, Rongpin Wang, Zaiyi Liu, Meiyun Wang, Hongna Tan, Yaping Wu, Xinfeng Liu, Hui Sun, Rui Yang, et al. Amnotation-efficient deep learning for automatic medical image segmentation. Nature communications, 12(1):5915, 2021.
[27] Mike Schaekermann, Graeme Beaton, Minahz Habib, Andrew Lim, Kate Larson, and Edith Law. Understanding expert disagreement in medical data analysis through structured adjudication. Proceedings of the ACM on Human- Computtr Interaction, 3CCSCW):1-23, 2019.
[28] David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pages 148-156. Elsevier, 1994.
[29] Andrew McCallum, Kamal Nigam, et al. Employing em and pool-based active learning for text. classification. In ICML, volume 98, pages 350-358. Citeseer, 1998.
12

---

[30] Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estimation of error reduction. ICML. Williamstown, 2:441-448, 2001.
[31] Ajay Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009 ieee conference on computer vision and pattern recognition, pages 2372-2379. IEEE, 2009.
[32] Hieu T Nguyen and Amold Smeulders. Active learning using pre-clustering. In Proceedings of the twenty-first international conference on Machine learning, page 79, 2004.
[33] Yuhong Guo. Active instance sampling via matrix partition. Advances in Neural Information Processing Systems, 23, 2010.
[34] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agar- wal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint ar0iv: 03671, 2019.
[35] Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34:1 1911-19394444 2021.
[36] Jonathan Folmsbee, Xulei Liu, Margaret Brandwein-Weiee and Scott Doyle. Active deep learning: Improved training efficiency of convolutional neural networks for tissue classification in oral cavity cancer. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 770-773, 2018.
[37] Peng Liu, Hui Zhang, and Kie B. Eom. Active deep learning for classification of hyperspectral images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(2):712-724, 2017.
[38] Sayantan, Kien, and KV Kadambari. Classification eeg beats using deep belief network and active learning. Medical and Biological Engineering and Computing. 56(10):1887-1895 2018.
[39] Michael Nalisnik, David Gutman, Jun Kong, and Lee AD Cooper. interactive learn- ing framework for scalable classification of pathology images. In 2015 IEEE International Conference on Big Data (Big Data), pages 928-935. IEEE, 2015.
[40] Ying Liu. Active learning with support vector machine applied to gene expression data for cancer classification. Journal of chemical information and computer sciences, 44(6):1936-19)) 2004.
[41] Weicheng Kuo, Christian Häne, Esther Yuh, Pratik Mukherjee, and Jitendra Malik. Cost- sensitive active learning for intracranial hemorrhage detection. In Medical Image Computing and Computer Assisted Intervention-MICCC 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part III 11, pages 715-723. Springer, 2018.
[42] Marc Gorriz, Axel Carlier, Emmanuel Faure, and Xavier Giró-i Nieto. Cost-effective active learning for melanoma segmentation. arXiv preprint 17 2017.
[43]
Schnitzerlab. ActSort-public. Github, 2024.
[44] Sadegh Ebrahimi, Jérôme Lecoq, Oleg Rumyantsev, Tugce Tasci, Yanping Zhang, Cristina Irimia, Jane Li, Surya Ganguli, and Mark Schnitzer. Emergent reliability in sensory cortical coding and inter-area communication. Nature, 605(7911):713-71,, 2022.
[45] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Har- ris. High-dimensional geometry 365, 2019.
geomerer of population responses in visual cortex. Nature, (77(6)53)
[46]
Allen Institute for Brain Science. Allen sdk documentation, 2024. Accessed: 2024-07-25.
[47] P Berens, L Theis, J Stone. N Sofroniew, A Tolias, M Bethge, and Freeman. Standardizing and benchmarking data analysis for calcium imaging. In Computational and Systems Neuroscience Meeting (COSYNE 2017), pages 66-67, 2017.
[48] Kenneth O McGraw and Seok P Wong. Forming inferences about some intraclass correlation coefficients. Psychological methods, 1(1):30, 1996.
[49]
Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In AAAI, volume 5, pages 746-751, 2005.
13

---

[50] Burr Settles and Mark Craven. An analysis of active learning strategies for sequence labeling tasks. In proceedings of the 2008 conference on empirical methods in natural language processing, pages 1070-1079. 2008.
[51] Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint urXiv:1907.06347 2019.
[52] Eran A Mukamel, Axel Nimmerjahn, and Mark J Schnitzer. Automated analysis of cellular signals from large-scale calcium imaging data. Neuron, 63(6):747-760, 2009.
[53] Zhe Chen, Garrett Blair, Changliang Guo, Jim Zhou, Juan-Luis Romero-Sosa, Alicia Izquierdo, Peyman Golshani, Jason Cong, Daniel Aharoni, and Hugh T Blair. A hardware system for real-time decoding of in vivo calcium imaging data. Elife, 12:e78344, 2023.
[54] Yaesop Lee, Jing Xie, Eungjoo Lee, Srijesh Sudarsanan, Da-Ting Lin, Rong Chen, and Shuvra S. Bhattacharyya. Real-time neuron detection and neural signal extraction platform for miniature calcium imaging. Frontiers in Computational Neuroscience, 14, June 2020.
[55] Flatiron Institute. Caiman demo pipeline for cnmf-e. https://github: com/
ipymb/ 2024. Accessed: 2024-08-12.
[56) Peter Rupprecht, Stefano Carta, Adrian Hoffmann, Mayumi Echizen, Antonin Blot, Alex C Kwan, Yang Dan. Sonja B Hofer, Kazuo Kitamura, Fritjof Helmchen, et al. database and deep learning toolbox for noise-optimized, generalized spike inference from calcium imaging. Nature Neuroscience, 249):1324-1337, 2021.
[57] Carsen Stringer, Tim Wang. Michalis Michaelos, and Marius Pachitariu. Cellpose: generalist algorithm for cellular segmentation. Nature methods, 18(1):100-100 1060000-66600000 2021.
[58] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[59] Math Works. ResNer-50: Pretrained Deep Neural Network. MathWorks, Inc., Natick, MA, USA, 2024. :/ wwwp::/.wtwww.s.cooom www.://w....oooooooooe/eeee
html
[60] Jones G Parker, Jesse D Marshall, Biafra Ahanonu, Yu- Wei Wu, Tony Hyun Kim, Benjamin F Grewe, Yanping Zhang, Jin Zhong Li, Jun B Ding, Michael D Ehlers, et al. Diametric neural ensemble dynamics in parkinsonian and dyskinetic states, Nature, 557(7704) 2018.
[61] Peter O'Connor. PeakSeek.
https: /wwww//wwww.... wwwhworks. wom./matlabcentraaaaa
fileexchangee26681111 /265811peakeeet 2024. MATLAB Central File Exchange, Retrieved
August 21, 2024.
[62] Hakan Inan, Murat A Erdogdu, and Mark Schnitzer. Robust estimation of neural signals in calcium imaging, Advances in neural information processing systems, 30, 2017.
[63] Oleg I Rumyantsev, Jérôme A Lecoq, Oscar Hernandez, Yanping Zhang. Joan Savall, Radoslaw Chrapkiewicz, Jane Li, Hongkui Zeng, Surya Ganguli, and Mark J Schnitzer. Fundamental bounds on the fidelity of sensory cortical coding. Nature, 580(7801):
2020.
[64] Eran A Mukamel, Axel Nimmerjahn, and Mark J Schnitzer. Automated analysis of cellular signals from large-scale calcium imaging data. Neuron, 6376777777 2009.
14

---

A Additional results
A.1 Expert-engineered features set strong baselines against deep learning approaches
Deep learning approaches have seen recent success in relevant domains of our application [57] In regards to cell sorting, a cell extraction pipeline. CAIMAN [22,, trained deep classifiers to cull out false positives from two-photon calcium imaging datasets. These classifiers were trained solely on cells' spatial footprint, not using any temporal or spatiotemporal information often used by human annotators. Consequently, this method was suboptimal compared to the consensus human annotation (see Table 1 in (2), and cannot process one-photon imaging movies (55] As we show in the main text, neither true for ActSort thanks to the generalization of features across broad imaging and experimental modalities. Yet, perhaps a better feature extraction may be performed in an unsupervised manner using deep learning approaches. Thus, as another strong baseline, we next investigated whether a pre-trained image model, a ResNet-50 architecture [58]. could enhance the unsupervised discovery of features.
To perform this test, we extracted features from cropped cell images and movie frames using the pre-trained model (Methods; Appendix D.3 The movie frames, consisting of time-averaged snapshots around the cell's location during Ca events, offered spatiotemporal information, which was designed to provide a fair comparison to our engineered features. Using these images, we extracted 2,000 features from the penultimate layer of ResNet-50 50 [58] [59. Notably, though extracted features led to increased true negative rate, on par with our engineered features, this came at the cost of significantly decreased true positives (down to ~ 90%, Fig. S4). Therefore, due to their superiority (Figs. and (4), we decided to use the newly engineered feature set in our ActSort pipeline.
A.2 Humans are inconsistent, but cell classifiers mitigate annotator biases
Can human annotations serve as ground truths for cell sorting? To answer this question, we tested the possibility that human annotators may be inconsistent due to, e.g., personal biases. To quantify this inconsistency, we performed intraclass correlation (ICC) analyses [48), a method commonly used to assess the reliability of ratings in a scale (here, binary annotations), either by individual raters or averaged across multiple raters. Since, in experimental scenarios, only one human annotator typically handles a dataset, we computed two-way mixed, single rater ICC 48]. which measures the consistency of individual annotators rather than the group average. Across all five mice, the intraclass correlation scores for individual human annotators were ICC 0.56 + 0.06 (mean = std), indicating that human annotators often disagree, and are inconsistent, on what they may classify as cells.
Next, we checked whether cell classifiers can mitigate individual biases and show higher consistency when trained human annotations. To quantify the classifiers' consistency, we split the annotation datasets into equal training and testing sets, trained the cell classifier on the former, and computed the label on the latter (See Appendix D.I). The cell classifiers had higher correlation scores, ICC = 0.64 + 0.07 (mean + std), showing improved consistency compared to individual human annotations despite being trained on them.
As the final test, we trained and evaluated the cell classifiers by reanalyzing a published one-photon Ca²+ imaging dataset covering the neocortex [44], which included 6,691 cell candidates and 26.764 annotations (Table S2). We once again computed the test probabilities from cell classifiers. These probabilities were compared with the empirical probability computed from annotator agreements (e.g.. a vote of 2-2 corresponds to a 50% probability, Fig. (5). We found that annotator expertise was crucial for the accuracy of the cell classifiers trained with these annotations. But, as overall trend, human annotators were overconfident in their choices. In contrast, the cell classifiers, especially those trained with more experienced annotators, were better calibrated (Fig. (5). Thus, we confirmed that cell classifiers with the newly engineered feature set can match or even surpass the consistency of individual human annotators.
A.3 ActSort culls out non-neuronal signals that may correlate with behavior
Taken together, our benchmarking experiments demonstrate the critical role of active learning query algorithms in automated cell sorting. DCAL, which optimally balances two canonical active learning approaches, consistently outperformed human annotators, sometimes with as few as < 1% of annotated samples. Yet, final question remains: Is cell sorting truly necessary?
15

---

To address this, we re-analyzed previously published recordings from the dorsomedial striatum of freely behaving mice 60). Each dataset includes neural calcium activity recorded using the fluorescent calcium indicator GCaMP6m in spiny projection neurons of either the direct (9 mice) or indirect pathway (10 mice) of the basal ganglia (dSPNs and iSPNs, respectively). These datasets, originally processed using an independent component analysis- cell extraction algorithm (ICA, [52), contained 336 + 132 (mean + std) cell candidates, which were sorted by human annotators. We re-sorted these datasets using ActSort, with human annotations as inputs and CAL as the query algorithm due to the low diversity in cell candidates. Due to the dataset's 's nature, the evaluation was performed against single annotator, yielding balanced accuracy of 90 + 4% (mean = std) for ActSort annotations, converging at 20% annotations.
As shown in Fig. S14 both human and SctSort-sorted dSPNs and iSPNs significantly predicted animal speeds. Remarkably, ActSort achieved the same level of predictive accuracy with as little as 1 % annotation, resulting in cells with nearly identical speed predictions (Fig. B-14 However, even the rejected cell candidates contained residual information about the animals' speeds, likely due to brain motion artifacts. This highlights a crucial issue: non-neuronal signals in calcium imaging movies can inadvertently correlate with animal behavior, leading to potentially misleading biological conclusions. Thus, the false-positives should be culled out to prevent unsolicited information leakage in the downstream analyses.
B Software details
In this work, we introduce ActSort, a user-friendly pipeline consisting of modules: a preprocessing module, a cell selection module, and an active learning module. The preprocessing module efficiently reduces the data size associated with 10, 000 neurons down to few GBs, which includes not only cells' Ca activity traces, but also spatial profiles, and movie snapshots during Ca events -allowing end users to check the quality of the data-, together with the engineered features. This compact representation allows the ActSort pipeline to be run locally in laptops, despite original movie sizes of up to few hundred GBs. The joint compressing of the movie snapshots and cell extraction information offers another key contribution to the neuroscience community with implications for sharing imaging datasets publicly. The cell selection module features a custom design with an casy-to-use interface that displays temporal, spatial, and spatiotemporal footprints, and incorporates closed-loop online cell classifier training system. During the annotation process, the software provides real-time feedback by displaying predicted cell probabilities and the progress made by the human annotator well as the fraction of unlabelled cells that ActSort is confident about. This allows experimentalists to monitor their work and make informed decisions regarding automated cell sorting outputs. The active learning module works in the background, strategically selecting candidates for human annotations, and trains cell classifiers with annotated candidates.
B.1 Preprocessing module
The preprocessing module is included within the GUI. This module reduces repetitive calculations by precomputing all necessary components.
The inputs for the module are the Ca imaging movie and the corresponding cell extraction output, which includes spatial data for each cell in the dataset and trace activity data. This module performs 1) a joint compression of Ca imaging movies and cell extraction results, 2) computes all the visualization components, and 3) performs feature engineering. The output for the module is 'sorting file' that contains the necessary components for cell visualization in GUT, features for each cell, and an 'info' field that provides useful statistics.
To accommodate different system capabilities, the user can select whether data is loaded entirely into memory or is read in chunks. Furthermore, for large datasets, parallel processing on both CPU and GPU can be selected. Note that for small datasets, parallel processing might slow down this process.
B.1.1 Joint compression of Ca 2+ imaging movies and cell extraction results
For efficient handling of the growing sizes of cellular data on local laptops (with limited memory capabilities), we designed a new method for compressing both imaging movies and cell extraction results.
Compression of Ca Imaging Movies Traditionally, cell sorter GUTs require the 2+ imaging movic to be loaded along with cell extraction results. However, the growing size of these movies
16

---

makes it impractical to store and load them both on hard disk and RAM. To mitigate this, we imple- mented Ca event detection and snapshot capturing steps in our precomputation process.
1. Ca Event Detection:
Using the trace activity obtain from the cell extraction algorithm, we identify the five largest Ca events for each cell that represent distinct high activity moments. We utilize publicly available spike detection algorithm (peakseek. 61 to determine these Ca events.
2. Snapshot Capturing:
For each selected Ca² event, cropped snapshot is extracted comprising ten frames in a small window that includes neighboring cells during that time. Only these movie snapshots are stored. This process eliminates the need load or share the entire movie file.
Compression of Cell Extraction Results ActSort requires two main components from the cell extraction process: spatial filters and trace activities of each cell.
The spatial filters are typically stored in three-dimensional arry ([Height Width Number of Cells]), where each layer contains the spatial filter of one cell. Given the small size of each cell, this array is sparse and requires significant memory to store and load into RAM. To address this, we convert the spatial filters array into a sparse matrix, which allows to save a great amount of memory. All future code is organized to ensure this sparse matrix is never converted back to its full form and computations are performed directly on the sparse matrix. Additionally, trace activities are converted from 'double' 'single' data type save memory but still keep the precision at the same time.
B.1.2 Visualization Components Calculation
On top the 2a²- imaging movie snapshots and the compressed spatiotemporal information from the cell extraction algorithm, we also compute several components for visualization purposes in GUI:
Cell Centers: Stores the center coordinates of cach cell.
Ca Event Indices: Indices of the five most informative Ca events for each cell's trace activity.
Cell Boundaries: Contains the boundary coordinates of each cell in spatial data.
Snapshots: Stores snapshot images taken from the movie.
Snapshot Filters: Filters applied to the snapshots.
Snapshot Traces: Traces corresponding to the snapshots, used for feature extraction.
Snapshot Cell Boundaries: Boundary coordinates of cells in the snapshots.
Snapshot Contrast Limits: Proper contrast limits to better visualize the snapshots.
Neighbor Boundaries: Boundaries of neighboring cells the snapshots.
Maximum Projection: The maximum intensity projection of the movie, used as background image during cell label visualization.
B.1.3 Feature engineering
Below is the list of features used for this work. We subscript to denote individual cell in the following section. The traditional ones are colored blue.
Features 36 are based the Ca trace activities (activity signals of the neurons detected by the cell extraction algorithm, denoted as T Ruoxxns) of the given cell dataset, where n4 the number of recorded frames of the Ca movie.
Feature 1 calculates the signal-to-noiss ratio for the given trace activity. A lower signal-to-noiss ratio indicates a noisy data, which more likely to contain false positives. A higher ratio indicates a higher quality data. This feature is taken from EXTRACT [3]
max Ti)
fi,
17

---

where n is estimated using an algorithm that computes half the power in the high frequency components of the Ca activity traces. [T3] has taken this approach to provide accurate noise estimates despite non-negative and sparse nature of the signals, which prevents computing noise as the standard deviation across the full spectrum unreliable. Below, we briefly summarize the algorithm used to compute the noise:
F FFT(
(compute the discrete fourier transform)
P. |F
(compute the Power Spectrum)
P:
P; =
(normalize the power spectrum before Parseval's theorem)
nh
Prolse
Pi [ne// ne/2]
(extract half of the high frequency components)
ne/2
P.N
(compute the average power)
nnise.
nt/4
k=
Feature 2 calculates how much cell's trace activity signal differs from its smoothed version. If the difference is high after smoothing, then it is likely for the cell trace activity to contain good amount of noise. This feature is taken from EXTRACT 13
T i
median(Ti..-1. Ti,, Ti.t
(median-fileee filtering)
a 0.3 senxnnnnnthe
(set a threshold identify signal regions where noise dominate)
V; {1 Sm
0} Th,t T im.othed,
f2,
1
Σ
V
max max(Te,t, smoothed,
TEV.
Feature 3 calculates the number of weak Ca events within the trace. We categorize the Ca events as follows: We compute the 90th quantiles of the Ca²t activity for each cell, and divide them by half. Any Ca event with lower Ca amplitude is considered "weak," whereas the ones with higher C amplitude are considered "strong Ca events."
f3.1 = [tt Tmaxnoom, 1) < mmaxoormx(0 (t) T Tmaxnorm, (t + 1)} and pt(t) < MPP,
MPP; = max(0.5 x (o. .), 0.1) is the thresh- old determine whether peak is prominent, Qo (o..("Tmaooomm.. is the 90th percentile of T ii (t), and p. is the prominence of each peak, defined as the difference with respect to the baseline activity.
where Tmaxxnorm.<0 (t) T.a
max, Ti.
Feature calculates the standard deviation of the upper top 10% of each Ca activity trace, which reveals the variability within the 2+ event amplitudes.
T T zscore Ti)
(zscore the temporal traces) (compute the 90th percentile value) (define the set of top 10% values)
q0.9, Quantile (T [Torm, 0.9)
S (Tnom. Toom,,i. 2 90.9.s}
f4, std(S:)
Features 7 measures the average time interval between two consecutive strong Ca events. Constant and rapid spiking is often associated with motion artifacts and/or blood vessels.
Define G, as set of positions of the peaks with prominence 2 MPP;, A, as a set of positions of all peaks, and B, set of positions of peaks with prominence < MPP,
18

---

Ela.1
(4-- - [x)
f3.1
,G f2,,
G2
(fx+1
[x)
f6,1
k==1
B: = (2,,)
1A.1 k=
(lx+1 (k))
f7,
A, = (t1,t2.-.)
[Ag]
Feature measures the average amplitude all Ca²+ events in a trace activity.
We define the width of peak happening at time k as the width at half height of the peak, denoted (kk).
fs. =
wi (k)
A
KEA.
Feature 9 measures the average length of Ca events. Stronger Ca events a good sign of good candidates, while had candidates tend to have Ca events that last longer.
for
wi(k)
G
kEG;
Feature 10 calculates the standard deviation of the Ca event width within trace of cell activity. A more widely distributed Ca event width can be good indicator of noisy and less precise activity.
f1a std (ww;(k) & E A.))
Feature 11 calculates the standard deviation of the Ca event width among the strong Ca events. This can serve as an indicator of their quality. A narrower distribution of Ca2+ event widths suggests higher level of consistency.
fu, = std ((wr(k)) & E G.))
Feature 12 measures the proportion of total activity in cell trace made up of significant Ca2+ events rather than noise, referred to relative power. This metric helps the distinction between actual C+ events (consistent signal energy) and the likelihood noise.
We define new array, T maaxnoru, =TT maxxoom.... T mmxxnorm.44 f.t Onoise,
E, (T "iaxnorrm...
f12,
Σ. (T (Tmaxnormss.e
Feature calculates the proportion of cell trace activity that surpasses the noise threshold. This tells the frequency of cell activities that is distinguished from the background noise.
f1a,1 avg (Tmoxnomm.
Feature 14 counts the instances when the cell's activity surpasses the noise threshold significantly. This provides insights into the firing frequency or notable cell activities. The function
processes the input trace 7 7 by first applying a median filter of length three to reduce impulsive noise and then smoothing this result through convolution with normalized
19

---

kernel to produce the smoothed trace s.mooth. A dynamic threshold is defined
noise,, max (Tsmoout.s): and event detection is performed by identifying peaks in T that exceeds threshold 8; and are separated by at least three frames; these candidate events are then refined by searching within a 2-fame window around each detected peak in the original trace T m to locate the precise local maxima, updating the event frames to the times where T m is maximized, and discarding any events corresponding to values in the original trace, resulting in the final set of significant event frames.
Features 15 29 are derived from different scales and threshold values: 3, 10, 30, and 100, denoted as a, respectively- for the calculation of relative power (Feature 12), the proportion of the cell trace activity that surpasses the noise level (Feature 13), and the number of strong Ca events (Feature 14).
We define array, T massorrm {T maxnorm,i,e T maxnorm, i, nuise,
Ee mmixnnormm...
S15 Ex (T maxxoormi..""
f16.1 avg av [Tmaxnoom.)
117 same as Feature 14 but set the threshold as noisei max
(famooth.s)
Features 30 34 calculate quality score that indicates how much of a neuron's 's activity occurs within certain frequency ranges of the spectrum. A higher score indicates that the majority the neuron's activity is happening within the specified frequency range, suggesting that the data quality is good and that the neuron is reliably detected. The frequency ranges determined by lower and higher threshold parameters. In our work, the thresholds are set to define four specific ranges within the frequency spectrum: the lowest 5% (0-0.05), the range from the lowest 5% up to the highest 95% (0.05-0.95). the lower half (0-0.5), and the upper half (0.5-1).
Sa 34,; count (PowerSpeetruu((T) [Joow Shigh))
Feature calculates for each cell the proportion of other cells in the dataset with which it has high correlation, using similarity threshold of 70%. This essentially measures how similar a cell's 's activity is to the activity of the other cells in the dataset.
count (comr (T, TT) > 0.7 1 1, Deells j + i)
fa5.1
Pleells
Feature 36 finds the total fluorescence that was caused by the highest 10% activity in a given cell activity trace. This metric can be useful detecting unusual neuron activity that doesn't reflect common neuron behavior in the dataset.
f151 = sum (Ti Tik 2 percentile,, (T:))
Features 37 to 46 are based on the spatial filters (the overall pictures of the cells according to the cell extraction algorithm, denoted as S E Rn x Xheelt, where nu x ny is the size of the movie frames) in the given cell dataset.
Feature calculates the total surface area of each cell the movie. This helps identify cells that are too large small, which may indicate outliers.
f3r.s surface area of cell i
20

---

Feature 38 finds the number of distinct bodies that cell candidate has. Cells that are detected to have more than one body tend to false positives.
J38.1 number of distinct bodies of cell
Feature 39 calculates the total sum of pixel values within each cell's spatial filter, which is measure the total activity or brightness. 'Bad' cells tend to have unusually high or low activity.
S39. = sum(S:)
Feature measures the circumference of the cell. Typically, cells with too large or too small circumference tend to be false positives.
f40,; circumference of cell
Feature 41 measures the distance from the cell center to the nearest edge of the movie's 's field of view. Cells that are located closer to the edges tend to be noise or artifacts due disruptions in the movie, such as boundary effects or uneven illumination.
f4L,3 = distance from center to nearest edge field of view for cell
Feature computes the circularity of a given cell, which measures how closely its shape aligns with perfect circle. Cells with a higher degree of circularity are generally preferable candidates.
This feature taken from EXTRACT 13
(perimeter of cell i)?
f12.:
4m x (area of cell i)
Feature 43 computes the eccentricity of a given cell, which indicates how stretched out cell's shape is. Candidates with higher eccentricity values tend be 'bad' cells. This feature is taken from EXTRACT [
f43,5 = eccentricity of cell i
Feature computes the average pixel value of the given cell, which a measure of the average activity. 'Bad' cells tend to have unusually high or low activity.
f44 = avg(S) (S,)
Feature 45 calculates the spatial corruption, which is known as errors or inconsistencies within the spatial filter of the cell by evaluating the local variance relative the overall variance the cell image. This feature is taken from EXTRACT [13
0 local,
f45,6
global,i
where o² ; represents the glohal spatial corruption and is calculated as
Σ
(Si, - Mgglobal,
globall
N kénon zzeu pixels
with & the mean intensity of non-zero pixels in cell i and N, the number of these pixels.
The local spatial corruption oi is obtained by applying spatial filter to produce locally smoothed version of S, denoted Siitered, and is defined as
1
Σ
(Si,
Spttere,
N
kénon-zero pixels
21

---

Feature 46 calculates the maximum spatial correlation of each cell with all other cells in the movie. This helpful identify cells that have similar activity patterns, This feature taken from EXTRACT 13
f46.1 (ax
J71
Features 47 76 based spatial filters and trace activities the cells that captured by the cell extraction algorithm, along with key movie frames that correspond to the highest points of cell activity. We call these spatiotemporal features.
Features -7- 50 compute the probabilities of non-Gaussian contamination the noise, as computed using EXTRACT algorithm (13). Feature 47 reports the mean probability across time points for each cell, feature 48 the minimum, feature 49 the 10th percentile, and feature 50 the 20th percentile.
Features 66 calculate the correlations of activity within the neuropil area -a dense network of neurons and their components that can cause false positives when activated- by analyzing the interactions between neighboring pixels in and around the cells. This feature is taken from EXTRACT 13
Features 76 calculates the correlation between the movie frames at the time of Ca activities and the extracted spatial profiles for each cell. This feature is used as a "trace quality metric" for evaluating EXTRACT [3] pipeline.
but is not used thresholding metric within EXTRACT
B.2 Graphical user interface (GUI)
We developed user-friendly graphical user interface to visualize the cell data. We added many useful tools to assist users in annotating cells. The memory optimization performed by the preprocessing module allows to use of this GUI in practically any local computer.
Below is the list of functionalities within the ActSort GUI, which we illustrate in Fig. S3
Cell Navigation: Includes slider bars, buttons, and number field to navigate through cells within the dataset.
Cell Stats: General statistics show the distribution of cells categorized good or bad, well as the number of cells that have been labeled and those that remain unlabeled.
Zooming Options: Includes zoom slider and a zoom cursor for scanning and zooming into specific areas.
Contrast Sliders: There are sliders to adjust the minimum and maximum contrast values for both the main map and snapshots.
-Cll Color Transparency: An option that allows the user to decide an appropriate contrast for the cells' spatial profiles.
-Cll Border Color Options:
Depending on the dataset, providing an option to change the cell border color can help users better visualize the cells on the main map.
Hiding Cells: An option to hide sorted cells, both good and bad, which is helpful for visualiza-
tion especially when working with large datasets.
Click to Select: An option to select cells by clicking on them in the map.
Multiple Selection: An option that allows selecting multiple cells from the map.
Sorting Unsorted Cells Only: An option, when selected, already sorted cells are no longer shown to the user when moving back and forth with the slider.
Classifier Threshold: An option to adjust the classifier threshold for prediction.
Prediitt Cells Button: An option that predicts the identities of the cells based on the decision thresholds (see below)
-Model selection: The user can change any of the active learning algorithms, CAL, DAL, or DCAL.
22

---

Algorithm S1 ActSort
(0) mi Data: ( = [[T 3i) wi)) _2(00) = {
b Initial Labeling
for / , d do
Train the cell classifier h using (()
Select based on the Query-Algorith thm and hg)
Human annotator label x(++1) with y E 1)
c(++1 (() U (z(t+1),))
u(++1) 400 (o(t1)
end for
ActSort Decisions: Whether ActSort labels the current annotation as a cell or not.
ActSort Statisties: The probability of the current annotation being cell, assigned by ActSort.
B.3 Active learning approaches
We define binary cell classifier, he, between cell and not cell, with input data X' R and binary ground truth labels y E TRNN in which N is the total number of extracted putative neurons and K 76 is the dimension of the features. Second, we defined label classifier, Pş. between labeled
data, _()) = (2,, }:}} for i = 1..... Bi
(t) and the unlabeled data, (t(*) = {xr} for i 1, I...., n2 N, forming new dataset D {(x,, 1) 0)
given iteration t such that n + m2 Vr; E (02 Vxz E J. The former classifier is responsible for both the final outputs and the query algorithm, whereas the latter is used by the query algorithm only. To allow real-time training, and
owing to its already sufficient accuracy thanks to the new feature set (Fig. A also see below), we used logistic regression for both classifiers:
h2(2) = 1 Po(x) = 1 1 X,
(SI)
which were trained using the loss function (same for both classifiers)
i = arg max ((0) = arg max yy log ho + (1 - y4) log(1 ho(x;))] + A|
(S2)
8
where À is the LI regularization strength The overall online closed-loop active learning pipeline is explained in Algorithm ST]
In this work, we tested four strategies query algorithms, random sampling, confidence-bais based active learning (CAL) [ 50]; discriminative active learning (DAL) [ii:: the newly developed
dnseriminative-coffddee based active learning (DCAL). The overall training structure is depicted in Algorithm ST
Traditional baselines
The random sampling algorithm randomly selects an unlabeled sample (1)) from U. After human annotation, this sample is moved the labeled set ((++1) and the cell classifier updated accordingly:
c(+1) = =(() u ,(,1),,0))) "))
(S3a)
(++1)= (1)))
(S3b)
Confidence-based active learning
The CAL algorithm selects sample according to the uncertainty sampling decision rule, which identifies unlabeled items that are near a decision boundary in the current cell classifier model. In multi-class setting, one can use entropy to define the uncertainty of the sample points in the unlabeled dataset, which is how we shall motivate the CAL algorithm. Let H(-) represent the entropy of a set
For the purpose of this work and real- time processing in the software, we used fixed default value for the regularization, which validated in Fig. SII
23

---

of samples. In round t + 1, we want to choose the sample that produces the maximum reduction in entropy given the trained cell classifier h/g in round
((++)) = arg max H (4)))2))) - 7 (1(+t)))((t))
arg min H ( (((tt))(((t)))
(S4)
(8) -h1 (xi) log (1100 (xi))
(1- ( log (1- ( (zi))
arg min aeu())
arg min 1 heg (1;)) 0.5
aEU(()
In this work, since entropy is the one where hg is as close to 0.5 as possible. However, extension of ActSort to multi-class problems, for example that includes the identification of dendrites, could benefit from the more general entropy minimization framework.
using only two classes, the probability distribution that maximizes the
Discriminative active learning
The DAL algorithm considers different type of uncertainty, i.e., how well data point is represented in the labeled pool [5 Here, one constructs new label space, 5 {1, u). that specifies whether the data sample from the labeled set or the unlabeled set. example, in round I, the training dataset becomes D { u), (I),.) Vz, E 409, Vx, c(*)). For each round of the active learning process, train binary classifier, Pá (ū]z), to predict whether the samples in D are labeled or not. Then, one selects the sample from the unlabeled set that satisfies
((+1) arg max =E((())
(ÿ = ul (D).
(S5)
In this work, we modified the original DAL algorithm from [51] to improve its performance on our benchmark by training the classifier p& (7 x) with random oversampling of the minority class samples in the dataset D.
Diseriminative-confddeee active learning
In order to achieve query score for the DCAL algorithm, we define an uncertainty-score, cf and discriminative-score d!"), corresponding to the two types of uncertainty, for each iteration as:
( - 2 (1) (z, ( 0.5 and de di'
p I3 11(1)
(S6)
Empirically, once the labeled set accurately represents the full data, annotating candidates at decision boundary yields the most significant improvements (See Fig. 3. Thus, we select the sample that maximizes the adaptively weighted sum of the uncertainty and discriminative scores from the unlabeled dataset:
x (++1) = arg max w ...d) + -.: c
(S7)
where is adaptive weight defined
w = wp 2 x
T (0)
0.5
N
(S8)
rescaled accuracy of the label classifier
where Wo is the pre-defined weight, 0." is the ground truth label and 0 is the predicted label of the label classifier for the sample This adaptive estimation of w, in which we multiply the original weight Wo with the rescaled accuracy of the label classifier, allows DCAL to revert to CAL once the discriminative classifier is just providing random labels. This leads to convergence to the same performance between CAL and DCAL for larger I, and preferentially picks outlier boundary cells, ignoring outlier cells that are not relevant for the classification. In this work, we refer to DCAL algorithm with the initial pre-defined weight wp DCAL-Wo.
24

---

C Benchmark details
We provide five large-scale Ca imaging datasets, totaling 40,000 cells with 160,000 annotations. These datasets include three one-photon Ca2+ movies spanning the hemisphere through 7mm window (Table one one-photon Ca movie spanning several neocortical regions (Table S2). and one two-photon Ca movie focused layer 2/3 cortical pyramidal neurons (Table S3
For the hemisphere dataset, we used triple transgenic GCaMP6f-77A-dCa mice from Allen Institute (Rasgrf2-2A-dee 1eee--2A-ddce/Caaaaaa1II Ai93). To prepare mice for in vivo imaging sessions, we performed surgeries while mice were mounted in a stereotaxic frame under anaesthesia. We created a cranial window by removing 7ymm diameter skull flap over the right cortical area SI and surrounding cortical tissue. We covered the exposed cortical surface with 7mm diameter glass coverslip. A custom wide-field fluorescence macroscope with field of view 7 mm covering the full cranial window, macro-objective lens with NA=0.5, was used for neural activities imaging. For epifluorescence illumination, we used a LED with spectrum centered 475 nm, power 1 10-40 mW. Using sCMOS camera (Hamamatsu, Orca Fusion) we acquired Ca2+ videos of neural activity (50 Hz frame rate, 1, 708 x 1, 708 pixels). We used EXTRACT to extract the neurons from these movies 13
To test ActSort on datasets with inflated false-positive rates, we used previously published imaging datasets imaging several neocortical brain regions [44]. We performed cell extraction with weak quality checking [3] 62) on one of the sessions of roughly 20 min duration, which led to 2345
false-positive regions of interest.
To record the two-photon Ca imaging movie, we employed custom two-photon mesoscope with a multi-foci illumination technique [63] This system facilitated recordings over an extensive field-of-view (2 x 2 mmm² ) at frame rate of 30 Hz and resolution of 2 μm pixels (1024 x 1024 pixels). Utilizing this equipment, we captured a movie of layer 2/3 cortical pyramidal neurons from the primary visual cortex in live triple transgenic mice from the Allen Institute. The movie was extracted using the EXTRACT pipeline [13.
With this benchmark, we tested five different active learning algorithms: random, CAL, DAL, and our newly developed algorithm, DCAL. We compared the performance of these algorithms on the three types of datasets using the following methods: 1) the canonical active learning workflow (Fig. 4A-)), 2) batch processing (Fig. E--G), and 3) cross-mice tine-tuning (Algorithm S2 Fig. 55. We evaluated performance using seven metrics: balanced accuracy, true positive rate, truc negative rate, area under the receiver operating characteristic curve, recall, precision, and F-score, as a function of the percentage of human-annotated samples. The results for the canonical active learning using the hemisphere dataset are shown in Fig. BBBD and Fig. ST3B (Table Batch processing results for the hemisphere dataset are presented in Fig. E-G and Fig. SI3C, while the cross-mice fine-tuning results are depicted in Fig. 5 and Fig. S13 D. The results for canonical active learning using the neocortical brain regions dataset (Table S2) and the two-photon microscope dataset (Table S3) shown in Fig. 59 and Fig. STO respectively.
Moreover, for the three experiments the hemisphere dataset, we also provided tables summarizing the experiments' performance for the canonical active learning (Table batch processing (Table S7p, and cross-mice fine-tuning experiments (Table (S8).
D Empirical experiments details
The experiments for large-scale Ca imaging cell sorting were conducted on an Intel Core i9 processor, utilizing parallel for loops in MATLAB. These experiments took 5 days to sort 50% of the cells across 3 datasets, each involving 4 annotators, with 10 repetitions. For batch processing experiments, we used the same Intel Core i9 processor, running sequentially in MATLAB. This process required 2 days to sort 3% of cells from a total of 28, 28,000 samples, with 6 repetitions. The cross-mice fine-tuning experiments were executed on Stanford Sherlock, utilizing MATLAB. These experiments were run in parallel across different methods and took 3 days to complete, with 3 repetitions. All cell classifier a logistic regression with ho (x) 2 0.65 predicted as cell, and hg < 0.65 predicted as not cell, see Appendix D.2 for the justification.
D.1 Estimating cell candidates' labels with cell classifier full dataset For a dataset containing N cell candidates, denoted as D = (1(1), _,)),,,,,, (N) and each cell candidate has feature dimension d, i.e., (+) Rd To estimate the predicted categorical probability
25

---

for each cell candidate by the cell classifier hg, we conducted an alternate train-test split. Firsdl, we split the dataset into set A and set B, each with 50% amount of data. Secondly, we train the cell classifier he on dataset A and estimate the categorical probability for each cell candidate (ph. (z(9) in set B. Next, we alternate set A and set B by training with data from set B and predict the label for each cell candidate in set A. Combining the second and third steps, we obtain the estimation of the categorical probability for each cell candidate without information leakage between the training and testing datasets. The final estimated categorical probability for each cell is by averaging over the 10 repetitions.
D.2 Picking the cell classifier thresholds for classification
Human annotators are prone to reject candidates that lie on the decision boundary during the annotation process. This tendency due the highly imbalanced nature of the dataset and the annotators' preference for avoiding artifacts, even if means missing a few actual cells. Therefore, using the default threshold of 0.5 does not accurately reflect the actual cell sorting process.
To find threshold that aligns with human annotation criteria, we tested various threshold values ranging from 0.1 to 0.95 as shown in Fig. [S12 using the three hemisphere datasets. The cell classifier labels a candidate x cell if hx(x)) Z B, and no cell if (ar) < B, where 3 is a predefined threshold that can be selected by human annotators. A smaller /3 results in a higher true positive rate, indicating more cells are correctly identified. On the other hand, higher results in higher true negative rate, indicating more artifacts are correctly excluded.
From Fig. S12B, E, and H we observe that, with 5% annotated samples, 3 < 0.8 is best for correctly accepting all cells, while 3 > 0.65 is best for correctly removing all artifacts. The precision graph (Fig (T2 shows that B = 0.65 most closely matches the human annotators' inherent threshold. Thus, we set the classifier threshold /3 to 0.65 for all our experiments to best reflect the human annotation process.
D.3 Feature Extraction Experiments with ResNet-50 Model
Deep learning has shown great potential in representation learning across various fields. However, training deep neural network is time-consuming and heavily dependent the size and diversity of the training data, as well as the standardization across the data. Brain recordings, particularly 1 1-photon (Ip) imaging videos, present significant challenges due to their diversity in background, imaging and experimental conditions, and cell shapes/types/size.. In this experiment, we compared eegier-en features with those derived from a deep-leaming approach.
We used ResNet-50 for feature extraction and classification on Ip calcium imaging snapshots. Specifically, we provided the extracted cell profiles and the average cropped movie snapshot, resized to 224 x 224 pixels and converted from grayscale to RGB, averaged over frames with cellular activity, to ResNet-50. We then collected 2,000 features from the final layer.
Our engineered features outperformed the deep learning approach, which, although surprisingly effec- tive at rejecting cells (and outperforming traditional features), did not surpass our engineered features as shown in Fig. [S4] The fact that deep learning-basse based feature extraction was less successful on Ip movies aligns with conclusions from prior research [12 56). Additionally, even this relatively simple experiment (extracting features from average movie frames with ResNet-50) took approximately 30 minutes for 10,000 cells using NVIDIA RTX 4090 GPU, and around 2 hours without one, which exceeds our design constraints.
We contend that our expert-engineered features, combined with the provided public dataset, establish a valuable benchmark for the cell quality control process.
D.4 Definition of boundary samples
The boundary samples for logistic regression classifier are defined as the samples that have predicted probability between 0.25 and 0.75: [(x y) 0.25 < he(x) < 0.75). The predicted probability for each sample is estimated by an average of the test set's predicted probability from 10 repeats. The test set's predicted probability obtained by i) splitting the dataset equally into two sets: set and set 2 ii) training the classifier using set 1 and outputting the predicted probability for set 2, and vice versa iii) eventually, after one iteration, we obtained the predicted probability for all samples, and we repeat this for 10 times.
26

---

D.5 Cross-mice fine-tuning
We first sort 50% of the pretrained dataset, denoted as 2.p Then we utilize the annotated labels and features from the pretrained dataset together with the new dataset's annotation to fine-tune the cell classifier. The pretrained classifier is parameterized by 0. During the fine-tuning process, the classifier is trained with oversampled concatenation of the pretrained dataset and the newly labeled dataset c)T) U L(T) at each iteration t to train the new dataset's cell classifier, parameterized by 6'. After training the cell classifier with oversampling techniques, we throw away one sample from the pre- trained dataset c/²' that has the highest classifier uncertainty, defined (1 (1-25 2 x hg (1) 0.5 The overall training structure for cross-mice fine-tuning is depicted in Algorithm S2
Algorithm S2 Cross-mice fine-tuning
Pretrained data: -,00 {(T,, U4) v:)) U {-;} 12
(0)
> Initial Labeling
Fine-tuned data: c {(T,, ((I,;,))) ) 50 {
Initial Labeling
for /. = ,,...,77 T do
Train the cell classifier (/ using -/-
Select (1+11) based on the Query-Algorithe and hg)
Human annotator label 2((++1) with y E {0,1}
c((+1)
-5 U (z((+1),
Up us Up (z(++1)
end for
Return C
for t = 1,..., T do
Train the cell classifier (0) using Randou[[versmmlligg[[
Remove (I&, uk) 4/ where 25 arg max sEC
2 x 11 (a) -0.5)
Select ((+1)) based on the Query-Algorithe and h
Human annotator label (((+1) with E 10, c((+1) - U (2(t++)),))
((+1)) Uz (oll+1)
end for
D.6 Speed decoding analysis with striatal neurons
To determine whether the quality control process, which aims to remove false-positive cell candidates from the dataset, is essential for subsequent biological analysis, we analyzed calcium imaging data previously acquired from the dorsomedial striatum of freely behaving mice using a head-mounted epifluorescence miniature microscope [60] We analyzed such movies extracting the Direct Pathway Striatal Projection Neurons (dSPNs, also noted as DI-type neurons) and 10 such movies extracting the Indirect Pathway Striatal Projection Neurons (iSPNs, also noted as D2-type neurons). The cell candidates for each movie are extracted by ICA (64 and have one human annotation for each cell candidate. The speed of each mouse is smoothed with a moving average of window size 30. We then did 1,000 train-test splits (70%-30%) and trained support vector machine regression models to predict the smoothed speeds of the animal using with ActSort trained on 1% of the human annotations. The support vector machine incorporates a (l regularizor with the hyperparameter X selected through 5-fold cross-validation. We calculated the Pearson correlation coefficient between the predicted speed and the ground truth speed for each dataset using human-rejecter candidates, human-selected candidates, and ActSort-selected candidates separately.
cells and ActSort-selected cells,
E Resources for reproducibility
Most experiments performed in this work are performed on institution's clusters, with MATLAB 2022b. Some experiments were performed on a desktop with an Intel(R) Core(TM) 19- 10900X CPU with 10 physical cores and an NVIDIA GeForce RTX 4070 Ti GPU, with MATLAB 202
27

---

If accepted, we will be open-sourcing the software, the benchmark, and the reproduction code to reproduce our results in this paper.
28

---

Table S1: The human annotation benchmark of one-photon (Ip) hemisphere dataset, 3 mice labeled by human annotators each. The datasets include 28,010 cell candidates and 112,040 annotations. The rows stand for the annotators who were used as ground truth for evaluations, whereas the columns stand for those who are being evaluated. Teal stands for the accuracy of accepting cells that were accepted by the evaluator and red stands for the rejection. Evaluators disagreed on several occasions, leading to average (balanced) accuracy of 80%. The movie images and the label distributions can be found in Fig S13
Ann.
Guinea pig
Koala
Lion
Panda
Dragon
Cheetah
Eval.
Guinea pig
NA
0.91 0.92
0.91 0.88
0.97 0.80
0.92 0.91 NA
0.97 0.69
Koala
0.99 0.39
NA
0.94 0.59
0.99 0.39
0.98 0.48
Lion
0.99 0.40
0.93 0.63
NA
0.98 0.55
0.95 0.63
0.99 0.32
Panda
0.99 0.61
0.88 0.85
0.94 0.76
NA
0.92 0.88
NA
Dragon
0.99 0.45
NA
0.93 0.72
0.99 0.50
NA
Cheetah
0.98 0.56
0.95 0.70
0.87 0.89
NA
NA
NA
0.99 0.48
0.92 0.78
0.92 0.77
0.98 0.56
0.93 0.81
0.98 0.50
Average
0.74
0.84 0.85+00005
0.77
0.87+0.06
0.74+0.07
Table S2: The human annotation benchmark of one-photon (1p) neocortex dataset with many false positives (2,345 out of 6,691 cell candidates) and 26,764 annotations. The rows stand for the annotators who were used as ground truth for evaluations, whereas the columns stand for those who are being evaluated. Teal stands for the accuracy of acceptance, red stands for the accuracy of rejection. Evaluators disagreed on several occasions, the balanced accuracy between annotators is 83%. The movie images and the label distribution can be found in Fig 59
Ann.
Guinea pig
Lion
Dragon
Cheetah
Eval.
Guinea pig
NA
0.641 0.972
0.831 0.873
0.790 0.956
Lion
0.988 0.432
NA
0.976 0.630
0.969 0.724
Dragon
0.959 0.592
0.730 0.963
NA
0.881 0,906
Cheetah
0.985 0.561
0.783 0.958
0.952 0.784
NA
0.977 0.529
0.718 0.964
0.920 0.762
0.880 0.862
Average
0.75+10.09
0.84+0.05
0.84+0.04
0.87=00.3
Table S3:
The human annotation benchmark of two-photon (2p) C movie with 5,276 cell candidates and 21,104 annotations. The rows stand for the annotators who were used as ground truth for evaluations, whereas the columns stand for those who being evaluated. Teal stands for the accuracy of acceptance, red stands for the accuracy of rejection. Evaluators disagreed on several occasions, the balanced accuracy between annotators 78%. The movie images and the label distribution can be found in Fig S10
Ann.
Cheetah
Panda
Koala
Dragon
Eval.
Cheetah
NA
0.917 0.782
0.780 0.850
0.888 0.801
Panda
0.969 0.560
NA
0.802 0.783
0.921 0.770
Koala
0.975 0.343
0.949 0.441
NA
0.924 0.474
Dragon
0.971 0..92
0.953 0.660
0.807 0.722
NA
0.971 0.465
0.940 0.628
0.796
0.785
0.911 0.682
Average
0.72+0..1
0.78
0.80+0.01
0.80+00.06
29

---

Table S4: The significance levels for the Wilcoxon signed-rank tests with Bonferroni- Holm corrections performed in Fig. BB. Each query method had twelve data points datasets and annotators each). For each annotation, we computed the fraction of the boundary samples.
Method
Random
CAL
DAL
DCAL-0.3
DCAL--0.5
DCAT.-0.7
DCAL.00.9
Random
N/A
0.0103
0.4023
0.0098
0.0093
0.0088
0.0083
CAL
N/A
N/A
0.0078
1.0000
1.0000
0.1665
0.5195
DAL
N/A
N/A
N/A
0.0073
0.0068
0.0063
0.0059
DCAL-003
NIA
N/A
N/A
N/A
1.0000
0.3252
0.8813
-CAL-0.5
N/A
N/A
N/A
N/A
N/A
0.2686
0.4629
DCAL-007
N/A
N/A
N/A
N/A
N/A
N/A
1.0000
Table S5: The significance levels for the Wilcoxon signed-rank tests with Bonferroni-Holm corrections performed in Fig. 3D. Each query method had twelve data points (3 datasets and 4 annotators each). For each annotation, we computed the average cosine distances between the boundary samples. To facilitate fair comparison, for each query algorithm, randomly subsampled the number of boundary samples to match the query algorithm with the lowest number of boundary samples.
Method
Random
CAL
DAL
DCAL-0.3
DCAL-0.5
DCAL-0.7
DCAL-0.9
Random
N/A
1.0000
0.0103
1.0000
0,8145
0.6172
0.6460
CAL
N/A
N/A
0.0098
0.4702
0.0093
0,0146
0.0479
DAL
N/A
N/A
N/A
0.0088
0.0083
0.0444
0,0078
-CAL-0.3
N/A
N/A
N/A
N/A
0.7568
0.0820
0.2310
DCAL
N/A
N/A
N/A
N/A
N/A
0.3418
0.7764
DCAL
N/A
N/A
N/A
N/A
N/A
N/A
0,8501
Table S6: ActSort performance after sorting 5% of the hemisphere Ca datasets (across mice). Rows, evaluation metrics, where ACC is balanced accuracy, TPR is true positive rate, TNR is true negative rate, and AUC is the area under the curve. Columns, active learning method, where the fraction next DCAL refers to the corresponding weight. Each entry shows the average metric value for sorting up to 5% across 12 annotators over three datasets. The entire sorting performance as function of the percentage of sorted cells up to 50% is illustrated in Figs. BBD and S13B.
Metrics.
ACC
TPR
TNR
Precision
F-score
AUC
Methods
Human
0.8354
0.9520
0.7189
0.9731
0.9614
NA
Random
0.8276
0.9476
0.7075
0.9720
0.9585
0.9353
CAL
0.8391
0.9649
0.7133
0.9728
0.9681
0.9423
DAL
0.8376
0.9508
0.7245
0.9735
0.9610
0,9426
DCAL-0.3
0.8396
0.9635
0.7156
0.9730
0.9675
0.9416
DCAL-0.5
0.8415
0.9641
0.7189
0.9733
0.9679
0.9421
DCAL-007
0.8390
0.9638
0.7141
0.9729
0.9675
0.9374
DCAL-009
0.8385
0,9638
0.7132
0,9728
0,9675
0,9364
30

---

Table S7: ActSort performance on 2% of the batch sorted cell candidates. Rows, evaluation metrics, where ACC is balanced accuracy, TPR is true positive rate, TNR true negative rate, and AUC is the area under the curve. Columns, active learning method, where the fraction next to DCAL refers to the corresponding weight. Each entry shows the average metric value for sorting up to 2% across 64 augmented annotators. The entire sorting performance function of the percentage of sorted cells up to 5% is illustrated in Figs. EEGG and SI3C.
Metrics.
ACC
TPR
TNR
Precision
F-score
AUC
Methods
Human
0.8354
0,9520
0.7189
0.9731
0.9614
NA
Random
0.8349
0.9545
0.7153
0.9733
0.9634
0.9391
CAL
0.8398
0.9715
0.7080
0.9730
0.9720
0.9434
DAL
0.8373
0.9601
0.7144
0.9733
0.9663
0.9453
DCAL-0.3
0.8431
0.9716
0.7146
0.9736
0.9723
0.9436
DCAL-0.5
0.8444
0.9713
0.7175
0.9738
0.9723
0.9439
DCAL-0.7
0.8465
0.9708
0.7222
0.9742
0.9723
0.9444
DCAL-009
0.8455
0.9703
0.7208
0.9741
0.9719
0.9433
Table S8: ActSort performance on fine-tuning experiment with 1 annotation. Rows, evaluation metrics, where ACC is balanced accuracy, TPR is true positive rate, TNR is true negative rate, and AUC the area under the curve. Columns, active learning method, where the fraction next to DCAL refers to the corresponding weight. Each entry shows the average metric value for sorting to 1 1% across 96 augmented annotators over dataset pairs. The experimental details are described in Fig.
5 and the entire sorting performance function of the percentage of sorted cells up 20% are
illustrated in Fig. B-D and Fig. SI3D.
Metrics.
ACC
TPR
TNR
Precision
F-score
AUC
Methods
Human
0.8354
0.9520
0.7189
0.9731
0.9614
NA
Random
0.8112
0.9363
0.6860
0.9702
0.9513
0.9208
CAL
0.8261
0.9551
0.6970
0.9714
0.9621
0.9131
DAL
0.8300
0.9346
0.7254
0.9735
0.9521
0.9335
DCAL-003
0.8323
0.9515
0.7130
0.9727
0.9608
0.9170
DCAL-0.5
0.8337
0.9509
0.7165
0.9730
0.9606
0.9186
DCAL-0.7
0.8346
0.9509
0.7183
0.9731
0.9607
0.9194
DCAL-0099
0.8364
0.9503
0.7225
0.9735
0.9605
0.9211
31

---

Dynamic Cell Label Visualization
Control Tabs
Gurert Cel
Good
Utiioveed
5228
Dynamic Movie
Libeld
23
Snapshots During Ca events
Plot of the Ca Activity Trace
100
Figure $1: ActSort has user-friendly cell sorting software for experimental neuroscientists. The user interface of ActSort offers options watch movie snapshots, study Ca activity traces, zoom into the maximum projection image of the movie, and has several other controls detailed in Appendix ActSort is available MATLAB apps and can be used on personal laptop thanks the data compression module.
Preprocesing Module
Ca imaging movie
Graphic user interface (GUI) features
Cell classifier features
Spatial profiles
Spatial
features
mm
Ca
Temporal
Cell candidates from the cell extraction algorithm
activity traces
features
Spatio-
Movie
temporal
snapshots
features
Figure S2: A visualization of ActSort's preprocessing module. In response to the challenge of managing rapidly increasing file sizes in Ca imaging movies and cell extraction results -often reaching TBs the preprocessing module is crucial for enabling memory efficient real-time pro- cessing. This module efficiently compresses the input data to just a few GBs, facilitating easy data transfer and use on personal laptops. Left. The inputs to the preprocessing module include the Ca²+ imaging movie and the cell extraction results. Middle. The processed data are transformed into CUl-compatible features such as spatial profiles, Ca activity traces, and Ca movie snapshots. These features are then utilized by the GUI module to support human annotation decisions. Right. The input data are converted into features, quality metrics that quantify the properties of individual cells (Method; Appendix B. These features are essential for training the classifiers later in the pipeline.
32

---

Looding
Lamtiry
Loating
Actra Leiiing
Zooming Options
Fraco Cola
Aative Settings
Uia
valto
AdBort Buoe
Tot Call
Co
Navigation
Col Barder Color
Aodlur Docken
Adtive Leering Deciaion Control
Un dell
Control
Heltice Catee.
Unoutter Cale Oily
General Stallstes
Activa Statislics
Snapshot
Libeot
Contral
Figure S3: The control tabs within the ActSort software. A visualization highlighting the broad functionalities of the ActSort software. Left. In the labeling tab, the user can navigate between cells using slider and make decisions by clicking on buttons. These shortcuts and by clicking on the cells from the cell map (Fig. STT. Middle. The tools tab allows the to zoom into the map, which can be performed via sliders or by clicking with the mouse. The user also has the option to adjust the map/snapshot contrasts, perform multiple selections, and many other utilies discussed in Appendix B Right. The active learning tab allows the user monitor the progress of the active learning algorithm. The user has the option to let the active learning make the decisions for the unsorted cells, which can be previewed on the cell map. Finally, the training frequency of the decoders can be decreased for exceptionally large datasets to allow faster sorting. This feature was designed mainly allow scalability and further development, i.e., for future active learning algorithms that may need more time to infer the next query.
also be achieved by keyboard
Area under the curve
Accuracy of matching annotator's 'ss cell rejection preference
n.a.
Accuracy of matching
acceptance preference
annotator's cell
0.9
0.5
0.5
Resnet-50 features Engineered features
Figure S4: Features derived from Resnet-50 failed match the performance of expert- engineered features. To test whether pre-trained deep networks can be used as cell classifiers, we once again used the hemisphere dataset. We ran cropped Ca imaging movie snapshots, av- eraged across frames at the time of cells' 2+ events, and cells' extracted spatial footprints pre-trained Resnet-50 model. We extracted the resulting 2,000 features from the penultimate layer, which we term as "ResNet-50 features." Left. Similar to Fig. A we trained cell classifiers (Method; Appendix D.I on the engineered and ResNet-50 features. Expert-engineered features outperformed ResNet-50 features, leading higher area under the receiver operating curve. Middle. The true negative rates obtained from the engineered or ResNet-50 features were mostly identical. Right. The expert-engineered features led to higher true positive rates, whereas ResNet-50 features rejected 10% of true cells, far below the human performance. Overall, our engincered features outperformed the features extracted from the Resnet-50 model. Each dot corresponds to single annotation instance. All tests are two-sided Wilcoxon signed-rank tests (***p p < 10-3 **p 10-2 "p < 0.05).
33

---

Annotator Guinea Pig
Annotator Lion
Annotator Dragon
Annotator Cheetah
probability being a cell
Classiffer predicted
Empirical probability computed from annotator agreements
Fraction candidates identified as cells within respective groups x By the human annotaor
By the classifier
Figure S5: Cell classifiers can mitigate human annotator bias. We trained the cell classifier on the neocortex datasets (Method; Appendix D.I) and predicted the test probabilities for each cell candidate. The empirical probability for each cell computed from annotator agreements (as discrete values: 0, 0.25, 0.5, 0.75, 1). Each plot corresponds to annotator. The performance for each annotator can be found in Table S2 Each dot represents a single cell from the neocortical dataset 44 Red lines, median; purple dots, mean of the corresponding distribution; red cross, mean of the corresponding human annotations' distribution. The classifier shows a better-calibrated prediction, while the human annotators (especially first two researchers) display strong biases.
Annotator Tacoūräc
Repeat model evaluation for annotators
Annotator
Annotator 4
Annotator 2
Labels via majority vote
Evaluator
Evaluation metrics
Claseifier accuracy
Cell classifier trained real-time
training
Annotator
Repeat model evaluation augmentated annotators
Annotator
Annotator
Annotator
Annolator
Cell classifler trained real-time
AL training
Dataset
Dataset
Dataset 3
Dataset
Evaluation
Classifier accuracy
mejority vote
mejority vote
majority vote
Evaluator
metrics
Annotator accuracy
Figure S6: A description the evaluation methods used in the main text for benchmarking. A Our workflow for testing online classifier' accuracies when trained with the labels from a distinct annotator. The majority votes from the rest of the annotators are used in place of the ground truth, which we call "evaluators." This workflow is used in all the experiments in Section 2.5) For experiments testing across mice generalization, the creation of annotator and evaluator repeated twice for both the pre-training stage and the fine-tuning stage. B When augmenting new annotators for experiments involving multiple imaging datasets, we first computed the evaluators through the majority vote for each annotator and concatenated both to obtain augmented annotators and evaluators. This way augmentation is used for Figs. E-G and SI3C.
34

---

A
B
d.al-0.3
dcal-0.5
Predision and Recall
0.98
Precision
(with adaptive weight)
Adaptive weight
0.8
dcal-0.7
dcal-0.9
Precision
0.6
(without adaptive weight)
0.96
0.4
Rlccll
(with adaptive weight)
0.2
Recall
10" 0.94
(without adaptive weight)
10
10
0.5
annotated cells
DCAL weight
Figure Adaptive estimation of the model parameter leads improved true positive rates. To study the role of the adaptive weight estimation in DCAL (Appendix(B.3) we visualized the evolution of the weight parameter throughout the sorting process of the hemisphere dataset and quantified the effectiveness of the adaptive weight strategy. A The adaptive weight value recorded ran ActSort on the hemisphere dataset, sorting up to 20%. The sudden drop in the weight values when only few samples sorted is due to insufficient training data, leading class imbalance. Fortunately, the adaptive estimation decreases the sensitivity to the initial choices of w. Solid lines: means. Shaded areas:
over 12 annotators. B We computed the precision and recall values after running ActSort with DCAL (with and without adaptive estimation) on the hemisphere dataset, sorting up 3% cell candidates. The weights were initially swept within the interval [0.02, 0.98). and later binned with a window size of 0.1 due to high variability and low number of annotations. Solid lines: precision (blue) and recall (also known as positive rate, red) values with adaptive weight estimation. Dashed lines: The same, but without adaptive estimation. Without adaptive weight estimation, DCAL became suboptimal with increased weight values, leading decreased true positive rates.
random
cal
dal
dcal-0.3
dcal-0.5
dcal-0.7
dcal-O.9
1%
5%
10%
PLS reduced dimensions
cell
not cell
Figure S8: The distribution of selected samples from various query algorithms. This is an extended visualization for Fig. 3A, but for 1%, 5%, and 10% annotated datasets.
35

---

Nents 4346
=P 2345
Balanced Accuracy
0.8
0.7
0.81
Cell
Not cell
0.6
50
E
F
1 mm
True Negative Rate
True Positive Rate
0.9
AUC
0.8
0.9
0.8
0.8
0.95
0.72
0.7
50
Precision
F-score
Recall
0.9
0.8
0.8
Human annotator
0.88
0.85
0.1
0.2
0.3
0.6
50
50
Precision
Percentage of human annotated cells
random
cal
dal
dcal-0.3
dcal-0.5
dcal-0.7
dcal-0.9
human
Figure S9: ActSort improves cell sorting quality in movies with many false positives. To assess the generalization ability of ActSort in one-photon Ca imaging movies with many false positives, we ran the active learning experiments on neocortex dataset (Method; Appendix C A Example cell map of the neocortical brain regions (44] Green lines indicate cell boundaries, while the red line indicates false positives. Supervised partial least squares (PLS) plots for all samples, similar to Fig. 3A. illustrating the separability of true and false positives by the engineered features. C-H The results of the active learning- training on individual annotations. Example precision-recall (PR) curve for the best annotator's 's annotation instance. The best annotator selected the one who has the highest average accuracy from Table S2 Solid lines: means. Shaded areas: s.e.m. over four annotators. Dashed lines and gray areas: mean and s.e.m. from four human annotations.
36

---

A
N Calls = 4431 N 845
B
Balanced Accuracy
0.8
0.7
Cell
Not cell
0.6
50
D
E
F
0.5 mm
True Negative Rate
0.8
True Positive Rate
0.9
AUC
0.94
0.8
0.8
0.6
0.7
0.9
0.85
0.7
0.5
0.6
50
50
50
Precision
0.9
F-score
0.9
Recall
0.52
0.8
0.8
0.9
0.92
0.18
0.7
50
0.7
50
precision
Percentage human annotated cells
random
cal
dcal-0.3
dcal-0.5
dcal-0.7
dcal-0.9
human
Figure S10: ActSort can process two-photon Ca imaging movies with residual motion. With
ActSort, we processed example two-photon Ca imaging movie recording layer 2/3 pyramidal cells in primary visual cortex (Method; Appendix C The movie had uncorrected residual motion by design, which led to several false positives in the form duplicates. B-1 Same as Fig. S9)
37

---

With 1% annotation
B
With 5% annotation
C
With 20% annotation
0.8
0.8
10
10
10
10
10
10a
10
10
10'
10
10
10"
10a
D
0..
0.9
Accuracy
0.7
0.7
0.7
0.5
0.5
0.5
10a
10a
10a
10"
10'
10
10-a
10"
10a
10
10a
10*
10*
10
10
Relative regularization parameter
random
dal
dcal-0.3
dcal-0.5
dcal-0.7
doal-0.9
Figure S11: The insensitivity to the regularization strength allows rapid training without cross-validation. Our emphasis on real-time training necessitated using a pre-defined regularization strength, which was taken to be Addault 1/ (Number of Samples). To validate this, reanalyzed the neocortex dataset with varying levels of regularization parameters for the cell classifiers. The first row illustrates the under the receiver operating curve (AUC) for the cell classifier's predictions different stages of annotation %, 5%, and 20% cell candidates with Arelativ = 1 corresponds the default value for the problem of interest. The second shows the balanced accuracy of the classifier's 's predictions at these same annotation stages. Solid lines: means. Shaded areas: s.e.m. over four annotators. Red dashed lines: performance metrics at the default regularization value.
38

---

A
With annotated
B
5% annotated
C
With 10% annotated
0.88
0.88
Balanced Accuracy
0.84
0.84
0.84
0.8
0.76
0.76
0.76
0.72
0.30
0.50
0.70
0.90
0.72
0.30
0.50
0.70
0.90
0.72
0.10
0.30
0,50
0.70
0.90
E
F
True Positive Rate
0.9
0.9
0.30
0.50
0.70
0.90
0.8
0.30
0.50
0.70
0.90
0.8
0.30
0.50
0.70
0.90
G
H
0.9
0.9
True Negative Rate
0.8
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.10
0.30
0.50
0.0
0.90
0.10
0.30
0.50
0.70
0.90
0.10
0.30
0.50
0.70
0.90
K
L
0.99
0.99
0.99
Precision
0.97
0.97
0.97
0.95
0.30
0.50
0.70
90
0.95
0.50
0.70
0.90
0.95
0.10
0.30
0.50
0.70
0.90
Percentage human annotated cells
random
dcal-0.3
dcal-0.5
dcal-0.7
dcal-0.9
Figure S12: ActSort performance on various classifier thresholds. A sampled collected from the decision boundary should be equally likely be true or false positive. contrast, we found that human annotators tend to reject cell candidates close to the decision boundary, diminishing the efficacy the default 0.5 threshold for the cell classifier. The need real-time training compelled us to establish predefined classifier threshold. We conducted additional active learning experiments on the hemisphere dataset and assessed the classifier's balanced accuracy, true positive rate, true negative rate, and precision at different annotation stages-1 5%. and 10%- under different classifier threshold, ranging from 0.1 to 0.95. In almost all cases, a threshold between 0.6 and 0.7 aligned well with human annotation preferences. Solid lines: means. Shaded areas: s.e.m. over 12 annotators. Dashed lines and gray areas:, means and s.e.m. from human annotations.
39

---

A
Noa. 6272 Mxp 521
Na. 9029 Ao 954
Non. 10376 ND 858
mm
mm
mm
Hemisphere active learning
Precision
F-score
AUC
0.97
0.9
0.96
0.9
0.9
50
50
50
Batch processing
0.98
Precision
0.97
F-score
AUC
0.9
0.98
0.9
0.96
0.8
0.96
0.96
0.95
0.8
0.7
Cross-mice fine-tuning
0.98
Precision
0.97
F-score
AUC
0.975
0.9
0.97
0.9
0.96
0.87
0.95
0.8
0.8
20
20
Percentage human annotated cells
random
deal-0.3
dcal-0.5
dcal-0.7
dcal-0.9
human
Figure S13: ActSort performance various types of experiments outperform human annota-
Example cell map from 7mm window hemisphere across three mice (Method: Appendix C
Green lines, cell boundaries; red lines, false-positives. B-D The remaining evaluation metrics for (B) Fig. 4B-D, (C) Fig. 4E-G, and (D) Fig. including recall, precision, F-score, and area under the receiver operating characteristic curve.
40

---

A
B
fruth
Smoothed speed predicted
Smoothed speed predicted ActSort
dSPNs
iSPNs
10
smoothed speed
Pearson's &s correlation coeffiiiiet (1) predicted and ground truth speed
annotated calls
annotated cells
Speed $*)
0.5
0.5
800
1200
Human rejected candidates
Humen selected candidates
ActSort (1%) annotated cells
Figure S14: Culling false-positinn is necessary to prevent spurious Ca signals leaking into data analysis. As a of whether omitting cell sorting can lead spurious biological conclusions, we studied Ca imaging data that was previously acquired in the dorsomedial striatum of freely behaving mice using a head-mounted, epifluorescence miniature microscope (6). To test whether the human annotated and the ActSort predicted cells (trained with CAL and using only 1% annotation for each mouse separately) provided similar levels of speed decoding, we trained support vector machines to predict the (smoothed) speeds of the animals. A Visualization of speed decoding from an example animal. B-C both dSPNs (B) and iSPNs (C), after controlling for cell numbers, human annotated cells provided better speed decoding compared to human rejected cells, and were equivalent to ActSort sorted cells. More importantly, however, cells that were rejected by human annotators were informative regarding the speed of the animal. This means that in brain regions not coding for speed, cell extraction artifacts, likely due to brain motion, can encode animal's 's behavior spuriously, potentially leading incorrect biological conclusions. In C, Each dot corresponds to results from a single animal, averaged across 1,000 speed decoders trained from distinct train-test splits of the data. All tests are two-sided Wilcoxon signed-rank tests (** p 10-3 ""p 10-2, "p < 0.05).
41

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided a proper justification
given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No]" " "o "[NA] " not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction listed the four main contributions, unique set of engineered features to aid classifier training; a benchmark of cell sorting for calcium imaging dataset; user-friee -riendriy publicly available cell sorting for calcium imaging dataset; the first to introduce active learning paradigm to calcium imaging; and a novel active learning query algorithm.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
42

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the discussion section, we talked about the limitations of cell classifier's architecture, the binding of ActSort with current existing cell extraction algorithms, and annotators' expertise are different.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
43

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The details of feature engineering, active learning algorithms, and experimental designs are all described in details either in the main text or the appendix. The code and the data can be found inside the public github repo.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We published all the dataset, code, and software with sufficient instructions to reproduce the main experimental results. The dataset is processed following standard strucutre and will be prepared with DCAT format for release. The code will be released with
44

---

detailed README, clear documentations, and both training and evaluating code. All can be accessed through the github repo url we provided the main text.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ public//qiides/Coeetttmi www.iioiiisiiiiiooooooooioliiiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The paper describes the hyperparameters and threshold used on all experiments in the appendix D Empirical experiments details.
Justification:
Guidelines: We described the classifiers and query algorithms details in the appendix.
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The bar plots include the error bar as the standard error and the two-sided Wilcoxon signed-rank test, All the curves include the standard error as the shaded area.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
45

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the computation resource in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: Yes]
Justification: There no potential harmful consequences in this research. This work aim to assist experimentalists in system neuroscience.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
46

---

Answer: [Yes]
Justification: We discussed the potential in decreasing human effort the tedious cell sorting process in the paper. The release of ActSort is able to decease human annotations from hours to minutes, thus pushing the advancement in the related fields of work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
[NA]
Justification: This work includes no high risk for misuse. This work provides experimental- ists software tools to curate extracted cell candidates.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
47

---

Answer: [Yes]
Justification: The dataset is created all within the lab. All assets are credited to either the published paper or the author listed in this paper. The utilization of open source github repo developed within the lab is also credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprssiitcooeeeooocapp. thoode com datastts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: We included the details of the dataset, including the structure and training, and we published the dataset our public github repo
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
We do not involve crowdsourcing research on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
48

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
49


Reviews:
[{"review_id": "KpYNP3r8aQ", "reviewer": "Reviewer_WHFf", "summary": "In this paper, the authors develop and open-source a software, ActSort, for cell sorting of large-scale calcium imaging datasets in neuroscience which integrates domain-expert features with an active learning framework. Alongside with the software, the authors provide a new benchmarking dataset which they use to evaluate their newly developed active-learning query algorithm that forms the backbone of ActSort. The active-learning based algorithm seeks to reduce and most efficiently use human annotator's time by interleaving automated cell classification with queries for human labels of most informative outlier boundary cells. The authors provide extensive benchmarking and evaluation of their approach across different real-world experimental conditions. In doing so, they demonstrate that ActSort significantly reduces the number of human-provided cell labels necessary to achieve sufficient true positive/negative rates, and thereby constitutes an important step towards alleviating the effect of this bottleneck in processing large-scale calcium imaging datasets.", "questions": "**Major comments and questions**\n\nFig. 2: Please give more guidance in the legend for interpreting the plots (A: provide definition of absolute discriminability index); B: differences seem minimal. Did the increase in accurate rejection of false-positive candidates gained by inclusion of novel features come at the cost of increased false-negatives?\n\nFig. 3D: It seems surprising that feature distance does not differ substantially between the different dcal versions. Can this be explained by showing the evolution of w during the adaptive estimation process?\n\n**Minor comments and questions**\n\nln. 149 following: this sentence is broken.\n\nFig. 3A: Provide legend indicating what the colours mean; visualize the decision boundary.", "limitations": "As mentioned above, a discussion of limitations of the presented approach (perhaps in comparison to other approaches) is lacking from the discussion.", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The presented work is excellent in originality, quality and significance, with sufficient clarity. \nThe main strength of the paper is in its core contribution, the development of DCAL, an active-learning query algorithm that combines the advantages of confidence-based and discriminative active learning. This combination leads to the query algorithm selecting outliers (DAL) near the decision boundary (CAL) to be labeled by the human annotator. \n\n**Originality**\n\nThe paper makes several original contributions to the field of cell sorting, the most valuable of which seems to be the 1) presentation of a novel active learning query algorithm that combines confidence-based and discriminative active learning and 2) the application of this algorithm to the problem of cell sorting in the form of 3) a GUI-based open-source software.\n\n**Quality** \n\nThe described benchmarking datasets and the algorithm, as well as the experimental designs for empirical evaluation are of high quality, in that they are large-scale, well-motivated and well-executed, respectively. \n\n**Significance**\n\nThis work is potentially very impactful in the field of systems neuroscience, if the claimed contributions, particularly the substantial reduction in the required number of human-labeled cells, mitigation of annotator bias, user-friendly design of software,  and generalization capabilities hold true in deployment.\n\n**Clarity**\n\nThe paper is well-motivated, clear in writing and provides extensive supplementary material. However, improvements are necessary (see Comments/Questions).", "weaknesses": "The main weaknesses of the paper are a somewhat shallow discussion and lack in accessibility for the putative audience, that is, experimentalists with less experience in reading technical papers. \n\n**Discussion**:\n\nInstead of a discussion, authors provide a conclusion that reiterates the contributions of the paper. It would be better to instead discuss potential limitations of the presented methods; interpret some of the more surprising results (see questions); discuss how much modification would be necessary to extend the approach to multi-class data, and data collected with different types of indicators (e.g. voltage-imaging). \n\n**Accessibility**:\n\nWhat I particularly like about the paper is that all components of the proposed active-learning based query algorithms are well-motivated and interpretable. It would be valuable to make the components of the algorithm (i.e. the CAL and DAL components and the adaptive estimation of w) more accessible, by adding interpretation aids (verbal and visual) to eqs. 3 and 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our manuscript and providing invaluable feedback. You caught everything in the paper, your suggestions were extremely helpful, and your review led to new experiments that increased the readability and impact of our paper! To us, this was an extremely well written, helpful, and insightful review. Thank you!\n\n**Accessibility for experimentalists** Thank you for highlighting this critical point while evaluating our work! We have prepared a user manual and lecture videos (with tutorials) for ActSort, but due to the anonymity requirements, we cannot share them yet. We are committed to providing a user-friendly introduction to ActSort, which will hopefully ease the experimental neuroscientists into the software. Currently, ActSort has a small user base of 20+ researchers globally (mainly collaborators), and we are actively collecting their feedback to further improve the user experience.\n\n**Discussion section** The reviewer is absolutely correct. Thanks to the additional page provided after the revisions, we now added a new discussion section. We are happy to share the full text if requested, but for brevity, we will list the discussion bullet points below:\n\n- ActSort is an active learning accelerated cell sorting pipeline for large-scale 1p and 2p calcium imaging datasets, which is compatible with any cell extraction algorithm (CAIMAN, Suite2p, EXTRACT, ICA, etc.).\n- Since ActSort is a quality control algorithm, any mistake made in the cell extraction process (for example, missing cells and/or motion artifacts) would automatically propagate to the cell sorting process. Though we have plans to incorporate merging of duplicate cells in the future, the current version does not support this. Thus, we require that movies are motion corrected and cells are properly extracted with your favorite cell extraction algorithm.\n- ActSort makes no assumption on the type of behavioral experiment, the Ca2+ indicator, the imaging conditions; and is robust to variations thanks to its standardized features.\n- Though currently validated for calcium imaging experiments, ActSort can be minimally modified (perhaps with the addition and subtraction of new features) to be applied to newly emerging technologies such as voltage imaging and/or multi-class datasets (for example, including dendrites). \n- Our optimally compressed data format has implications for sharing Ca2+ imaging datasets publicly. To date, only post processed activity traces were shared, but with our format, now movie snapshots can be shared too, allowing end users to check the quality of the data.\n\nIf you have additional comments, please let us know!\n\n**Clarification for query algorithms** Great suggestion! We added an explanation to equation (3) as “The uncertainty-score $c_i^{(t)}$ represents the uncertainty regarding the sample $i$’s position relative to the decision boundary. The higher the score, the closer it is to the decision boundary. The discriminative-score $d_i^{(t)}$ represents the uncertainty regarding whether the sample $i$ faithfully represents the full dataset. Higher scores indicate the sample is unique and underrepresented by the labeled dataset.”\n\nWe added an explanation to equation (5) as “ $w$ approaches to the predefined weight $w_0$ if the labels classifier $p_\\phi^{(t)}$ correctly differentiates between labeled and unlabeled data, indicating unique samples in the unlabeled dataset. $w \\to 0$ if there are no underrepresented samples in the unlabeled dataset, resulting in the query algorithm selecting only the boundary cells as the CAL algorithm does.\n\nTo address your concerns about adaptive updates, we added a new figure (Fig. N5). In the beginning, when only a few samples are sorted, there is an initial drop in the weight values. This drop occurs because the label classifier fails to differentiate between labeled and unlabeled samples. This indicates that initially, the CAL component dominates the sample selection process. As the process continues, the DAL component starts to take over, selecting unique and underrepresented samples from the unlabeled dataset, and finally converging back to CAL again.\n\n**Questions about Fig. 2:** Thank you for pointing out the confusion regarding the absolute discriminability index. We added an illustration to explain its definition in the PDF (see Fig. N1). The absolute discriminability index is a metric used to quantify the effect size between two distributions. If the two distributions can be easily told apart, we will obtain a higher d' value. The two distributions here are feature values conditioned on whether a sample is a cell or not. Additionally, we included a statistical analysis comparing the effect sizes of traditional features and new features (Fig. N2). \n\nThe illustration on Fig. 2 B makes the difference seem minimal. The AUC changes from 0.94 to 0.97. We will update the figure to have a zoom in version. We added a sentence explaining that the increase in true negative rate did not sacrifice the accuracy of true positive rate - “Re-analyzing the dataset from Fig. 2 B, we found that the new features increased the rejection accuracy from 67% to 75% (Fig. 2 B) without decreasing the accuracy of accepting true-positives (97% to 97%), leading to more effective separation between cell and not cell samples in our benchmarks (Fig. 2 A). ”\n\n**Question about Fig 3D:** Wonderful suggestion! To address this point, we changed the feature distance measurement from euclidean distance to cosine distance, which has a much better dynamical range (Fig. N3). \n\n**Typos:** Thanks for pointing out our typos! We will revise the writing! The Fig 3A legend is placed under Figure B-D. We moved the legend above to make it more accessible for the reader as you suggested. \n\nThanks to your feedback, we were able to improve the clarity of our work. We hope that you are satisfied with our responses and consider supporting our submission with a very strong acceptance!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cMIjIpytlT", "reviewer": "Reviewer_h7Vq", "summary": "The paper proposes an active learning framework for improving the accuracy of cell sorting in calcium imaging datasets. The method rests upon three main components: (1) Preprocessing module which uses an already existing cell segmentation algorithm and reduces the size of the dataset using a set of engineered features. (2) Cell selection module which allows the annotator to visualize the features of specific detections and label them as cells or no cells. (3) Active learning module which trains a cell classifier (whether or not a detection is a cell or not) and a label classifier (whether or not a cell is labeled or not) and uses a discriminative confidence-based strategy to select the next cell for annotation. The authors show improved results over random and mere confidence-based strategies and demonstrate that the model surpasses human performance with a small number of annotations. A benchmark cell sorting dataset and software are also accompanied by the paper as other parts of the contribution.", "questions": "- It appears that the preprocessing step selects a number of cells that are then used in the software for the human annotators to label as *cell* or *not cell*. Therefore if the preprocessing step (e.g. EXTRACT algorithm) misses some of the cells there’s no recovery. Can the authors include a discussion of this in the paper?\n\n- In addition to this, if there are imperfections such as residual motion in the videos the preprocessing step might mark a cell as multiple cells across different frames of the video (a common issue in tracking which requires stitching *tracklets*). Is there anything in the presented framework that allows for the recovery of these mislabelings? Related to this, if there's motion in the videos, other metrics developed in the multi-object tracking community are used to assess the quality of identifying a cell and maintaining its identity throughout the video. How does your framework deal with cell identity losses? Are these issues redirected to the preprocessing step?\n\n- This question might be a high-level yet naive question given that I haven't caught up with the latest advances in cell sorting literature. How does the method compare against, say, a fully supervised approach where you collect multiple datasets and train a large model (say a vision transformer or a UNet) for sorting the cells? Given the advances in AI and the foundation models, this seems to be a natural direction to pursue.\n\n- How does the classifier compare with an architecture well-suited for vision that takes in a zoomed-in crop of the image (or video) of a cell and predicts if it's a cell or not? This control would be important to support the argument for engineered features and a simple classifier.", "limitations": "The technical novelty of this paper is limited. The simplicity of the methods is a double-edged sword improving the readability yet making the paper a less than ideal candidate for a technical conference such as NeurIPS.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The problem considered is significant for the neuroscience community. Improved cell sorting methods could save significant amounts of time for experimentalists and allow them to spend their time on more specialized tasks.\n\n- The paper is written very clearly. I appreciate the clarity of the introduction and motivation of the paper. The summary of the contributions is fair and straightforward and solves a well-defined and well-motivated problem.\n\n- The figures are informative and professional. They include the necessary information in the main text and wherever the extra information is not crucial the information is presented in the supplementary.\n\n- The results are thorough and significant. The methods show clear advantages of different components of the contributions (features used and DCAL active learning strategy) across multiple scenarios.", "weaknesses": "- The main weakness of the paper is the technical part which does not offer a novel contribution. In principle, the fact that a simple model leads to the presented improvements is a strength of the paper. But given the simplicity of the technical parts of the paper one might argue that a specialized field journal could be a better venue for such a contribution than NeurIPS.\n\n- There are no comparisons performed against any other method and all the comparisons are against different versions of the same model. What is the state of the art for this problem and what are the competing methods? How do they compare to your proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and effort in reviewing our manuscript and providing insightful feedback. We have addressed your concerns regarding where ActSort stands on within the entire calcium imaging processing pipeline and explained our fit into the NeurIPS venue in the *General Response*. Note that we omit citations from text excerpts due to character limits. \n\n**Prior work:** Great point! We should have done a better job in explaining that 1p movies are non-standard and lack specificity. As a result, there is no published baseline for the problem we are aiming to solve in the field. We are introducing the first benchmark on these types of movies. The current method for the curation process of removing false positives rely entirely on human laborers or other pipelines use simple thresholding based on what we call traditional features (CAIMAN, ICA, EXTRACT), or classifiers defined on these features (Suite2p). Please also refer to “The concerns about comparison to prior work” section in the general response for more detailed explanations. \n\n**Imperfections in the data such as motion correction, cell extraction, stitching, cell identity** You are absolutely right about all these concerns. To recap, there are two distinct problems when processing calcium imaging movies. \n\nThe first problem is the cell extraction, in which the motion should be corrected, cells should be properly identified, cells suspected of duplication should be stitched or removed, and/or cell identity should be faithfully tracked across sections. Field has been working on these problems over two decades. \n\nActSort comes as a quality control (second problem) pipeline on several existing pipelines designed to address these aforementioned problems. Traditionally, quality control on these outputs was done by experimentalists manually, by going over each cell with custom softwares. However, with thousands and millions of neurons being recorded in a single session nowadays, automated quality control became necessary. Yet, we should have addressed this in the discussion for the broader ML community, which is what we do now as follows:\n\n“The quality control process focuses on identifying the true positive samples correctly while also correctly rejecting the true negative samples that are misidentified by the cell extraction algorithm. [...] However, since ActSort is a quality control algorithm, it requires that movies are motion-corrected and cells are extracted with the experimenter's favorite cell extraction algorithm, therefore, any mistake made in the cell extraction process will propagate to the cell sorting process.” and “One limitation of ActSort is that it relies on correct cell extraction by the cell extraction algorithm used by the experimenter. Future additions to ActSort could mitigate some of the common errors, such as the ability to merge segmented or duplicated cells.”\n\n**If the preprocessing step misses some of the cells there’s no recovery** You make an excellent point! Although EXTRACT and ActSort all fall into the calcium processing pipeline, they have different purposes. ActSort is a standalone pipeline for QUALITY control instead of cell finding. We modified the discussion section based on your suggestions as: \n“ActSort is a novel standalone pipeline for quality control that can be added as a further step after using any cell extraction algorithm such as EXTRACT, Suite2p, ICA, CAIMAN, and many others (Pachitariu et al., 2016, Ren et al., 2021, Giovannucci et al., 2019, ).”\n\n**Using foundation models** Wonderful question! Let us illustrate the shortcomings of foundation models on two examples on different, but related, problems:\n\nCell extraction: A foundational model, Cellpose, was developed for segmentation from images but is not used for cell extraction from videos. The main reason is that brain recordings, particularly the 1p imaging videos, are very diverse in their backgrounds, imaging and experimental conditions, and cell shapes/types/sizes. Therefore, even for cell extraction, the foundational models have not been successful to date. \n\nEvent extraction: Cascade (Rupprecht et al., 2021) utilized deep models to extract events from calcium activity traces in two-photon (but not one-photon) movies. The model was trained from scratch on a massive public dataset, and did not generalize to 1p movies due to aforementioned reasons. \n\nSimilarly, for cell sorting, a new model may need to be trained as more public benchmarks (our being the first with such scale) become available.\n \n**Control with a deep classifier** What a great idea! To address this, we added an experiment using ResNet-50 for extracting features and classification using 1p calcium imaging snapshots. Specifically, we provided the extracted cell profiles and the average cropped movie snapshot, averaged over frames with cells’ activities, to ResNet-50 and collected 2,000 features from the final layer. We repeated our experiment in Fig. 2 for this dataset (Fig N8). The engineered features demonstrated significantly higher AUC than using deep learning.\n\n**Technical novelty** We agree that the simplicity of the introduced method may seem not technical enough at first, but we kindly disagree. Please refer to the  “Regarding fit to the venue and our contributions” section in the general response for further discussion. In short, we believe the impact of our technical contributions, no matter how simple, (now that we have also shown the suboptimality of a classifier trained on ResNet-50 features) is significant, as neither CAL or DAL can reach the success of DCAL, nor DCAL without adaptive updates reaches the same levels (new experiment, data not shown due to page issues, but happy to elaborate further). \n\nThank you for your helpful suggestions. Specifically, the deep net classifier was a very significant addition to our work, thanks to your feedback! If you believe we addressed your concerns, would you consider increasing your scores?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BB2nUwNEej", "reviewer": "Reviewer_uvup", "summary": "This paper introduces a new semi-supervised active learning algorithm, ActSort, designed to accelerate cell sorting in large-scale calcium imaging datasets. The method leverages domain expert feature engineering and a novel active learning framework, optimizing the cell sorting process with minimal human input. The paper also presents a user-friendly custom software and validates its performance through a large-scale benchmark study involving six domain experts and approximately 160,000 candidate cells. Empirical results indicate that semi-automation reduces the need for human annotation to only 1%-5% of the candidate cells, while also improving sorting accuracy by mitigating annotation bias. As a robust tool validated under various experimental conditions and applicable across different animal subjects, ActSort addresses the primary bottleneck in processing large-scale calcium imaging videos, paving the way for fully automated preprocessing of neural imaging datasets in modern systems neuroscience research.", "questions": "Regarding the algorithmic issues mentioned above, I have some technical questions for your reference:\n\n1. Can you provide a detailed statistical analysis of these 76 handcrafted features? How is the discriminative power of these features objectively evaluated?\n2. What is the label distribution of the 160,000 candidate cells annotated by domain experts? How do you address the class imbalance problem?\n3. How were the hyperparameters for the comparative methods (Random, CAL, DAL, etc.) chosen? Were they selected in a way that might favor ActSort?\n4. In Figure 4D, why does ActSort outperform human performance with just 1% of the data? Does this imply poor quality in human annotations?\n5. Besides logistic regression, what other classifiers were attempted? How did they perform?\n6. What are the specific details of the pre-training and transfer learning experiments? How significant are the generalization performance differences across different datasets and annotators?\n7. How was the batch size in active learning chosen? Have you considered querying the same sample multiple times to mitigate annotation noise?\n8. How is the ground truth defined in the experiments? How are samples with significant disagreement among experts handled?", "limitations": "Overall, this paper represents a substantial amount of work and provides executable software and code, which is of significant importance to the neuroscience field. My concerns are detailed in the shortcomings and issues section. Here, I will focus more on the scope of the paper.\nIf I were reviewing for Nature Methods, I would choose to accept this paper. However, I am uncertain whether this paper will attract widespread interest from the NeurIPS community. Therefore, I am giving a marginal score and would like to see the discussion from other reviewers before providing my final score.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Developed a new active learning-accelerated cell sorting algorithm for large-scale calcium imaging datasets, significantly reducing the workload of human annotation.\n- Utilized domain expert knowledge for feature engineering, enhancing classification robustness and accuracy across different animals and experimental conditions.\n- Created user-friendly custom software, allowing experimental scientists without programming backgrounds to use it easily.\n- Constructed a large-scale cell sorting benchmark dataset involving annotations by six experts, five mice, and approximately 160,000 candidate cells, which can be used for algorithm development and testing.\n- Demonstrated the method's effectiveness through empirical studies on multiple datasets, reducing the need for human annotation to 1-5%.", "weaknesses": "I believe this paper is a solid piece of scientific research, suitable for publication in a Nature sub-journal or a top-tier neuroscience journal (with the addition of more statistical analysis and biological significance studies). However, as a NeurIPS reviewer, I need to focus more on the algorithmic innovation and fair comparisons to help you improve the paper. I see the following shortcomings:\n\n1. **Lack of Innovation in Active Learning Strategy**: While the authors propose a new query strategy, DCAL, it essentially combines existing uncertainty and diversity sampling strategies through simple weighting, without sufficient theoretical justification and discussion on the necessity and effectiveness of this combination. Additionally, there is a lack of systematic analysis and guidance on adjusting the weights, and the ablation study in the experimental section is insufficient.\n\n2. **Lack of Experimental Comparison with Existing Semi-Automated Cell Sorting Methods**: Although the authors mention some prior semi-automated methods in the Related Work section, they do not conduct any quantitative comparative analysis in the experiments. This makes it difficult to assess the performance and efficiency advantages of ActSort over existing methods, rendering the \"state-of-the-art\" claim less convincing.\n\n3. **Insufficient Feature Representation Learning**: The authors rely heavily on handcrafted features designed by domain experts, lacking data-driven automatic feature learning capabilities. Good performance on the given dataset does not guarantee generalization to new datasets and experimental paradigms (e.g., new calcium indicators, neuron types, brain regions). The authors should consider utilizing pre-trained models such as CNNs to automatically learn visual and dynamic features of cells, reducing the burden of feature engineering.\n\n4. **Underestimation of Annotation Noise and Error**: The accuracy of manual annotations is the foundation of training and evaluation, but the paper does not sufficiently address this. Relying solely on voting to determine the \"ground truth\" overlooks the statistical properties of annotation errors. Additionally, the active learning process does not consider querying the same sample multiple times to reduce annotation noise, potentially overestimating the actual performance of the current method.\n\n5. **Unreasonable Pooling Method for Human Performance Comparison**: The paper compares the classifier's output with the annotations of a single annotator on the entire dataset in each iteration, while the classifier only uses a small subset of samples. A more reasonable approach would be to compare the classifier and human performance on the currently annotated subset, which might reveal a greater advantage for humans.\n\n6. **Insufficient Reporting of Experimental Setup Details**: The paper lacks descriptions of many critical implementation details, such as hyperparameter selection, training loss convergence, network architecture design, etc. In particular, the technical details of how general features pre-trained on ImageNet are transferred to the cell classification task are not reported, affecting the reproducibility of the results.\n\n7. **Insufficient Scalability and Robustness Experiments**: Although the authors emphasize ActSort's ability to handle large-scale datasets, the \"large-scale\" in the experiments is only 150GB, which is still far from the current terabyte-scale neural datasets. Moreover, there is no sensitivity analysis regarding different imaging parameters such as resolution, signal-to-noise ratio, and frame rate.\n\n8. **Lack of Theoretical Analysis on Active Learning Batch Size and Convergence**: The authors simply set the batch size to 1 without discussing its relationship with convergence speed and generalization performance. Particularly, the setting of the hyperparameter \\( m \\) lacks theoretical basis and sensitivity analysis. The convergence of active learning sampling is challenging both theoretically and practically, but the paper lacks necessary analysis and discussion on this.\n\n9. **Unknown Applicability to Other Types of Neural Activity Data**: The authors only tested on calcium imaging data, but neural electrophysiological data, such as in vitro patch clamp and in vivo multi-channel recordings, differ significantly from calcium imaging in terms of morphology and spatiotemporal resolution. Can ActSort be applied to these data types? Would feature redesign be necessary? These questions need validation.\n\n10. **Lack of Analysis on Annotator Variability**: The differences in knowledge background and experience among annotators can introduce annotation biases, affecting the performance of the trained classifier. The authors use annotations from multiple experts in the experiments but do not analyze the variability among experts and its impact. This variability itself is an important research issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed report and excellent summary! We truly appreciate your vote of confidence in our work and the excellent suggestions for improving our technical contributions. Thanks to you, our paper now includes several diverse, convincing control experiments. To save space, we refer to weaknesses with “W” and questions with “Q”, please excuse us. \n\n**W1 and W2** Please see our general response regarding the fit to venue and comparison to prior work. To address further concern, we performed two additional control studies. First, an ablation study by removing the adaptive estimation process showed that DCAL was significantly less effective without it (no space in PDF, happy to elaborate). The second study, shown in Fig. N5, demonstrates the evolution of weights over time. Also see Fig. N3 for additional evidence supporting the necessity of DCAL. \n\n**W3** Great idea! We added a new experiment to address this (Fig. N8). Please refer to the “New experiments” section in the general response. In short, we tried ResNet-50 for feature extraction from movie frames and cell extraction outputs, resulted in suboptimal features.\n\n**W4, Q7, and Q8** Excellent point! We evaluated manual annotation accuracy in Fig. S4, Tables S1, S2, and S3. We used four annotators per dataset, with majority vote as ground truth, to mitigate inconsistencies and annotation noise (See Fig. S3 for the evaluation process). Human annotators could revisit samples multiple times, while ActSort trains on the final annotations.\n\nTo address annotator reliability, we performed intraclass correlation analysis. Individual annotators were inconsistent (ICC = 0.56±0.06), but classifiers trained on these annotations mitigate human bias and were more consistent (ICC = 0.64±0.07). Majority votes across annotations were quite consistent (ICC = 0.79±0.05). This shows individual annotators are unreliable, but majority votes (In experiments, only one annotator rates!) approximate ground truths well.\n\n**W5** We compare the classifier’s prediction with single annotators across the entire dataset, NOT just a small subset. Also, as evident from the limit of 100% annotation, the classifiers do outperform humans on trained samples.\n\n**W6** We used logistic regression for the cell classifier to ensure real-time prediction speed, which converges to global convex optimum. The regularization effect is analyzed in Fig. S7, and a new hyperparameters analysis on the classifier threshold is in Fig. N4. AL convergence is depicted in Figs. 4, 5, S6, S8, S9, S10, and Tables S5, S6, S7. We now include a control for using ImageNet for pre-training general features (Fig. N8), whic the engineered features demonstrated significantly higher AUC than using deep learning.\n\n**W7** Good point! The process is dependent on cell numbers, not movie size. We added: “...(data compression) resulting in approximately 270+-90 MB/1,000 cells (mean +- std over 5 mice) data sizes.” A TB-scale movie with 10,000 cells (for example, the movie from Ebrahimi et al., 2022) can be compressed to less than 3GB. The five imaging datasets had different imaging conditions (1p-2p), apparatus, and frame rates (20, 30, and 50), demonstrating feature robustness, which is illustrated by the fact that ActSort works on both 2p and 1p movies without modification.\n\n**W8 and W9** These are outside our scope. We use linear classifiers with instant training, making batch size of one (a desirable property to mimic real-work cell sorting) attainable. Other data types have not achieved the recently attained 1 million neurons mark in calcium imaging, thus not of interest here.\n\n**W10** This key conclusion motivates ActSort. Human annotators are inconsistent, especially with large datasets. Thus, 4 annotations per dataset is a key contribution of our benchmark and evaluation. For instance, historical benchmarks like Neurofinder (2p, 100s of neurons, 1 annotator) later showed experts missed many cells in Suite2p. \n\n**Q1** We added an illustration explaining the absolute discriminability index (Fig. N1). We also added statistical quantification of traditional and new feature effect sizes (Fig. N2). We also updated the math for all features accordingly. \n\n**Q2** The label distribution of the 160,000 cell candidates is shown in Figs. S6, S8, and S10. We tested our algorithm on imbalanced datasets with more true positives (Figs. S6 and S10) and artificially inflated datasets with many false positives (Fig. S8). Our results are consistent, demonstrating our approach’s robustness to imbalance.\n\n**Q3** The Random query algorithm has no hyperparameters. CAL, DAL, and DCAL use the same cell (and/or label) classifiers to predict cell probability. We included a sweep over regularization parameters (Fig. S7) and a new experiment on classifier threshold sweeping (Fig. N4). \n\n**Q4** This indicates 1) humans are inconsistent over time, 2) humans get tired after sorting 10k cells, and 3) the active learning algorithm selects representative samples. Points 1 and 2 are supported by new intraclass correlation results, and point 3 is illustrated in Fig. 3. In short, linear classifiers primarily care about boundary samples; once those are found, classification is mostly optimal.\n\n**Q5** Other algorithms would be harder to judge (e.g., random forest, neural network) and would not allow instant training, which is needed for real-time human annotation. Convexity of LR ensures reproducibility, unlike random forest classifiers or neural networks.\n\n**Q6** We apologize for the confusion. We should not have called this pretraining and will update the text as “prelabeling”. Details of the fine-tuning process are in Algorithm S1 and Appendix C.2.\n\nThank you for your detailed review. We implemented your feedback whenever applicable, which improved our manuscript tremendously! If you have additional concerns, we look forward to discussing them further. If your concerns are addressed, would you consider increasing your scores?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6TdYyZ6xcz", "reviewer": "Reviewer_iZ5n", "summary": "The authors introduce an active learning framework for cell sorting in two photon imaging analysis. They develop a software interface to it and conduct a large scale benchmark using multiple datasets and involving multiple domain experts. They show that their method can reduce the needed manual human input to a small fraction of what other algorithms require. In contrast to many other works, they use hand-engineered features and simple classification algorithms.", "questions": "* Line 113f: The authors discuss features using the number of spikes. How are the spikes counted from the imaging data? This seems surprising.\n* Fig 2: Could the authors clarify what is shown here? This is d-prime between which distributions? How were the positive and negative examples defined here?\n* Fig 2: What is meant by traditional features?\n* Appendix B.1.2: Could the authors give mathematically precise feature definitions where possible?\n* Line 140: The authors may want to define their acronym DCAL. \n* Fig. 3: In A, what are the different colors?", "limitations": "Adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The idea to use active learning for cell sorting in large scale two photon experiments is interesting.\n* The interpolation between confidence and discrimination based sample selection is a small but clever addition to the literature.", "weaknesses": "* The algorithm is part of a large framework, for which many components have been described in other papers (e.g. the cell extraction framework EXTRACT). This, and the reference to some of the data papers, make the paper poorly anonymized. \n* There is extensive use of supplementary figures and the appendix, making the paper quite hard to read.\n* The figures are super dense and very hard to comprehend in detail. The line width in several figures is very thing, lines are overlapping and not all figures are properly annotated. \n* The paper spends too much space on the software and the overall framework, but has many very dense figures that are quite hard to take apart.\n* It remains somewhat unclear which parts of the software and the overall framework were constructed for this paper and which were already present before.\n* A comparison to a deep learning based cell classifier seems missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our manuscript. Your feedback was extremely helpful for increasing the clarity of our manuscript, and most importantly in distinguishing ourselves from published material.\n\n**Summary** Respectfully, we do not agree with your summary of our work in several key domains: First, we are not designing software for two photon imaging analysis, our dataset consists mainly of 1p movies (4 out of 5 movies). Second, we are not aware of any published work with deep classifiers that is used to sort cells in 1p movies. Next, there is no work prior to us designing an active learning routine for cell sorting, so we are not sure which many other works are referred to here. Finally, we believe we have several other contributions left out in your summary, please kindly see our general response above.\n\n**Anonymity:** ActSort is a standalone quality control pipeline, not part of any previous publication. It is compatible with not only EXTRACT, but also all other cell extraction algorithms. Moreover, ActSort is not affiliated with EXTRACT and does not endorse it. We simply used EXTRACT, since other state of the art alternatives to process 1p movies (CAIMAN, or earlier studies used ICA), were less efficient and time consuming. To mitigate this confusion, we added a new experiment on using ActSort on ICA extracted cells as shown in Fig N7 in the PDF. We are happy to address our contributions accordingly if the reviewer can let us know which specific previously published paper you are referring to.\n\nWe were carefully following the guidelines to maintain anonymity. For instance, we had not shared the user manual or links to our lecture videos for this very reason. In the references, we did our best to cover a broad community, including multiple calcium imaging techniques and cell extraction algorithms from multiple groups in the field, using published datasets with permissions from authors. Fortunately, your assumption is simply not correct.  \n\n**Comments on figures and use of appendices** Respectfully, we disagree with the reviewer that the extensive additional benchmarking and the detailed explanation of the algorithm in the appendices and the figures are a weakness. This is, in our opinion, a strength of our work. Moreover, as evident from other reviewer reports, our work is self-contained in the main text, and only uses appendices to refer to methodological details and additional benchmarking studies to strengthen our claims. If you can let us know which figures you find confusing, we would love to work on them.\n\n**Regarding discussion of software and framework** The software, the benchmark, and the active learning framework together form our novel conceptual contributions in addition to the specific active learning query algorithms. All three of them are equally spaced in the main text. We are the first to introduce this framework to experimental neuroscience, so it is reasonable that we spend time explaining them carefully, as a big fraction of our readership will be divided between active learning researchers and experimental neuroscientists.\n\n**Deep classifiers** Amazing point! We added a new experiment to address this (Fig. N8). Please refer to the “New experiments” section in the general response. In short, we tried ResNet-50 for feature extraction directly from movie frames and cell extraction outputs. Not only were the extracted features suboptimal to engineered features, the time to run cell data over the network was beyond our design.\n\n**Line 113f: spikes** Good question! We use the peakseek built-in function in MATLAB to obtain the spikes count from the extracted cell traces by the cell extraction algorithm (ICA or EXTRACT). \n\n**Clarification on Fig. 2.** Great point! We apologize for the confusion here. The d-prime is between the distribution of $p(X|c = \\text{cell}) $and $p(X|c = \\text{not cell})$ where $X$ represents the target feature. The positive examples represent the candidate being an actual cell and vice versa. To clarify, we added an illustration to explain the definition of absolute discriminability index, a metric for quantifying the distance between these two probability distributions (Fig. N1). Additionally, as requested, we also added a statistical quantification on the effect size of the traditional features and the new features (Fig. N2). \n\n**Fig 2: Traditional features:** Great question, one we should have made clear! The traditional features represent the features that CAIMAN and CLEAN used for cell classification on 1p imaging movies. Please see our general response to reviewers.\n\n**Detailed math for appendix and typos** Thank you for pointing these out. We have updated the math for all features accordingly. We changed  L140 to “In this work, we designed an active learning query algorithm, Discriminative Confidence based Active Learning query algorithm, in short DCAL, and compared its performance with traditional random selection and the two other query strategies. ” The Fig 3A legend is placed under Figure B-D. We moved the legend above to make it more accessible for the reader as you suggested.\n\n**Final clarifications** As we are concluding our response, we wish to address the weaknesses 1 and 5 from your report directly. We want to make it super clear that we have NOT copied any part of ActSort from any published work, nor did any part of ActSort was published before anywhere. We apologize if there was any confusion, and hope the rebuttal, the general response, all additional experiments we shown in the PDF, and modifications, make it clear that ActSort is NOT part of a large framework previously described in other papers.\n\nWe are looking forward to your additional comments if you have any on how we can improve clarity. Now that we have addressed all your raised weaknesses and corrected the misconceptions about our contributions - for which we thank you -, we hope you will consider increasing your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Many thanks to all the reviewers’ comments! We appreciate your time and efforts. We have addressed all concerns either with written edits to the manuscript, by performing requested experiments (see attached PDF), and/or by citations to relevant literature. Here, we would like to address some of the common concerns raised and further clarify the contributions of our work.

**Comparisons to Prior Work** ActSort is a novel post-cell extraction quality control algorithm, not part of existing pipelines. Historically, manual annotation was feasible for small datasets, but with large-scale imaging techniques recording up to one million cells (Manley et al., 2024), automated quality control is essential, as manual review is unrealistic for such volumes. For instance, Manley et al. (2024) seems to have used thresholding based on quality metrics (not even classifiers!) rather than human annotation (https://github.com/vazirilab/MAxiMuM_processing_tools/blob/main/planarSegmentation.m).

Most algorithms like CAIMAN and Suite2p focus on 2p movies, whereas our work targets broader solutions, including 1p movies with lower resolution, higher background noise, and reduced specificity. Applying pre-trained classifiers from these algorithms to 1p movies is infeasible, as even for 2p movies, these models are suboptimal (CAIMAN, Giovannucci et al., 2019). CAIMAN advises against using CNN classifiers for 1p data (https://github.com/flatironinstitute/CaImAn/blob/main/demos/notebooks/demo_pipeline_cnmfE.ipynb). Suite2p, validated only for two-photon movies, and uses features + linear classifiers that are represented in our traditional features. Similarly, EXTRACT seems to use thresholding based on some features.

We designed a feature set called "traditional features," including features used by classifiers in these prior works, such as SNR and spike width. The CLEAN framework (https://bahanonu.github.io/ciatah/) employs predefined features to train cell classifiers, though CLEAN is unpublished and lacks public code. We included these features in our traditional set and compared them with our new features. Unlike prior methods, CLEAN did not use an active learning query algorithm, sampling cell candidates randomly. Our baseline represents prior methods as random sampling and compares them with our novel query algorithm.

**Fit to the Venue and Contributions** Over a long period, tools for “Neural Information Processing” (Prank et al., 1998; Pachitariu et al., 2013; Andilla et al., 2014; Inan et al., 2017; Giovannucci et al., 2017; Aitchison et al, 2017; Choi et al., 2020; Dinc et al., 2023) have been welcomed by the NeurIPS conference. As NeurIPS stands at the intersection of AI and neuroscience, it has a rich tradition of embracing both technical and conceptual novelties, as well as new computational tools for neuroscientists. We believe our work upholds this tradition by offering three main contributions:

- *First Active Learning Benchmark for Cell Sorting:* We introduce the first public benchmarks for cell extraction quality control on 1p and 2p imaging, with ~160,000 annotations. This dataset is unparalleled, and we kindly ask that reviewers consider its value for the active learning community as well.

- *Framing Cell Sorting as an Active Learning Problem:* We introduce and frame cell sorting as an active learning problem, providing essential tools for further studies, including features, datasets, and software.

- *Novel Query Algorithm:* We propose a novel query algorithm which, we believe, makes a significant contribution. While some reviewers found the technical novelty limited, we respectfully disagree. We wish to bring forth the argument that technical novelty should be evaluated based on its impact rather than complexity. Our algorithm, reducing human effort from 100% to 1%, is a substantial technical advancement. We also performed more control studies to provide evidence for this point (see below).

**Summary of New Experiments** We provide a summary of new figures and experiments addressing reviewers’ concerns. Additional details are in the reviewer responses and figure captions.
- Figs. N1 and N2: Added figures explaining the absolute discriminability index and summarizing effect sizes for all features.
- Fig. N3: Replotted Fig. 3D with cosine distance for better dynamical range.
- Fig. N4: Added sensitivity analysis of classifier thresholds.
- Fig. N5: Recorded DCAL weights throughout the cell sorting process, showing adaptive estimation decreased sensitivity to initial conditions.
- Figs. N6 and N7: Added experiments on cell candidates from the ICA algorithm, demonstrating the necessity of cell sorting. **This figure also showcases the necessity of cell sorting, as even discarded garbage components can encode speed (perhaps due to brain motion or other contaminations), which might lead to incorrect biological conclusions if not culled out.**
- Fig. N8: We used ResNet-50 to extract a total of 2,000 features directly from the movie and cell extraction data, which we used to classify the cell candidates. Our engineered features outperformed this approach, which was surprisingly good at rejecting cells (outperforming traditional, but not our, features). As noted above, the fact that deep learning based feature extraction was not as successful in 1p movies is in line with the conclusions of prior research (CAIMAN, Cascade etc.). **Finally, we wish to emphasize that even this simple experiment (gathering features from average movie frames with ResNet-50) took us roughly 30 minutes for 10,000 cells with an NVIDIA RTX 4090 GPU, and about 2 hours without one. This waiting time is beyond our design restrictions.**

Meta-review: The paper introduces a novel approach to cell sorting in large-scale calcium imaging datasets through a semi-supervised active learning framework called ActSort. The proposed method is well-motivated, addressing a significant bottleneck in the processing of such datasets, where manual annotation becomes impractical due to scale. The paper is structured around three primary components: preprocessing, cell selection, and an active learning module that leverages discriminative confidence to optimize cell annotation.

The reviewers noted that the method integrates existing knowledge, particularly through the use of hand-engineered features and simple classification models, and enhances these with an active learning strategy that reduces the need for extensive manual annotation. The authors have effectively benchmarked the method across various datasets and conditions, involving multiple domain experts, which strengthens the empirical validation of their approach.

In the rebuttal, the authors provided additional experiments and clarifications, particularly in relation to comparisons with prior work. They have successfully demonstrated that ActSort offers distinct advantages over existing cell-sorting algorithms, especially in its applicability to both one-photon and two-photon imaging data, addressing limitations where prior methods falter.

The inclusion of user-friendly software and extensive benchmarking supports the potential for broad adoption within the neuroscience community. The authors have also responded adequately to all major concerns raised by the reviewers, providing detailed justifications and expanding their experimental results.

Given the novelty of the approach, its clear practical utility, and the thoroughness of the validation, this paper makes a meaningful contribution to the field.
Decision: Accept (poster)