dataesr/scientific-paragraphs-categorization huggingface dataset:
Batch processing of large scale datasets on standalone machines The batch processing pipeline mentioned above can become a computational bottleneck when tackled without customized solutions. For instance, a naive approach to the problem might have as a rst step to load in-memory the full dataset; this approach is non-scalable as datasets typically exceed available RAM (and extra memory is required by any analysis pipeline). To limit memory usage, as well as computation time, CAIMAN BATCH relies on a MapReduce approach  (Dean and Ghemawat, 2008). Unlike previous work  (Freeman et al. , 2014) , CAIMAN BATCH assumes minimal computational infrastructure (up to a standard laptop computer), is not tied to a particular parallel computation framework, and is compatible with HPC scheduling systems like SLURM  (Yoo et al.
, 2003). Naive implementations of motion correction algorithms need to either load in memory the full dataset or are constrained to process one frame at a time, therefore preventing parallelization. Motion correction is parallelized in CAIMAN BATCH without signi cant memory overhead by processing several temporal chunks of a video data on di erent CPUs. CAIMAN BATCH broadcasts to each CPU a meta-template, which is used to align all the frames in the chunk. Each process writes in parallel to the target le containing motion-corrected data, which is stored in as a memory mapped array. This allows arithmetic operations to be performed against data stored on the hard drive with minimal memory use, and slices of data to be indexed and accessed without loading the full le in memory.
Automated component evaluation and classi cation A common limitation of matrix factorization algorithms is that the number of components that the algorithm seeks during its initialization must be pre-determined by the user. For example, Pnevmatikakis et al. (2016)  suggest a large number of components which are then heuristically ordered according to their size and activity pattern. When processing large datasets in patches the target number of components is passed on to every patch implicitly assuming a uniform density of (active) neurons within the entire FOV. In general this assumption does not hold and can generate a large number of spurious components. CAIMAN introduces tests to assess the quality of the detected components and eliminate possible false positives. These tests are based on the observation that active components are bound to have a distinct localized spatio-temporal signature within the FOV.
CNN based classi cation: We also trained a 4-layer convolutional neural network (CNN) to classify spatial footprints into true or false components (Fig. 2e ), where a true component here corresponds to a spatial footprint that resembles the soma of a neuron. The classi er, named batch classi er, was trained on a small corpus of manually annotated datasets (full description given in section Benchmarking against ground truth) and exhibited similar high classi cation performance on test samples from di erent datasets. While CAIMAN uses the CNMF algorithm, the tests described above can be applied to results obtained from any source extraction algorithm, highlighting the modularity of our tools.
Online analysis with CAIMAN ONLINE CAIMAN supports online analysis on streaming data building on the core of the prototype algorithm of Giovannucci et al. (2017) , and extending it in terms of qualitative performance and computational e ciency: Initialization: Apart from initializing CAIMAN ONLINE with CAIMAN BATCH on a small time interval, CAIMAN ONLINE can also be initialized in a bare form over an even smaller time interval, where only the background components are estimated and all the components are determined during the online analysis. This process, named BAREINITIALIZATION, can be achieved by running the CNMF algorithm  (Pnevmatikakis et al. , 2016)  over the small interval to estimate the background components and possibly a small number of components. The SEEDEDINITIALIZATION of Alg.
Creating ground truth data through manual annotation We collected manual annotations from multiple independent labelers who were instructed to nd round or donut shaped 2  active neurons on 9 two-photon in vivo mouse brain datasets. The datasets were collected at various labs and from various brain areas (hippocampus, visual cortex, parietal cortex) using several GCaMP variants. A summary of the features of all the annotated datasets is given in Table  2. Details about the annotation procedure are given in Methods and Materials. To address human variability in manual annotation each dataset was labeled by 3 or 4 independent labelers, and the nal ground truth dataset was created by having the di erent labelers reaching a consensus over their disagreements (Fig. 3a ). The result of this process was de ned as ground truth for the evaluation of CAIMAN as well as each individual labeler against the consensus (Fig.
3b )  3. More details are given in Methods and Materials (Collection of manual annotations and ground truth). We believe that the current database, which will be made publicly available, presents an improvement over the existing neuro nder database (http://neuro nder. codeneuro. org/) in several aspects: Consistency: The datasets are annotated using exactly the same procedure (see Methods and Materials), and in all datasets the goal is to detect only active cells. In contrast, the annotation of the various neuro nder datasets is performed either manually or automatically by segmenting an image of a static (structural) indicator. Even though structural indicators could be used for ground truth extraction, the segmentation of such images is not a straightforward problem in the case of dense expression, and the stochastic expression of indicators can lead to mismatches between functional and structural indicators.
Manual annotations show a high degree of variability We compared the performance of each human annotator against a consensus ground truth. The performance was quanti ed with a precision/recall framework and the results of the performance of each individual labeler against the consensus ground truth for each dataset is given in Table  1. The range of human performance in terms of F 1 score was 0. 69-0. 94, with average 0. 83± 0. 07 (mean ± STD). All annotators performed similarly on average (0. 83±0. 05, 0. 83±0. 08, 0. 84±0. 06, 0. 85±0. 08). We also ensured that the performance of labelers was stable across time (i. e. their learning curve plateaued, data not shown). As shown in Table  1  (see also Fig  4b ) the F 1 score was never 1, and in most cases it was less or equal to 0. 9, demonstrating signi cant variability between annotators.
CAIMAN BATCH and CAIMAN ONLINE detect neurons with near-human accuracy We rst benchmarked CAIMAN BATCH and CAIMAN ONLINE against consensus ground truth for the task of identifying neurons locations and their spatial footprints, using the same precision recall framework (Table  1 ). Fig. 4a  shows an example dataset (K53) along with neuron-wise matches and mismatches between CAIMAN BATCH and consensus ground truth (top) and CAIMAN ONLINE vs consensus ground truth (bottom). The results indicate a similar performance between CAIMAN BATCH and CAIMAN ONLINE; CAIMAN BATCH has F 1 scores in the range 0. 68-0. 79 and average performance 0. 75±0. 04 (mean±STD). On the other hand CAIMAN ONLINE had F 1 scores in the range 0. 68-0. 82 and average performance 0. 76±0. 04. While the two algorithms performed similarly on average, CAIMAN BATCH tends to perform better for shorter datasets whereas online processing tends to lead to better results for longer datasets (see Table  2  for characteristics of the various datasets).
CAIMAN approaches but is in most cases below the accuracy levels of human annotators (Fig. 4b ). This can be attributed to a number of reasons: First, to demonstrate the generality and ease of use of our tools, the results presented here are obtained by running CAIMAN BATCH and CAIMAN ONLINE with exactly the same parameters for each dataset (see Methods and Materials (Implementation details)): ne-tuning to each individual dataset can signi cantly increase performance. Second, CNMF detects active components regardless of their shape, and can detect non-somatic structures with signi cant transients. While non-somatic components can be ltered out to some extent using the CNN classi er, their existence degrades performance compared to the ground truth that consists only of neurons. Lastly, the ground truth is by construction a subset of the union of all individual annotations, which can bias upwards the scores of individual labelers.
To evaluate the performance we computed a precision metric as the fraction of inferred components  Both distributions exhibit a sharp derivative close 1 (last bin), with the batch approach giving better results. above a certain SNR threshold that are matched with a ground truth component (Fig. 4c , shaded blue). Similarly we computed a recall metric as the fraction of ground truth components above a SNR threshold that are detected by CAIMAN BATCH (Fig. 4c , shaded red), and an F 1 score as the harmonic mean of the two (Fig. 4d ). The results indicate that the performance signi cantly grows as a function of the SNR for all datasets considered, growing on average from 0. 73 when all neurons are considered to 0. 92 when only neurons with traces having SNR g 9 are considered (Fig. 4d )  5.
CAIMAN reproduces the ground truth traces with high delity Testing the quality of the inferred traces is a more challenging task due to the complete lack of ground truth data in the context of large scale in vivo recordings. As mentioned above, we considered as ground truth the traces obtained by running the CNMF algorithm seeded with the Manuscript submitted binary masks obtained by consensus ground truth procedure. After alignment of the ground truth with the results of CAIMAN, the matched traces were compared both for CAIMAN BATCH and for CAIMAN ONLINE. Fig. 5a , shows an example of 5 of these traces for the dataset K53, showing very similar behavior of the traces in these three di erent cases. To quantify the similarity we computed the correlation coe cients of the traces (ground truth vs CAIMAN BATCH, and ground truth vs CAIMAN ONLINE) for all the 9 datasets (Fig.
5b-c ). Results indicated that for all but one dataset (Fig. 5b ) CAIMAN BATCH reproduced the traces with higher delity, and in all cases the mean correlation coe cients was higher than 0. 9, and the empirical histogram of correlation coe cients peaked at the maximum bin 0. 99-1 (Fig. 5c ). The results indicate that the batch approach extracts traces closer to the ground truth traces. This can be attributed to a number of reasons: By processing all the time points simultaneously, the batch approach can smooth the trace estimation over the entire time interval as opposed to the online approach where at each timestep only the information up to that point is considered. Moreover, CAIMAN ONLINE might not detect a neuron until it becomes strongly active. This neuron's activity before detection is unknown and has a default value of zero, resulting in a lower correlation coe cient.
Online analysis of a whole brain zebra sh dataset We tested CAIMAN ONLINE with a 380GB whole brain dataset of larval zebra sh (Danio rerio) acquired with a light-sheet microscope  (Kawashima et al. , 2016). The imaged transgenic sh (Tg(elavl3:H2B-GCaMP6f)jf7) expressed the genetically encoded calcium indicator GCaMP6f in almost all neuronal nuclei. Data from 45 planes (FOV 820x410 m 2 , spaced at 5. 5 m intervals along the dorso-ventral axis) was collected at 1Hz for 30 minutes (for details about preparation, equipment and experiment refer to Kawashima et al. (2016) ). With the goal of simulating real-time analysis of the data, we run all the 45 planes in parallel on a computing cluster with 9 nodes (each node is equipped with 24 CPUs and 128-256 GB RAM). Data was not stored locally in each machine but directly accessed from a network drive.
The algorithm was initialized with CAIMAN BATCH run on 200 initial frames and looking for 500 components. The small number of frames (1885) and the large FOV size (2048 ù 1188 pixels) for this dataset motivated this choice of increased number of components during initialization. In Fig. 6  we report the results of the analysis for plane number 11 of 45. For plane 11, CAIMAN ONLINE found 1524 neurons after processing 1685 frames. Since no ground truth was available for this dataset, it was only possible to evaluate the performance of this algorithm by visual inspection. CAIMAN ONLINE identi ed all the neurons with a clear footprint in the underlying correlation image (higher SNR, Fig. 6a ) and missed a small number of the fainter ones (low SNR). By visual inspection of the components the authors could nd very few false positives.
Analyzing 1p microendoscopic data using CAIMAN We tested the CNMF-E implementation of CAIMAN BATCH on in vivo microendosopic data from mouse dorsal striatum, with neurons expressing GCaMP6f. 6000 frames were acquired at 30 frames per second while the mouse was freely moving in an open eld arena (for further details refer to Zhou et al. (2018) ). In Fig. 7  we report the results of the analysis using CAIMAN BATCH with patches and compare to the results of the MATLAB ® implementation of Zhou et al. (2018). Both implementations detect similar components (Fig. 7a ) with an F 1 -score of 0. 89. 573 neurons were found by both implementations. 106 and 31 additional components were detected by Zhou et al. (2018) and CAIMAN BATCH respectively. The median correlation between the temporal traces of neurons detected by both implementations was 0.
Computational performance of CAIMAN We examined the performance of CAIMAN in terms of processing time for the various analyzed datasets presented above (Fig. 8 ). The processing time discussed here excludes motion correction, which is highly e cient and primarily depends on the level of the FOV discretization for non-rigid motion correction  (Pnevmatikakis and Giovannucci, 2017). For CAIMAN BATCH, each dataset was analyzed using three di erent computing architectures: i) a single laptop (MacBook Pro) with 8 CPUs and 16GB of RAM (blue in Fig. 8a ), ii) a linux-based workstation (CentOS) with 24 CPUs and 128GB of RAM (magenta), and iii) a linux-based HPC cluster (CentOS) where 112 CPUs (4 nodes, 28 CPUs each) were allocated for the processing task (yellow). Fig. 8a  shows the processing of CAIMAN BATCH as a function of dataset size on the 5 longest datasets, whose size exceeded 8GB, on log-log plot.
As expected the cost of updating spatial footprints can be signi cantly larger if done simultaneously for all components (Fig. 8c , bottom). However, the average cost of updating a single spatial footprint is roughly 8ms, enabling real-time processing for each frame, when this step is evenly distributed among di erent frames/volumes, or is performed by a parallel independent process  (Giovannucci et al. , 2017). The cost of processing 1p data in CAIMAN BATCH using the CNMF-E algorithm  (Zhou et al. , 2018)  is shown (Fig. 8d ) for the workstation hardware. Splitting in patches and processing in parallel can  lead to computational gains at the expense of increased memory usage. This is because the CNMF-E introduces a background term that has the size of the dataset and needs to be loaded and updated in memory in two copies.
Towards surpassing human neuron detection performance To evaluate the performance of CAIMAN BATCH and CAIMAN ONLINE, we generated a corpus of multiply annotated two-photon imaging datasets. The results indicated a surprising level of disagreement between individual labelers, highlighting both the di culty of the problem, and the non-reproducibility of the laborious task of human annotation. CAIMAN reached near-human performance with respect to this ground truth, by using the same parameters for all the datasets without dataset dependent parameter tweaking. Such tweaking could for example include setting the SNR threshold based on the noise level of the recording, the complexity of the neuropil signal based on the level of background activity, or specialized treatment around the boundaries of the FOV to compensate for eventual imaging artifacts.
Future directions While CAIMAN uses a highly scalable processing pipeline for two-photon datasets, processing of one-photon microendoscopic imaging data is less scalable due to the more complex background model that needs to be retained in memory during processing. Adapting CAIMAN ONLINE to the onephoton data processing algorithm of Zhou et al. (  2018 ) is a promising way for scaling up e cient processing in this case. The continuing development and quality improvement of neural activity indicators has enabled direct imaging of neural processes (axons/dendrites), imaging of synaptic activity  (Xie et al. , 2016) , or direct imaging of voltage activity in vivo conditions  (Piatkevich et al. , 2018). While the approach presented here is tuned for somatic imaging through the use of various assumptions (space localized activity, CNN classi ers trained on images of somatic activity), the technology of CAIMAN is largely transferable to these domains as well.
Such decisions signi cantly a ect the speed at which data is read or written: in column-major order reading a full column is fast because memory is read in a single sequential block, whereas reading a row is ine cient since only one element can be read at a time and all the data needs to be accessed. Therefore, the original dataset must be saved in the right order to avoid performance problems. In the context of calcium imaging datasets, CAIMAN BATCH represents the datasets in a matrix form Y , where each row corresponds to a di erent imaged pixel, and each column to a di erent frame. As a result, a column-major order mmap le enables the fast access of individual frames at a given time, whereas a row-major order les enables the fast access of an individual pixel at all times. To facilitate processing in patches CAIMAN BATCH stores the data in row-major order.
In practice, this is opposite to the order with which the data appears, one frame at a time. In order to reduce memory usage and speed up computation CAIMAN BATCH employs a MapReduce approach, where either multiple les or multiple chunks of a big le composing the original datasets are processed and saved in mmap format in parallel. This operation includes two phases, rst the chunks/ les are saved in multiple row-major mmap format, and then chunks are simultaneously combined into a single large row-major mmap le. In order to reduce preprocessing steps, if the le(s) need to be corrected for motion artifacts, chunks of the registered data can be stored on-the-y during motion correction.
To calculate its SNR we rst compute a type of z-score: z i = c i + r i * BASELINE(c i + r i ) NOISE(c i + r i ). ( 6 ) The BASELINE( ) function determines the baseline of the trace, which can be varying in the case of long datasets exhibiting baseline trends, e. g. , due to bleaching. The function NOISE( ) estimates the noise level of the trace. Since calcium transients around the baseline can only be positive, we estimate the noise level by restricting our attention only to the points t n where c i + r i is below the baseline value, i. e. , t n = {t : c i (t) + r i (t) f BASELINE(c i + r i )}, and compute the noise level as the scale parameter of a half-normal distribution (Fig. 2b ): NOISE(c i + r i ) = std([c i + r i ](t n ))_ u 1 * 2 ⇡. (7) We then determine how likely is that the positive excursions of z i can be attributed just to noise.
We can also use a similar test for the signi cance of the time traces in the spike domain after performing deconvolution. In this case, traces can be considered as spiking if the maximum height due to a spike transient is signi cantly larger than a threshold. If we assume that the shape of each calcium transient has been normalized to have maximum amplitude 1, then this corresponds to testing Òs i Ò ÿ g ✓ SNR i , where s i represents the deconvolved activity trace for component i, and ✓ SNR is again an appropriate SNR threshold, e. g. , ✓ SNR = 2, and i is the noise level for trace i. Classi cation through convolutional neural networks (CNNs) The tests described above are unsupervised but require ne-tuning of two threshold parameters (✓ sp , ✓ SNR ) that might be dataset dependent and might be sensitive to strong non-stationarities.
Collection of manual annotations and ground truth We collected manual annotations from four independent labelers who were instructed to nd round or donut shaped neurons of similar size using the ImageJ Cell Magic Wand tool Walker  (2014). We focused on manually annotating only cells that were active within each dataset and for that reason the labelers were provided with two summary statistics: i) A movie obtained by removing a running 20th percentile (as a crude background approximation) and downsampling in time by a factor of 10, and ii) the max-correlation image. The correlation image (CI) at every pixel is equal to the average temporal correlation coe cient between that pixel and its neighbors Smith and Häusser (2010) (8 neighbors were used for our analysis). The max-correlation image is obtained by computing the CI for each batch of 33 seconds (1000 frames for a 30Hz acquisition rate), and then taking the maximum over all these images.
Neurons that are inactive during the course of the dataset will be suppressed both from the baseline removed video (since their activity will always be around their baseline), and from the max-correlation image since the variation around this baseline will mostly be due to noise leading to practically uncorrelated neighboring pixels. 9 di erent mouse in vivo datasets were used from various brain areas and labs. A description is given in Table  2. To create the consensus ground truth, the labelers were asked to jointly resolve the inconsistencies with each others annotations. The annotation procedure provides a binary mask per selected component. On the other hand, the output of CAIMAN for each component is a non-negatively valued vector over the FOV (a real-valued mask). The two sets of masks di er not only in their variable type but also in their general shape: Manual annotation through the Cell Magic Wand tool tends to produce circular shapes, whereas the output of CAIMAN will try to accurately estimate the shape of each active component.
REGISTERMULTI di ers from the CELLREG method of Sheintuch et al. (2017)  in a few key ways, that highlight its simplicity and robustness: • REGISTERMULTI uses a very simple intersection over union metric to estimate the distance between two neighboring neurons after the FOV alignment. Cells that have a distance above a given threshold are considered di erent by default and are not tested for matching. This parameter is intuitive to set a priori for each dataset. In contrast CELLREG uses a probabilistic framework based on the joint probability distribution between the distance of two cells and the correlation of their shapes that makes speci c parametric assumptions about the distributions of centroid distances between the same and di erent cells, as well as their shape correlations. This model needs to be re-evaluated for every di erent set of sessions to be registered and potentially requires a lot of data to learn the appropriate distance metric.