dataesr/scientific-paragraphs-categorization huggingface dataset:
The gold standard preprocessed imaging datasets can be downloaded from https://github. com/BigNeuron/Data/releases/tag/Gold166_v1. After generating single-cell reconstructions for any subset of the data, the obtained automatic reconstructions can be uploaded by specifying the dataset identity (id) of each reconstruction in the filename (see id lookup table https://github. com/lmanubens/BigNeuron/blob/main/lookup_gold166. csv). Once uploaded, the user needs to name the novel reconstruction algorithm and include it in the interactive analysis and benchmarking. We invite developers and users to send novel algorithm implementations and imaging datasets for inclusion in the platform. All submissions will be assessed once per year and introduced in the platform to ensure maintenance of the platform.
Variance among reconstructions is explained by both image quality and tree morphology features An overview of the morphological features of the analyzed trees shows that there is high morphological diversity in the analyzed neurons. To quantify the heterogeneity of the data, we computed the coefficient of variation (CV) for a set of morphological features that was also explored in a more extensive dataset (5,099 neurons) in a published neuron reconstruction data mining work using a subset of the reconstructions publicly available at NeuroMorpho. org  (Polavaram et al. 2014 ). For 7 out of 9 neuromorphological features shared among both datasets, the CV of the Gold166 dataset was similar to or higher than its counterpart in the NeuroMorpho. org dataset (Fig. 3A ), indicating that the diversity of the Gold166 dataset is sufficient to sample the performance of automatic tracing algorithms in heterogeneous neuron types.
To identify the features that account for the variance among datasets, we performed dimensionality reduction using Principal Component Analysis (PCA). The first two principal components explain 43. 3% of the variance in the data (Fig. 3B ). Principal Component 1 (PC1, 24. 8% explained variance) is mainly contributed by measures that account for image quality: the focus score computed on the SWC nodes, the percentage of minimal intensity voxels in the SWC nodes, and the standard deviation of the intensity in the image volume (Fig. 3C ). Following those metrics, the median intensity in the image volume, and the median intensity on the SWC nodes show considerable contributions to PC1. Principal Component 2 (PC2, 18. 5% explained variance) is mainly contributed by neuromorphological metrics: the total length, followed by the maximum path distance, the maximum branch order, and the number of tips of the trees (Fig.
C-D Bar plots show the percentage of explained variance for each principal component. Bars represent the contribution (%) of each variable and the red dashed lines indicate the expected average contribution. The first principal component is a composite variable accounting for focus score and the percentage of voxels with minimum intensity in SWC nodes. The second principal component is a composite variable accounting mainly for tree morphology metrics: the total length, maximum path distance, maximum branch order, and the number of tips. E Sholl analysis of the neurons in the Gold166 dataset. Each line expresses the average number of intersections quantified at a given distance from the soma for the neurons of a given model organism (colorcoded). Neurons from different model organisms present a high diversity of branching patterns.
F-G Box plots and dot plots show the bounding box volume (F) and density of branch points per unit of neurite length (G) of the reconstructed cells for each model organism (color-coded). Center line is the median value, the box is limited by Q1 and Q3, and the whiskers add 1. 5x interquartile range. Colored dot points are overlayed for all measures. H Proportionality constant between the total length and Sholl Intersection Profile (SIP) as a function of the centripetal bias k for planar 2D neurons (dashed black line) and 3D neurons (solid black line). Experimental values for each of the cells in the Gold166 dataset are shown as solid dots color-coded by model organisms. 2D trees are shown as circles and 3D trees as triangles. I Hierarchical clustering using both neuromorphological and image quality features.
Colors indicate different clusters obtained with a Gaussian Mixture ellipsoidal, equal volume, shape, and orientation (EEE) model (see Supplementary Materials and Fig. S2 ). Labels indicate the neuron dataset id and the organism they belong to. A 2D projection overview of those principal components shows that the datasets are clustered by the labs that provided them, being separated along both PC1 and PC2. The variables contributing more to the difference between datasets are mostly related to image quality features, while tree morphology-related features explain most of the variance within each dataset. Even though a systematic comparison between species should be done for as highly similar cell types (beyond the scope of this work), we present here a comparison of broad neuromorphological differences.
3G ). While frog neurons show values closer to human and mouse counterparts, the number of cells we analyzed is too small to conclude that this result has biological relevance. Fruitfly neurons differ from silkmoth neurons mainly in terms of maximum path distance and maximum branch order, while showing increased average diameter and bifurcation angles in comparison to zebrafish and chicken neurons, the latter having values closer to human, mouse and chicken neurons. Chicken neurons show high dendritic complexity at small radii (Fig. 3E ), but they are closer to zebrafish and fruit fly larvae neurons in regard to size and density of branches per unit of dendritic length (Figs. 3F  and 3G ). To explore putative functional heterogeneity in the dataset, we quantified the centripetal bias k. When the centripetal bias is 0, neurites are not preferentially radial to the soma, and their angles toward the root of the tree are distributed uniformly.
Finally, the high confidence set of consensus nodes obtained through this process is connected in a single tree using the Minimum Spanning Tree algorithm (Fig. 4B  and 4C , Generation of consensus trees). To assess the ability of the consensus tree algorithm to provide informed best estimates of gold standard reconstructions, we bench-tested all the automatic reconstruction algorithms by measuring the distance between each automatic reconstruction and their corresponding gold standard tree. The bench testing results showed that, in 5 of 16 clusters identified in the data (Variance among the datasets is explained by both image quality and tree morphology features, Fig. 3I ), the consensus tree algorithm provided the most accurate approximation to gold standard trees (Fig. 4C  and 4D ). However, in the other clusters, different algorithms outperformed the consensus tree strategy (Fig.
Nodes from all input neurons are merged to form consensus nodes and reliability weights among consensus nodes are established by collecting votes from the connections from individual input neuron trees. A Maximum Spanning Tree algorithm is used to connect consensus nodes to form the consensus tree. C An example of the results obtained with the consensus tree algorithm using a set of 12 automatic reconstructions. D Bidirectional entire structure average distance between automatic reconstructions and the gold standard of the dataset cluster number 9 (see Fig. 3I ). Mean and Standard Errors are shown as a barplot. E Bidirectional entire structure average distance between automatic reconstructions and the gold standard of the dataset cluster number 8 (see Fig. 3I ). Mean and Standard Errors are shown as a barplot.
Image quality metrics correlate with algorithm performance To explore how reconstruction quality varies among different images, we tested how the distance between consensus reconstructions and gold standard trees correlate with every feature of the datasets. We performed this analysis by obtaining a correlation matrix among reconstruction quality, image quality, and neuron morphology features of all the automatic reconstruction results (Fig. 5A ). Distance metrics correlated with diverse image quality and tree morphology features: CNR, CNR in SWC nodes, the median intensity of the image, parent-daughter ratio, and remote bifurcation angle (Fig. 5A , cluster 3). The second cluster of features showed a negative correlation with distance metrics: focus score (together with its equivalent obtained in SWC nodes), the percentage of image voxels with minimal intensity (together with its equivalent in SWC nodes), the average diameter of the automatic reconstruction segments and the standard deviation of the image intensity in SWC nodes (Fig.
661; Fig. 6A ). Learning curves with increasing data percentage used for training show that the regression quality reaches a plateau with slight improvement once more than 30% of the data is used for training (Fig. 6B ), suggesting that this strategy will generalize well when it is used in novel neuron reconstruction datasets. We assessed the quality of the predictions by comparing the percentage of different structures of all the automatic reconstructions, the true best algorithms, and the predicted best algorithms for each dataset. Our analysis showed that both known best algorithms and predictions in most of the cases performed better than algorithms chosen by chance (Fig. 6C  and 6E ; the percentage of difference distribution is positively skewed, medians of 18%, 22%, and 44% for true best, predicted and all algorithms respectively; Wilcoxon test All algorithms vs.
True best V=20145, p=5. 12e-10, n1=1113, n2=20; All algorithms vs. Predicted best V=16470, p=2. 32e-4, n1=1113, n2=20; Predicted best vs. Predicted worst V=19, p=3. 03e-8, n1=20, n2=20). Similarly, algorithms predicted to be worse showed a negatively skewed distribution in the percentage of difference to gold standards (Fig. 6C ; median of 99%). Interestingly, this analysis highlighted the fact that for a few neuron datasets, none of the automatic reconstruction algorithms was able to recapitulate the gold standard annotations. This was found to be the case, for example, of chicken cells, which showed particularly low signal-to-noise ratio in their neurites and have specific distinctive morphological aspects (such as an increased soma size, high branch density, and high centripetal bias).
Showcase of best algorithm prediction in fMOST data Recent community efforts, like the BRAIN Initiative Cell Census Network (BICCN, https://biccn. org/), are using fluorescence micro-optical sectioning tomography (fMOST) to map neuron architecture within whole mouse brains. To illustrate the value of BigNeuron in this community, we used the SVM regression model trained with the Gold166 dataset to predict best-performing algorithms in fMOST image volumes. The SVM model provided predicted values for the percentage of difference to gold standard trees (not available for those images). The algorithms predicted to provide the best results were neutube and Consensus (Fig. 7A  and 7E ). Visual inspection of the reconstructions suggests that the regression model provides a reasonable approximation toward the best automatic algorithm selection (Fig.
DISCUSSION The complexity of multi-modal cell type characterization requires reproducible and high-fidelity neuronal reconstructions and precise estimates of their variability. Although quantification of neuronal morphology is an important approach in neuroscience, how to accurately and efficiently perform this task across diverse types of neuronal images in an unbiased fashion has been a long-standing challenge. This challenge has become even more acute with the introduction of exciting new technologies that generate 3D, complete neuronal morphology datasets at high speed  (Chung and Deisseroth 2013; Zheng et al. 2013; Verveer et al. 2007; Jiang et al. 2022). Moreover, largescale brain science projects such as the BRAIN Initiative in the USA  (Alivisatos et al. , 2012 ), Europe's Human Brain Project  (Kandel et al.
, 2013) , and the Allen Cell Types Database (http://celltypes. brain-map. org/) integrate reliable automatic reconstruction in their production pipelines to allow fast quantification of the structure of hundreds of thousands of neurons. Motivated by these demands, the BigNeuron project has contributed to improving existing automatic reconstruction standards, focusing on the interaction and discussion among neuroscientists and developers, and integrating data generation and development efforts from several laboratories. BigNeuron takes an innovative approach toward the standardization and development of new automatic reconstruction methods, including events for discussion and identification of major issues, and coordinated hackathons for the development of tools to overcome them. Data and some results from this effort have been published online for the community to take advantage of them.
In this work, we developed the BigNeuron interactive analysis web app, a platform for the neuroscience community that allows end-users of automatic reconstruction tools to explore interactively the results obtained with a great diversity of open-source algorithms and to predict which of them will provide the best results in their specific datasets. BigNeuron allows developers to interactively benchmark their tools against the methods assessed in this work, and to introduce datasets and novel algorithms within the benchmarking set upon request. This collaborative standardized approach is significant because the problem of choosing a method to perform automatic reconstruction in project-specific datasets is an arduous task. Having a standard resource to compare and analyze the methods' performance, while introducing tools to quantitatively assess features of the images relevant to the problem, is an approach that can provide easy-to-apply contrastable methods to solve it.
The diversity in the data that we analyzed allowed us to gain relevant insights toward future steps in improving automatic reconstruction tools. We first focused on a major aspect usually overseen when novel reconstruction algorithms are developed and tested: the variance among the datasets obtained by different teams. We found that both neuromorphology and image quality features explain comparable amounts of variance among the analyzed data, image quality being a relevant aspect to take into account in the process of choosing the most suitable tool for a specific dataset. By considering this information, we benchmarked existing and newly developed algorithms for different types of datasets. We found that different types of datasets are best reconstructed by specific algorithms. Also, individual neuromorphological features are best approximated by particular algorithms.
Thus, the information obtained through the pooling of datasets and benchmarking of reconstruction tools can be used to generate novel knowledge-based improved algorithms. Previous works have used the BigNeuron datasets to benchmark novel algorithms  (Li et al. 2017; Liu et al. 2018; Gu et al. 2020; Radojević and Meijering 2019)  and to develop neuromorphological features for the analysis of dendritic trees  (Kanari et al. 2018). However, to our knowledge there is only one study that explores the idea of using an ensemble learning algorithm to generate improved automatic reconstruction results  (Wang et al. 2017) , an algorithm included within our analysis. As another step in this direction, we developed a tool to generate consensus trees based on a set of automatic reconstructions. Our results highlight the potential of such an approach, providing the best estimates of the ground truth neuron structure in many of the datasets we analyzed.
We think future efforts could focus in the iterative search of optimal parameter sets of specific methods by applying optimization algorithms with maximization of reconstruction quality as the objective function. Finally, to help end-users identify optimal performing algorithms for their datasets, we used a support vector machine regression to predict the best algorithm performance. We developed a set of Vaa3D plugins that allows users to run all the automated reconstruction algorithms included in BigNeuron and quantify the image quality features of their datasets. By performing regression against all the analyzed datasets, users obtain a prediction for the best algorithm without the need of generating a set of gold standard reconstructions for bench-testing. In most cases, our regression method provides accurate estimates of the automatic reconstruction quality.
However, there is a small number of datasets for which there is a substantial difference between the true and the predicted best algorithms. Thus, this tool should be used with caution. Interestingly, in a few cases, none of the automatic reconstruction methods was able to produce any result close to the ground truth (>50% of different structure). This happened in datasets with extremely low signal-to-noise ratios and overall bad imaging quality. Quantitative analysis of those features can help users to find objective thresholds for image quality features below which datasets should be discarded before spending time and resources on automatic reconstruction methods. We think it is necessary to explore further refinement of the machine learning predictions of best-performing algorithms introduced in this work.
Recent projects are generating unprecedented volumes of gold standard reconstructions (e. g. Peng et al. 2021). We expect that applying the methods presented in standardized, larger gold standard datasets will provide a valuable tool toward full automation of neuron reconstruction in specific combinations of microscopic imaging and labeling techniques that are becoming standards in the field of structural interrogation of single neurons in the whole mouse brain. in R (version 3. 4. 1). To generate learning curves, we used the "generateLearningCurveData" of the mlr (version 2. 18. 0). To generate the regression results of Fig. 6 , the data was split by the ids of the datasets in 15% for testing and 85% for training sets. We predicted the value for the percentage of different structure between the automatic reconstruction and the gold standard with the regression model.
We obtained the regression coefficient of determination with the "msr" function of the mlr3 package (version 0. 12. 0). We did Wilcoxon tests (two-sided) using the "stat_compare_means" function of the ggpubr package (version 0. 4. 0). The code for this analysis can be found at: https://github. com/lmanubens/BigNeuron/blob/main/mlr_regression/mlr3_regression_3DIQ_3. R.
Showcase of best algorithm prediction on fMOST data We predicted best-performing algorithms in fMOST datasets. 40 fMOST image volumes were kindly provided by Dr. Liya Ding (20 dendritic trees and 20 axonal trees). We processed the images with the same steps used in Gold166 dataset (see Section Image preprocessing), and obtained automatic and consensus reconstructions as described in Algorithm bench-testing and Generation of consensus trees. We correspondingly obtained image quality and neuromorphological features as described in Image quality measurement and Morphological analysis of neuronal reconstructions. After applying a Box-Cox transformation, all the features obtained were used as inputs to the SVM regression model obtained as described in the previous section.
Acknowledgements This project was supported by  Allen Institute  for initialization and a series of events, and a  DOE 's  Oak Ridge National Lab Leadership Computing  award to  Hanchuan Peng ; the cross-platform bench-test was also supported intensively at  Lawrence Berkeley National Lab, Allen Institute for Brain Science , and  Blue Brain Project at EPFL, Southeast University at Nanjing , and various other facilities. This project also used large-scale display wall and immersive VR facilities at  Imperial College of London ,  Oak Ridge National Lab, Janelia Research Campus of HHMI , and  Southeast University. This project was also supported by  Tencent Inc. and Southeast University  for interactive online analysis. This project has inspired more than 260 other publications to date. We acknowledge  Prabhat ,  Zhijiang Wan ,  Jinzhu Yang ,  Hang Zhou ,  Arunachalam Narayanaswamy ,  Shaoqun Zeng ,  Przemyslaw Glowacki ,  Dezhe Jin ,  Zhihao Zheng ,  Pengyu Hong ,  Tao Zeng ,  Rongjian Li ,  Hidetoshi Ikeno ,  Yu-Tai Ching ,  Tingwei Quan ,  Jan-Felix Evers ,  Chloe Murtin ,  Shan Gao ,  Yan Zhu ,  Yang Yang ,  Hiroyuki Ai ,  Shin Ishii ,  Edward Hottendorf ,  Takashi Kawase , and a number of other colleagues for assistance in developing and porting automated reconstruction algorithms.
We thank  Staci Sorensen  for assistance in discussing and tracing some gold standard neuron reconstructions, and  Ryohei Kanzaki ,  Daisuke Miyamoto ,  Rachel Wong ,  Yun Wang ,  Ed Lein ,  Cher Bass ,  Steve Danzer  and many other colleagues for providing neuron image datasets. We acknowledge  Allan Jones  for giving the project the name and support throughout the project. We thank  Rafael Yuste  and  David Van Essen  for discussion;  John Isaac  and  Kevin Moses  for support from  Wellcome Trust  to organize the  Cambridge University  hackathon;  Gerald Rubin  and  Nelson Spruston  for support, data sharing and event organization at the  Janelia Research Campus of HHMI ;  INCF  for financial support for organization of a series of meetings;  Beijing University of Technology, Imperial College of London , and  Southeast University  for support of hackathons and workshops.
We thank  Xuan Zhao  for assistance in hosting the shiny app in the neuroXiv website;  Penghao Qian ,  Zuohan Zhao  and  Xin Chen  for assistance in formatting the manuscript. We thank  Anne Carpenter  for discussion in organizing a tracing hackathon. We acknowledge the  National Science and Technology Innovation 2030 -"Brain Science and Brain-Inspired Research" Program   of China  (Grant  2021ZD0204002 ). Bing  Ye was supported by the  NIH  grant  R01 EB028159. Ann-Shyn Chiang  was supported by the  Higher Education Sprout Project  co-funded by the  Ministry of Education  and the  Ministry of Science and Technology in Taiwan. We acknowledge  National Center for High-Performance Computing in Taiwan  for managing the FlyCircuit data.
Data availability 3D image volumes of the Gold166 dataset and gold standard reconstructions are available at http://web. bii. astar. edu. sg/bigneuron/gold166. zip. Bench-testing automated reconstructions can be downloaded from https://github. com/BigNeuron/Data/releases/tag/gold166_bt_v1. 0. The complete set of image volumes gathered throughout the project, amounting to ~4TB of data, is available upon request.
Code availability The source code developed is released open source and is available at https://github. com/lmanubens/BigNeuron. The Shiny web app can be used at https://linusmg. shinyapps. io/BigNeuron_Gold166/ and https://neuroxiv. net/bigneuron/. Source code utomated reconstruction algorithms developed throughout the project can be found at https://github. com/Vaa3D/vaa3d_tools/tree/master/released_plugins/v3d_plugins.
Dataset gathering 14 neuroscience research labs and institutions worldwide acquired the imaging datasets used in this work, combining different imaging methods, model organisms, and neuron types. Table  1  in the Supplementary Materials summarizes metadata describing the relevant aspects of each dataset. All the datasets analyzed in this manuscript are freely accessible at https://github. com/BigNeuron/Data/releases. The complete set of image volumes gathered throughout the project, amounting to ~4TB of data, is available upon request.