dataesr/scientific-paragraphs-categorization huggingface dataset:
Tridesclous is among the top performers for both MEAREC study sets and for PAIRED_MEA64-C_YGER, but had a substantially lower accuracy for most of the other datasets. This algorithm appears to struggle with lower-SNR units. Klusta is substantially less accurate than other sorters in most of the study sets, apart from MAN-UAL_FRANKLAB where, surprisingly, it found the most units above accuracy 0. 8 of any sorter. It also has one of the highest crash/timeout rates. The version of WaveClus used in SpikeForest is only suited for (and only run on) monotrodes; a new version of WaveClus now supports polytrodes, but we have not yet integrated it. We included both paired and synthetic monotrode study sets with studies taken from selected single electrodes of other recordings. Four sorters (HerdingSpikes2, JRCLUST, KiloSort, and KiloSort2) were unable to sort this type of data.
We observe that these relationships are highly dependent on the spike sorter. For IronClust, the SNR and log ISIvr are predictive of accuracy, whereas firing rate is much less predictive. For KiloSort and SpyKING CIRCUS, firing rate and SNR are both predictive, but log ISI-vr does not appear to correlate. For Kil-oSort2 and MountainSort4, firing rate is the only predictive metric of the three. The final column in this plot shows that a linear combination of metrics is a better predictor than any metric alone. We note that this predictive ability will also depend on the recording type, and, in this case, fidelity of the simulation. For these reasons it is an important future task to extend this type of study across the entire database; see the discussion below.
Discussion We have introduced a Python framework and public website for evaluating and comparing many popular spike sorting algorithms by running them on a large and diverse set of curated electrophysiology datasets with ground truth. We have described the principal features of the website. The Materials and methods section details how we prepared ground truth data of varying types (paired, synthetic, and manually-curated) and apply the algorithms in a uniform, transparent, and unbiased manner. We have summarized initial findings comparing sorter accuracies, and invite the reader to explore the continually updating results on the live site. 2  except showing aggregated precision and recall scores rather than accuracy. Precision measures how well the algorithm avoids false positives, whereas recall is the complement of the false negative rate.
An asterisk indicates an incomplete (timed out) or failed sorting on a subset of results; in these cases, missing accuracies are imputed using linear regression as described in the Materials and methods. Empty cells correspond to excluded sorter/study set pairs. These results reflect the analysis run of March 23rd, 2020. One conclusion is that (as of this time) there is no single sorter that is the most accurate across a diversity of probe types, experimental conditions, metric types and SNR cut-offs. Rather, several different sorters are optimal in different settings. For example, MountainSort4 performs especially well for low channel counts (tetrodes and monotrodes), IronClust excels in the simulated and drifting datasets, while KiloSort2 retains higher accuracy (for units that are found) at lower SNR than other sorters.
, 2012; Freeman, 2015; Berens et al. , 2018)  which allow contributions of (potentially non-reproducible) sorting results, and which report accuracy on held-out data whose ground truth is necessarily private, and thus cannot be interrogated by the community. Although some effort has been made to choose good parameters for each sorter, optimal choices are not guaranteed. We anticipate that a use case for SpikeForest will be finding the best combination of sorter and parameters for a particular recording type. However, automated parameter optimization (as in ClustEval by  Wiwie et al. , 2015)  has not yet been implemented, since it would multiply the total CPU cost by a large factor. For now, we encourage the community to contact us with improved settings or algorithms. Indeed, we reached out to the developers of all ten tested sorters in mid-2019 with preliminary results, and several developers (including at least those of SpyK-ING CIRCUS, HerdingSpikes2, IronClust, and Tridesclous; see Table  1 ) have already used SpikeForest to improve their software.
We emphasize that, for reproducibility, sorter versions and parameter choices used for each date-stamped analysis are available in the downloadable analysis archives linked from the website. This raises the issue of potential overfitting. Because all SpikeForest data are public, the community may continue to optimize sorters using SpikeForest as a benchmark, so one might wonder if this will lead to SpikeForest metrics which overestimate true real-world performance. We believe that such bias will be small, and certainly smaller than the bias of studies conducted in order to propose a single sorting algorithm, for the following reasons. 1) Our database is large and diverse, comprising 13 different types of study sets. It is difficult to imagine overfitting to 650 different recordings by optimizing even dozens of parameters, and, while we allow the potential (as yet unused) for different parameter choices for different studies, multiple recordings would still have to be overfit by one parameter set.
2) All code and parameter settings are available for public inspection, making the above style of parameter and algorithm hacking obvious, were it to occur. We feel that the benefit to the community of improved, validated sorters far outweighs the risk of overfitting. Naturally, Spi-keForest results are biased towards the types of neurons and conditions for which ground-truth data exist; by keeping results for different study sets separate we expose such biases as much as possible. In the long run, we appreciate that both held-out and open data benchmarks will play a useful role in comparison and standardization. Another issue is the paucity of paired ground-truth units in the database, a consequence of the time-consuming nature of their collection. For instance the study set PAIRED_KAMPFF has 15 units, of which only 11 have sufficient SNR to be sortable by any of the sorters.
2. SpikeForest does not directly address the common laboratory task of deciding, given a new sorting of electrophysiology recordings, which neural units are to be trusted and which discarded. Assessing the credibility of putative neural units output by spike sorters remains a crucial question, and many quality metrics are used in practice, including SNR, firing rate, metrics of cluster isolation or separation from the 'noise cluster', stability, and cross-correlograms  (Pouzat et al. , 2002; Schmitzer-Torbert et al. , 2005; Hill et al. , 2011; Neymotin et al. , 2011; Barnett et al. , 2016; Chung et al. , 2017 ). Yet, because of its scale, the SpikeForest database provides a unique opportunity to tabulate such quality metrics for multiple sorter outputs, then correlate each to the actual ground-truth accuracy, with the goal of assessing the predictive power of each metric, or combinations thereof.
For instance, currently KiloSort2 has a larger proportion of crashes or time-outs than many other sorters, yet this is not fully understood and not quantified in accuracy summary results. 8. Finally, we envision that the machinery of SpikeForest could be used for a web-based spike sorting platform, to which users would upload their data (which becomes public), run possibly more than one sorter, visualize, curate, and download the results. This could render all spike sorting and human curation affecting the downstream science analysis accessible and reproducible. We anticipate that, as the use and scale of spike sorting as a tool continues to grow, SpikeForest will become a resource for comparison and validation of sorter codes and encourage more rigorous reproducibility in neuroscience. To this end, we seek contributions and input from the electrophysiology community, both to optimize parameter settings for existing algorithms, and to further expand the set of algorithms and ground-truth recordings included in the analysis.
Sorting algorithms and wrappers Table  2  provides details on the ten algorithms included in the SpikeForest analysis. For each spike sorter, SpikeForest contains a Python wrapper and a Docker (and Singularity) container defining the exact execution environment including all necessary dependencies. For the sorters that are implemented in MATLAB, the MATLAB Compiler tool was used to generate standalone binary files inserted into the containers so that a MATLAB license is not required to run spike sorting. The Spike-Forest framework flexibly allows running of each sorter either within the native operating system or within the container. The former method is useful during development or in an environment where the spike sorting software is already installed. The latter is crucial for ensuring reproducibility and for avoiding conflicts between sorters due to incompatible dependencies.
Analysis pipeline The analysis pipeline of SpikeForest depicted in Figure  6  is built using Python utilities developed by the authors for creating shareable and reproducible scientific workflows. This system provides a formal method for creating well-defined Python procedures that operate on input parameters and files and produce output files. These are known as processors. SpikeForest defines processors for running the spike sorters, computing properties of ground-truth units (e. g. , SNR), comparing the spike sorting outputs with ground truth, and computing summary data for the plots shown on the website. Once these processors are set up, the framework provides several advantages including: (a) automatic execution of processing inside Singularity or Docker containers; (b) automatic caching of processing results; and (c) job management and queueing mechanisms for running batches of processing on a compute cluster.
This allows the analysis pipeline to be defined as a standard Python script, with a simple nested loop iterating through the sorters and ground-truth datasets (as in Figure  1 ). The script may then be configured to run in a variety of settings: on a standard workstation for development, testing, or reproducing of a subset of results; a shared computer with large memory and many cores; or on a compute cluster. A crucial and novel feature of our framework is that all files (both input and output) are represented by SHA-1 URIs, for example of the form: sha1://88b62db6fc467b83ba0693453c59c5-f538e20d5c/firings_true. mda. The hexadecimal code embedded in the URI is the SHA-1 hash of the content of the file, and therefore this URI uniquely identifies the desired file (in this case the ground-truth firing data for one of the SpikeForest recordings) without specifying the actual location of the data.
In contrast, explicit location references (e. g. , a path on the local computer, an IP address, or a web URL) can be problematic because over time data archives may stop being maintained, may change locations, or files may be renamed or updated with new content. The SHA-1 URI system alleviates these difficulties by separating the mechanism for storing archives of files from the representation of these files, via universal hash strings within scripts. All SpikeForest input recordings, ground-truth data, sorting outputs, and other processing data are stored in a public content-addressable storage (CAS) system called kachery. The API of a kachery database simply allows downloading of files referenced only by their SHA-1 hashes (or URIs). Interaction of software and hardware components of the SpikeForest system, showing the flow of data from the server-side analysis (left) to the user's web browser (right).
The processing pipeline automatically detects which sorting jobs need to be updated and runs these in parallel as needed on a compute cluster. Processing results are uploaded to two databases, one for relatively small JSON files and the other for large binary content. A NodeJS application pulls data from these databases in order to show the most up-to-date results on the front-end website. Therefore, as long as the environment is configured to point to a kachery database with the relevant files available, the SpikeForest pipeline may be executed (in whole or in part) on any computer connected to the internet. The automatic caching capability we developed is also crucial for the SpikeForest system. If updates are made to either the database of ground-truth recordings or to the sorting algorithms and parameters, the system can automatically detect which processing needs to be rerun.
In this way, the pipeline can simply be executed in full at regular intervals, and the website will continuously update with the latest changes. This facilitates a conceptually simple method for adding new datasets or sorting algorithms because changes are represented by modifications to pipeline configuration files. A limitation of the reliance on caching is the implicit assumption that the sorting algorithms are deterministic, that is given the same inputs, parameters, and code, the outputs should be exactly the same. However, this assumption can be violated by unfixed random seeds or even differences between implementations of floating-point arithmetic on different hardware. Controlling these factors is currently out of scope for SpikeForest, but we aim to measure the degree of stability  (Barnett et al.
, 2016)  with respect to repeated runs of each sorting task in future versions. To ensure that all results may be reproduced at a later date and/or independently verified by a third party, the Python wrapper for each spike sorter includes a reference to a Docker image containing the entire operating system and environment where the sorter is installed. Our system handles the automatic download of these images (if not already on the local computer) and the execution of the sorting inside Docker or Singularity containers. For development and testing purposes, it is also possible to configure the pipeline to run outside the container, provided that the sorter software is installed on the operating system. The output of processing is a single JSON file (only around 15 MB at this time) containing SHA-1 URI references to all recording files, ground-truth data, containers, spike sorting parameters, sorting outputs, comparisons with ground truth, and other information used by the website.
Website front end The primary user interface for the SpikeForest platform is an isomorphic JavaScript web application with overall structure as shown on the right side of Figure  6. Built from reusable React components, the front end utilizes the D3 library to render the interactive tables and plots. A Node. JS backend, organized using the Redux state container, queries a MongoDB database to retrieve JSON files for each datatype and generate the comparative visualizations. For more detailed plots, like the spike sprays, larger data objects are retrieved via the content-addressable storage database. We optimized the interface for efficient hierarchical navigation through the results and rapid loading and interactive response (for instance, when a user adjusts the SNR or accuracy cut-off slider bar).
When the user clicks on results at the individual recording or unit level, they are taken to a page (with an auto-generated, shareable URL) specific to the study in question. This page allows one-click comparison of the sorters on this study. Clicking on individual units in the scatter plot for a given sorter-study pair brings up spike waveforms for that unit (Figure  3 ) and a link to a sub-page with details specific to the particular sorter-recording run. This latter page includes sorter parameters used, the console output of the run and a link to an auto-generated Python script with humanreadable documentation to reproduce that sorter run within SpikeForest. A permanent top-level menu bar allows access to all meta-data about sorters (as in Table  1 ), study sets (as in Table  2 , also allowing hierarchical drilling down to the individual recording level), historical snapshots, metrics used, and an explanation of the project.
All of the website data and results sub-pages are automatically generated from the SpikeForest databases. Enabling users to run all spike sorters locally on their own data The benefit of having wrappers and Docker images for these ten spike sorting algorithms extends beyond usage within the SpikeForest pipeline. Researchers may also utilize these wrappers and images to spike sort their own data without needing to install and configure the individual sorting packages. As mentioned in the previous subsection, the website provides, for each sorting result, a self-contained Python script for reproducing that result offline. This script may easily be modified to operate on datasets that are not registered within the SpikeForest database. Here is an excerpt of the auto-generated script for running SpyKING CIRCUS on a dataset: # Run the spike sorting with hither.
Figure3. Screenshots from the SpikeForest website. (left) Scatter plot of accuracy vs. SNR for each ground-truth unit, for a particular sorter (KiloSort2) and study (a simulated drift dataset from the SYNTH_JANELIA study set). The SNR threshold for the main table calculation is shown as a dashed line, and the user-selected unit is highlighted. Marker area is proportional to the number of events, and the color indicates the particular recording within the study. (right) A subset of spike waveforms (overlaid) corresponding to the selected ground truth unit, in four categories: ground truth, sorted, false negative, and false positive.
Figure6. Interaction of software and hardware components of the SpikeForest system, showing the flow of data from the server-side analysis (left) to the user's web browser (right). The processing pipeline automatically detects which sorting jobs need to be updated and runs these in parallel as needed on a compute cluster. Processing results are uploaded to two databases, one for relatively small JSON files and the other for large binary content. A NodeJS application pulls data from these databases in order to show the most up-to-date results on the front-end website.
Acknowledgements We have benefited from discussions with  Leslie Greengard ,  Eftychios Pnevmatikakis ,  Andrea Giovannucci ,  Karel Svoboda ,  Loren Frank , and the developers of several of the sorting codes, and from the comments of the anonymous reviewers. The BioNet simulations were made possible by  Catalin Mitelut  and the  Neuroscience Gateway Portal. We thank  Dan English  and colleagues for making their hybrid paired recordings available. We are grateful to the organizers of the Janelia spike sorting workshop, held in March 2018, which helped to inspire this project. The  Flatiron Institute  is a division of the  Simons Foundation.
The risk assessment of railway accidents requires the implementation of several safety analysis methods such as Preliminary Hazard Analysis, Functional Safety Analysis, Analysis of Failure Modes, their Effects and their Criticality and Software Error Effect Analysis (SEEA). The study proposed within the framework of this article concerns the SEEA method whose objective is to determine the nature and severity of the consequences of software failures, to propose measures to detect errors and improve the robustness of the software. The goal is to develop a new approach to analysis and evaluation of the safety of critical software, based on machine learning and more precisely on the Case-Based Reasoning (CBR). The approach adopted involves two main activities: The first step in acquiring safety knowledge consists in extracting, modeling and archiving dangerous situations to produce a library of standard cases which covers the entire problem.
Introduction The basic principle of CBR is to deal with a new problem by remembering similar experiences which have occurred in the past. The objective of the study is the development of a case-based reasoning (CBR) system to help safety experts judge the completeness and consistency of Software Error Effect Analysis (SEEA). The purpose is to exploit historical SEEAs, which have already been carried out on approved safety-critical software, in order to assess SEEA of new software. Very schematically, the objective of this study is to exploit a case base formed by historical SEEA (source case), carried out on already validated and certified software, in order to explain or evaluate a new case of SEEA on new software (target case) and therefore help and stimulate the imagination of experts in the field in the search for new critical situations contrary to safety that requires the implementation of safety barriers or instructions and adequate preventive measures.
This bibliographic study enabled us to position, in paragraph four, our contribution with respect to the state of the art. The fifth paragraph finally proposes a new method of assessment of critical software safety based on the CBR. In order to demonstrate the feasibility and appropriateness of the proposed method, the sixth paragraph presents an example of application which is based on 224 SEEA cases from the knowledge acquisition phase of already certified rail transport systems and commissioning in France (experience feedback). The results obtained are presented in the last paragraph. The harmful consequences of rail accidents, which sometimes lead to loss of life and the destruction of the system and its environment, are at the basis of the implementation of a REX (experience feedback) system considered as the essential means to promote the improvement of safety.
It would not be a feedback on the use and exploitation of data on rail accidents and incidents, but rather data from existing safety files relating to SEEA of two rail transport systems put into service in France. In this context, the REX concerns the data carried out, from the design phase, by the system builders and safety experts. At this stage of the critical software safety analysis (design phase), some of the assumptions developed by the SEEAs can lead to incidents or accidents. However, the use of other complementary safety assessment methods (see Figure  1 ), the implementation of a certification process (carried out by an independent body) and finally the authorization procedure for setting circulation of the system carried out by the national safety authority (EPSF in France) can mitigate or avoid the occurrence of undesirable situations contrary to safety.
Safety software analysis and assessment methods In order to guarantee an acceptable level of risk vis-à-vis humans, the system and its environment, safety experts use several methods of safety analysis shown in Figure  1 : Preliminary Hazard Analysis (PHA), Functional Safety Analysis (FSA), Analysis of Failure Modes, their Effects and their Criticality (AFMEC), Software Error Effect Analysis (SEEA), Fault Tree Analysis (FTA), etc. The PHA aims to identify potential accidents related to the transport system and its interfaces in order to evaluate them and propose solutions to remove reduce or control them. The FSA aims to justify that the design architecture of the system is safe against potential accidents identified by the PHA and therefore to ensure that all safety provisions are taken into account for cover potential hazards or accidents.