dataesr/scientific-paragraphs-categorization huggingface dataset:
Outlier reduction The final component of the proposed matching pipeline is the outlier reduction network. We evaluate its performance in classifying the correspondence heatmaps of the test dataset. We further investigate the effects the inclusion of the ORN has on matching accuracy and finally we evaluate the full matching framework in an end-to-end manner on the test dataset. A binary classification accuracy of 81%, with a precision of 76. 1% and a recall of 89. 5%, was achieved when evaluating the ORN on the test dataset. This shows that the classification of successful matches can be achieved based on the correspondence heatmap alone. Fig. 17    The correspondence surface shapes, as shown in Fig. 17 , highlight that the network relies on more than just the local characteristics of the peak for classification although these do appear to have a relatively strong effect.
In Table  8  we investigate the effect of the outlier reduction network on matching performance. To do so we apply the ORN to the matching heatmaps of both the test dataset matching results, as shown in Table  5 , as well as the minimum fusion (Cross-Min) goodness results, Table  7. The latter resulting in an equivalent end-to-end evaluation of the network. From Table  8  it is clear that the addition of the outlier reduction network substantially increases the accuracy of the resultant set of correspondences, irrespective of the features or regions used for matching. However, the matching performance using the full framework in an end-to-end manner achieves an overall better result with higher accuracy and improved precision.
Large-scale scene matching While we have evaluated the performance the individual subcomponents of our framework, as well as the framework as a whole, these investigations have remained limited to the patch-based test dataset. Thus to fully evaluate the end-to-end performance and applicability of our proposed framework, we apply it to the task of determining correspondence on a large-scale test scene (approximately 0. 8km × 1. 8km) which has not undergone manual co-registration. The example scene is taken from the city of Portsmouth, England and is depicted in Fig. 18  with the final set of correspondences overlaid. To examine the improvement in co-registration we take the mean shift derived from the final set of correspondences and apply this to the optical scene in order to align it with the SAR image.
, 2012) , in the case of our relatively small and flat test scene, such an approach is unlikely to provide a large increase in accuracy over the mean shift method which we followed. Although these results demonstrate the ability of our proposed methodology to accurately determine correspondences between SAR and optical imagery across spatially diverse European test scenes, it is unlikely that they are reflective of the networks performance in regions which have significantly different natural, and man-made structures. For this to be the case the proposed framework would need to be retrained, or fine-tuned on a globally diverse dataset of corresponding pairs of SAR and optical imagery.
Conclusion In this paper we proposed an end-to-end framework for the sparse matching of SAR and optical imagery. The framework consists of three sub-components, each of which were trained to perform a specific task within the standard proposal, matching, outlier detection pipeline. The goodness network proposes candidate patches with a high chance of being matchable in both domains. The correspondence network performs cross correlation on a multi-scale, feature space to produce a correspondence heatmap, which is finally filtered by the outlier reduction network in order to reduce the number of false positive correspondences. We demonstrated that, individually, each of these sub-components improves the matching accuracy and precision achieved on a test dataset in comparison to existing SAR-optical matching approaches, namely NCC  (Burger and Burge, 2009)  and pseudo-siamese  (Hughes et al.
Much development has been directed towards improving the performance and automation of spike sorting. This continuous development, while essential, has contributed to an over-saturation of new, incompatible tools that hinders rigorous benchmarking and complicates reproducible analysis. To address these limitations, we developed SpikeInterface, a Python framework designed to unify preexisting spike sorting technologies into a single codebase and to facilitate straightforward comparison and adoption of different approaches. With a few lines of code, researchers can reproducibly run, compare, and benchmark most modern spike sorting algorithms; pre-process, post-process, and visualize extracellular datasets; validate, curate, and export sorting outputs; and more. In this paper, we provide an overview of SpikeInterface and, with applications to real and simulated datasets, demonstrate how it can be utilized to reduce the burden of manual curation and to more comprehensively benchmark automated spike sorters.
Introduction Extracellular recording is an indispensable tool in neuroscience for probing how single neurons and populations of neurons encode and transmit information. When analyzing extracellular recordings, most researchers are interested in the spiking activity of individual neurons, which must be extracted from the raw voltage traces through a process called spike sorting. Many laboratories perform spike sorting using fully manual techniques (e. g. XClust  [53] , SimpleClust  [72] , Plexon Offline Sorter  [8] ), but such approaches are nearly impossible to standardize due to inherent operator bias  [73]. To alleviate this issue, spike sorting has seen decades of algorithmic and software improvements to increase both the accuracy and automation of the process  [62]. This progress has accelerated in the past few years as high-density devices  [27, 12, 28, 10, 55, 75, 46, 40, 24, 9] , capable of recording from hundreds to thousands of neurons simultaneously have made manual intervention impractical, increasing the demand for both accurate and scalable spike sorting algorithms  [65, 59, 44, 20, 74, 36, 39, 23].
Despite the development and widespread use of automatic spike sorters, there still exist no clear standards for how spike sorting should be performed or evaluated  [62, 11, 18, 47]. Research labs with spikeinterface version 0. 10. 0 which is available on PyPI. The code to perform this analysis and produce all figures can be found at https://spikeinterface. github. io/ which also showcases other experiments performed using SpikeInterface. The datasets are publicly available in NWB format on the DANDI archive (https://gui. dandiarchive. org/ -ID: 00034).
Spike sorters show low agreement for the same high-density dataset The dataset we use in this analysis is a Neuropixels recording from a head-fixed mouse acquired at the Allen Institute for Brain Science (  [69]  -Allen Brain Observatory Neuropixels dataset; ©2019 Allen Institute for Brain Science). The recording has a duration of 15 minutes, 246 active recording channels (the remaining of the 384 Neuropixels channels were either not inserted in the brain tissue or below a firing rate of 0. 1 Hz), and a sampling frequency of 30 kHz. The probe records from part of the cortex (V1), the hippocampus (CA1), the dentate gyrus, and the thalamus (LP). During the experiment, the mouse was presented with a variety of visual stimuli while freely running on a rotating disk (for more details see  [69] ).
An activity map of the probe and a 1-s snippet of the traces on 10 channels are shown in Figure  1A. For this analysis, we select six different spike sorters: HerdingSpikes2  [36] , Kilosort2  [58] , IronClust  [39] , SpyKING Circus  [74] , Tridesclous  [31] , and HDSort  [23]   3. As most of these algorithms have been tuned rigorously on multiple ground truth datasets (including the recent large-scale evaluation from  [47] ), we fix their parameters to default values to allow for straightforward comparison. We do not include Klusta  [65] , WaveClus  [19] , Kilosort  [59] , or MountainSort4  [20]  in this analysis as Klusta can only handle up to 64 channels, WaveClus is designed low channel count probes, Kilosort is superseded by Kilosort2, and MountainSort4 does not scale well to recordings with high channel count (such as Neuropixels ones).
The time window to consider spikes as matching is 0. 4 ms). Of the 2031 total detected units, all 6 sorters agree on just 33 of the units. This is surprisingly low given the relatively undemanding criteria of a 50% spike train match. We also find that two or more sorters agree on just 263 of the total units. To further break down the disagreement between spike sorters, Figure  1D  shows the number of units per sorter for which k other sorters agree. For most sorters, over 50% of the units that they find do not match with any other sorter (with the notable exceptions of Ironclust and Tridesclous). The analysis performed on this dataset suggests that agreement among spike sorters is startlingly low. To corroborate this finding, we repeat the same analysis using different data sets including a Neuropixels recordings from another lab and several in vitro retinal recordings from a planar, highdensity array.
In both cases, we find similar disagreement among the sorters (Figures  S2  and S3 , the results can also be seen at https://spikeinterface. github. io/). This low agreement raises the following question: how many of the total outputted units actually correspond to real neurons? To explore this question, we turn to simulation where the ground truth spiking activity is known a priori. B) The number of detected units for each of the 6 spike sorters (HS=HerdingSpikes2, KS=Kilosort2, IC=IronClust, TDC=Tridesclous, SC=SpyKING Circus, HDS=HDSort). C) The total number of units for which k sorters agree (unit agreement is defined as 50% spike match). D) The number of units (per sorter) for which k sorters agree; Most sorters find many units that other sorters do not.
Evaluating spike sorters on a simulated dataset In this analysis, we simulate a 10 minute Neuropixels recording using the MEArec Python package  [17]. The recording contains the spiking activity of 250 biophysically detailed neurons (200 excitatory and 50 inhibitory cells from the Neocortical Micro Circuit Portal  [60, 49] ) that exhibit independent Poisson firing patterns. The recording also has an additive Gaussian noise with 10µV standard deviation. A visualization of the simulated activity map and extracellular traces from the Neuropixels probe is shown in Figure  2A. A histogram of the signal-to-noise ratios (SNR) for the ground truth units is shown in Figure  2B. We run the same six spike sorters on the simulated dataset, keeping the parameters the same as those used on the real Neuropixels dataset.
We then utilize SpikeInterface to evaluate each spike sorter on the ground-truth dataset. Afterwards, we repeat the agreement analysis from the previous section to diagnose the low agreement among sorters. The main result of the ground-truth evaluation is summarized in The signal-to-noise ratios (SNR) for the ground truth units. C) The number of detected units for each of the 6 spike sorters (HS=HerdingSpikes2, KS=Kilosort2, IC=IronClust, TDC=Tridesclous, SC=SpyKING Circus, HDS=HDSort). D) The accuracy, precision, and recall of each sorter on the ground truth units. E) A breakdown of the detected units for each sorter (precise definitions of each unit type can be found in the SpikeComparison Section of the Methods). Figure  2C , the sorters, again, have a large discrepancy in the number of detected units.
Low-agreement units are mainly false positives Similarly to the real Neuropixels dataset, we compare the agreement among the different spike sorters on the simulated dataset. Again, we observe a large disagreement among the spike sorting outputs with only 139 units of the 1921 total units (7. 24%) being in agreement among all sorters (Figure  3A ). We can break down the overall agreement by sorter (Figure  3B ), highlighting that some sorters are more prone to finding low agreement units (HDSort, SpyKING Circus, Kilosort2) than other sorters (HerdingSpikes2, Ironclust, Tridesclous). Given that we know the ground-truth spiking activity of the simulated recording, we can now investigate whether low-agreement units actually correspond to ground-truth units or if they are falsely detected (false positive) units.
D) Signal-to-noise ratio (SNR) of ground-truth unit with respect to the number of k sorters agreement. Results are split by sorter. we turn to manually curated sorting outputs. Two experts (which we will refer to as C1 and C2) manually curate the spike sorting output of Kilosort2 using the Phy software. During this curation step, the two experts label the sorted units as false positives or real units by rejecting, splitting, merging, or accepting units according to spike features  [63]. Figure  4A  shows the agreement between expert 1 (C1) and expert 2 (C2). While there are some discrepancies (as expected when manually curating spike sorting results  [73] ), most of the curated units (207 out of 351 -64. 2%) are agreed upon by both experts. Notably, 174 units found by Kilosort2 are discarded by both experts, indicating a large number of false positive units.
We then compare the output of each of the spike sorters to C1 and C2 and find that, in general, only a small percentage of units outputted by any single sorter is matched to the curated results (Figure  4 ). showing the agreement between Curator 1 and 2. 174 units are discarded by both curators from the Kilsort2 output. B) Percent of matched units between the output of each sorter and C1 (red) and C2 (blue). Ironclust has the highest match with both curated datasets. C) Similar to C, but using the consensus units (units agreed upon by at least 2 sortersk ≥ 2). The percent of matching with curated datasets is now above 70% for all sorters, with Kilosort2 having the highest match (KS c ∩ C1 = 84. 55%, KS c ∩ C2 = 89. 55%), slightly higher than Ironclust (IC c ∩ C1 = 82. 63%, IC c ∩ C2 = 83. 83%).
D) Percent of non-consensus units (k = 1) matched to curated datasets. The only significant overlap is between Curator 1 and Kilosort2, with a percent around 18% (KS nc ∩ C1 = 18. 58%, KS nc ∩ C2 = 24. 34%). The highest percentage match is actually IronClust which is surprising given that the initial sorting output was curated from Kilosort2's output (IC ∩ C1 = 59. 83%, IC ∩ C2 = 61. 1%, KS ∩ C1 = 50. 67%, KS ∩ C2 = 56. 25%). Next, for each sorter, we take all the units that are matched by at least one other sorter (k ≥ 2consensus units) and all units that are found by only that sorter (k = 1non-consensus units). We refer to the consensus units of a sorter as Sorter c and the non-consensus units of a sorter as Sorter nc. In Figure  4C , we show the match percentage between consensus units and curated units.
Overview of SpikeInterface SpikeInterface consists of five main Python packages designed to handle different steps in the spike sorting pipeline: (i) spikeextractors, for extracellular recording, sorting output, and probe file I/O; (ii) spiketoolkit for low level processing such as pre-processing, post-processing, validation, curation; (iii) spikesorters for spike sorting algorithms and job launching functionality; (iv) spikecomparison for sorter comparison, ground-truth comparison, and ground-truth studies; and (v) spikewidgets, for data visualization. These five packages can be installed and used through the spikeinterface metapackage, which contains stable versions of all five packages as internal modules (see Figure  5 ). With these five packages (or our meta-package), users can build, run, and evaluate full spike sorting pipelines in a reproducible and standardized way.
SpikeExtractors The spikeextractors package 5 is designed to alleviate issues of any file format incompatibility within spike sorting without creating additional file formats. To this end, spikeextractors contains two core Python objects that can directly and uniformly access all spike sorting related files: the RecordingExtractor and the SortingExtractor. The RecordingExtractor directly interfaces with an extracellular recording and can query it for four primary pieces of information: (i) the extracellular recorded traces; (ii) the sampling frequency; (iii) the number of frames, or duration, of the recording; and (iv) the channel indices of the recording electrodes. These data are shared across all extracellular recordings allowing for standardized retrieval functions. In addition, a RecordingExtractor may store extra information about the recording device as "channel properties" which are key-value pairs.
This extra information is either loaded from the sorting output, loaded manually by the user, or loaded automatically with built-in post-processing tools (discussed in the SpikeToolkit Section). Taken together, the SortingExtractor is an object representation of a sorting output along with any associated post-processing. Critically, both Extractor types can lazily query the underlying datasets for information as it is required, reducing their memory footprint and allowing their use for long, large-scale recordings. While this is the default operation mode, Extractors can also cache parts of the dataset in temporary binary files to enable faster downstream computations at the cost of higher memory usage. All extracted data is converted into either native Python data structures or into numpy arrays for immediate use in Python.
Additionally, each Extractor can be dumped to and loaded from a json file, a pickle file, or a dictionary, ensuring full provenance and allowing for parallel processing. The following code snippet illustrates Extractors can be used to retrieve raw traces from an extracellular recording and spike trains from a sorting output: import spikeinterface. extractors as se recording = se. MyFormatRecordingExtractor(file_path='myrecording') sorting = se. MyFormatSortingExtractor(file_path='mysorting') traces = recording. get_traces() #2D numpy array (channels x time) spike_train = sorting. get_unit_spike_train(unit_id=1) #1D numpy array Along with using Extractors for single files, it is possible to access data from multiple files or portions of files with the MultiExtractors and SubExtractors , respectively.
SpikeToolkit The spiketoolkit package 6 is designed for efficient pre-processing, post-processing, validation, and curation of extracellular datasets and sorting outputs. It contains four modules that encapsulate each of these functionalities: preprocessing , postprocessing , validation , and curation. Pre-processing. The preprocessing module provides functions to process raw extracellular recordings before spike sorting. To pre-process an extracellular recording, the user passes a RecordingExtractor to a pre-processing function which returns a new "preprocessed" RecordingExtractor. This new RecordingExtractor , which can be used in exactly the same way as the original extractor, implements the preprocessing in a lazy fashion so that the actual computation is performed only when data 6 https://github.
com/SpikeInterface/spiketoolkit is requested. As all pre-processing functions take in and return a RecordingExtractor , they can be naturally chained together to perform multiple pre-processing steps on the same recording. Pre-processing functions range from commonly used operations, such as bandpass filtering, notch filtering, re-referencing signals, and removing channels, to more advanced procedures such as clipping traces depending on the amplitude, or removing artifacts arising, for example, from electrical stimulation. The following code snippet illustrates how to chain together a few common pre-processing functions to process a raw extracellular recording: import spikeinterface. spiketoolkit as st recording = st. preprocessing. bandpass_filter(recording, freq_min=300, freq_max=6000) recording_1 = st.
preprocessing. remove_bad_channels(recording, bad_channels=  [5] ) recording_2 = st. preprocessing. common_reference(recording_1, reference='median') Post-processing. The postprocessing module provides functions to compute and store information about an extracellular recording given an associated sorting output. As such, post-processing functions are designed to take in both a RecordingExtractor and a SortingExtractor , using them in conjunction to compute the desired information. These functions include, but are not limited to: extracting unit waveforms and templates, computing principle component analysis projections, as well as calculating features from templates (e. g. peak to valley duration, full-width half maximum). One essential feature of the postprocessing module is that it provides the functionality to export a RecordingExtractor / SortingExtractor pair into the Phy format for manual curation later.
Phy  [63, 65]  is a popular manual curation GUI that allows users to visualize a sorting output with several views and to curate the results by manually merging or splitting clusters. Phy is already supported by several spike sorters (including klusta, Kilosort, Kilosort2, and SpyKING Circus) so our exporter function extends Phy's functionality to all SpikeInterface-supported spike sorters. After manual curation is performed in Phy, the curated data can be re-imported into SpikeInterface using the PhySortingExtractor for further analysis. The following code snippet illustrates how to retrieve waveforms for each sorted unit, compute principal component analysis (PCA) features for each spike, and export to Phy using SpikeInterface: import spikeinterface. toolkit as st waveforms = st. postprocessing.
get_unit_waveforms(recording, sorting) pca_scores = st. postprocessing. compute_unit_pca_scores(recording, sorting, n_comp=3) st. postprocessing. export_to_phy(recording, sorting_MS4, output_folder='phy_folder') Validation. The validation module allows users to automatically evaluate spike sorting results in the absence of ground truth with a variety of quality metrics. The quality metrics currently available are a compilation of historical and modern approaches that were re-implemented by researchers at Allen Institute for Brain Science 7  and by the SpikeInterface team (see Table  2 ). Each of SpikeInterface's quality metric functions internally utilize the postprocessing module to generate all data needed to compute the specified metric (amplitudes, principal components, etc. ). The following code snippet demonstrates how to compute both a single quality metric (isolation distance) and also all the quality metrics with just two function calls:.
Silhouette score The ratio between cohesiveness of a cluster (distance between member spikes) and its separation from other clusters (distance to non-member spikes). [66] Table  2 : Currently available quality metrics in Spikeinterface. Re-implemented by researchers at Allen Institute for Brain and by the SpikeInterface team. Curation. The curation module allows users to quickly remove units from a SortingExtractor based on computed quality metrics. To curate a sorted dataset, the user passes a SortingExtractor to a curation function which returns a new "curated" SortingExtractor (similar to how preprocessing works). This new SortingExtractor can be used in exactly the same way as the original extractor. As all curation functions take in and return a SortingExtractor , they can be naturally chained together to perform multiple curation steps on the same sorting output.
Currently, all implemented curation functions are based on excluding units with respect to a userdefined threshold on a specified quality metric. These curation functions will compute the associated quality metric and then threshold the dataset accordingly. The following code snippet demonstrates how to chain together two curation functions that are based on different quality metrics and apply a "less" threshold to the underlying units (exclude all units below the given threshold): import spikeinterface. toolkit as st sorting_1 = st. curation. threshold_firing_rates(sorting, threshold=2.