dataesr/scientific-paragraphs-categorization huggingface dataset:
For an increasing number of cell types, the available experimental data already provide diverse constraints on the expected physiological behavior of the neuron under a variety of conditions. Based on various (typically small) subsets of the available constraints, a large number of different models of several cell types have been developed to investigate diverse aspects of single-cell behavior, and for inclusion in realistic circuit models. As an example, there are currently 131different models related to the hippocampal CA1 pyramidal cell (PC) in the ModelDB database  [18]. However, even though these models are publicly available, it is still technically challenging to verify their behavior beyond the examples explicitly included with the model, and especially to test their behavior outside the context of the original study, or to compare it with the behavior of other models.
Implementation of HippoUnit HippoUnit is a Python test suite based on the SciUnit  [19]  framework, which is a Python package for testing scientific models, and during its implementation the NeuronUnit package  [20]  was taken into account as an example of how to use the SciUnit framework for testing neuronal models. In SciUnit tests usually four main classes are implemented: the test class, the model class, the capabilities class and the score class. HippoUnit is built in a way that keeps this structure. The key idea behind this structure is the decoupling of the model implementation from the test implementation by defining standardized interfaces (capabilities) between them, so that tests can easily be used with different models without being rewritten, and models can easily be adapted to fit the framework.
Each test of HippoUnit is a separate Python class that, similarly to other SciUnit packages, can run simulations on the models to generate model predictions, which can be compared with experimental observations to yield the final score, provided that the model has the required capabilities implemented to mimic the appropriate experimental protocol and produce the same type of measurable output. All measured or calculated data that contribute to the final score (including the recorded voltage traces, the extracted features and the calculated feature scores) are saved in JSON or pickle files (or, in many cases, in both types of files). JSON files are human readable, and can be easily loaded into Python dictionaries. Data with a more complex structure are saved into pickle files. This makes it possible to easily write and read the data (for further processing or analysis) without changing its Python structure, no matter what type of object or variable it is.
In addition to the JSON files a text file (log file) is also saved, that contains the final score and some useful information or notes specific to the given test and model. Furthermore, the recorded voltage traces, the extracted features and the calculated feature scores are also plotted for visualization. Similarly to many of the existing SciUnit packages the implementations of specific models are not part of the HippoUnit package itself. Instead, HippoUnit contains a general ModelLoader class. This class is implemented in a way that it is able to load and deal with most types of models defined in the HOC language of the NEURON simulator (either as standalone HOC models or as HOC templates)  [34]. It implements all model-related methods (capabilities) that are needed to simulate these kinds of neural models in order to generate the prediction without any further coding required from the user.
This enables the validation of these models without any manual intervention needed from the user. The section lists required by the tests of HippoUnit are also created automatically using the morphology files of these models (for details see the "Classify apical sections of pyramidal cells" subsection). For neural models developed using other software and methods, the user needs to implement the capabilities through which the tests of HippoUnit perform the simulations and recordings on the model. The capabilities are the interface between the tests and the models. The ModelLoader class inherits from the capabilities and must implement the methods of the capability. The test can only be run on a model if the necessary capability methods are implemented in the ModelLoader. All communication between the test and the model happens through the capabilities.
In this test step currents of varying amplitudes are injected into the soma of the model and the voltage response is recorded. The simulation protocol is set according to an input configuration JSON file, which contains all the current amplitudes, the delay and the duration of the stimuli, and the stimulation and recording positions. Simulations using different current amplitudes are run in parallel if this is supported by the computing environment. As the voltage responses of neurons to somatic current injections can strongly depend on the experimental method, and especially on the type of electrode used, target values for these features were extracted from two different datasets. One dataset was obtained from sharp electrode recordings from adult rat CA1 neurons (sharp electrode data set)  [3] , and the other dataset is from patch clamp recordings in rat CA1 pyramidal cells (data provided by Judit Makara (patch clamp dataset)).
For both of these datasets we had access to the recorded voltage traces from multiple neurons, which made it possible to perform our own feature extraction using eFEL. This ensures that the features are interpreted and calculated the same way for both the experimental data and the models' voltage response during the simulation. Furthermore, it allows a more thorough comparison against a large number of features extracted from experimental recordings yielded using the exact same protocol, which is unlikely to be found in any paper of the available literature. However, to see how representative these datasets are of the literature as a whole we first compared some of the features extracted from these datasets to data available on Neuroelectro. org [37]  and on Hippocampome. org [38]. The features we compared were the following: resting potential, voltage threshold, after-hyperpolarization (AHP) amplitudes (fast, slow), action potential width and sag ratio.
Although these databases have mean and standard deviation values for these features that are calculated from measurements using different methods, protocols and from different animals, we found that most of the feature values for our two experimental datasets fall into the ranges declared as typical for CA1 PCs in the online databases. The only conspicuous exception is the fast AHP amplitude of the patch clamp dataset used in this study, which is 1. 7 ± 1. 5 mV, while the databases cite values between 6. 8 and 11. 64 mV. This deviation could possibly stem from a difference in the way that the fast AHP is measured. We also performed a more specific review of the relevant literature to compare the most important somatic features of the patch clamp dataset to results from available patch clamp recordings (Table  1 ).
Our analysis confirmed that the values of several basic electrophysiological features such as the AP voltage threshold, the AP amplitude, the AP width, and the amplitude of the hyperpolarizing sag extracted from our patch clamp dataset fall into the range observed experimentally. We conclude that the patch clamp dataset is in good agreement with experimental observations available in the literature, and will be used as a representative example in this study. The observation data are loaded from a JSON file of a given format which contains the names of the features to be evaluated, the current amplitude for which the given feature is evaluated and the corresponding experimental mean and standard deviation values. The feature means and standard deviations are extracted using BluePyEfe  [45].
The Back-propagating AP Test This test evaluates the strength of action potential back-propagation in the apical trunk at locations of different distances from the soma. The observation data for this test were yielded by the digitization of Figure  1B  of  [27] , using the DigitizeIt software  [46]. The values were then averaged over distances of 50, 150, 250, 350 ± 20 µm from the soma to get the mean and standard deviation of the features. The features tested here are the amplitudes of the first and last action potentials of a 15 Hz spike train, measured at the 4 different dendritic locations. The test automatically finds current amplitudes for which the soma fires, on average, between 10-20 Hz and chooses the amplitude that leads to firing nearest to 15 Hz. For this task, the following algorithm was implemented.
The PSP Attenuation Test The PSP Attenuation test evaluates how much the post-synaptic potential attenuates as it propagates from different dendritic locations to the soma in CA1 pyramidal cell models. The observation data for this test were yielded by the digitization of Figure  1E  and Figure  2B  of  Magee and Cook, 2000 [29]  using the DigitizeIt software  [46]. The somatic and dendritic depolarization values were then averaged over distances of 100, 200, 300 ± 50 µm from the soma and the soma/dendrite attenuation was calculated to get the mean and standard deviation of the attenuation features at the three different input distances. (1) weight = -EPSC_amp / Vm where EPSC_amp is read from the config dictionary, and the synaptic reversal potential is assumed to be 0 mV. To get the somatic and dendritic maximum depolarization from the voltage traces, the baseline trace (weight = 0) is subtracted from the trace recorded in the presence of the input.
Classifying the apical sections of pyramidal cells Some of the validation tests of HippoUnit require lists of sections belonging to the different dendritic types of the apical tree (main apical trunk, apical tuft dendrites, radial oblique dendrites). To classify the dendrites NeuroM  [50]  is used as a base package. NeuroM contains a script that, starting from the tuft (uppermost dendritic branches in Fig  1 ) endpoints, iterates down the tree to find a single common ancestor. This is considered as the apical point. The apical point is the upper end of the main apical dendrite (trunk), from where the tuft region arises. Every dendrite branching from the trunk below this point is considered an oblique dendrite. However, there are many CA1 pyramidal cell morphologies where the trunk bifurcates close to the soma to form two or even more branches.
In these cases the method described above finds this proximal bifurcation point as the apical point (see Fig  1A ). To overcome this issue, we worked out and implemented a method to find multiple apical points by iterating the function provided by NeuroM. In particular, if the initial apical point is closer to the soma than a pre-defined threshold, the function is run again on subtrees of the apical tree where the root node of the subtree is the previously found apical point, to find apical points on those  We note that this function can only be used for hoc models that load their morphologies from a separate morphology file (e. g. , ASC, SWC) as NeuroM can only deal with morphologies provided in these standard formats. For models with NEURON morphologies implemented directly in the hoc language, the SectionLists required by a given test should be implemented within the model.
We also added section lists of the radial oblique and the trunk dendritic sections to those models where this was not done yet, as some of the tests require these lists. To ensure that the standalone versions have the same properties as the original models, we checked their parameters after running their built-in simulations (in case including any run-time modifications), and made sure they match the parameters of the standalone version. The modified models used for running validation tests are available in this GitHub repository: https://github. com/KaliLab/HippoUnit_demo.
Results The HippoUnit validation suite HippoUnit (https://github. com/KaliLab/hippounit) is an open source test suite for the automatic and quantitative evaluation of the behavior of neural single cell models. The tests of HippoUnit automatically perform simulations that mimic common electrophysiological protocols on neuronal models to compare their behavior with quantitative experimental data using various feature-based error functions. Current validation tests cover somatic (subthreshold and spiking) behavior as well as signal propagation and integration in the dendrites. These tests were chosen because they collectively cover diverse functional aspects of cellular behavior that have been thoroughly investigated in experimental and modeling studies, and particularly because the necessary experimental data were available in sufficient quality and quantity.
However, we note that the currently implemented tests, even in combination, probably do not fully constrain the behavior of the cell under all physiological conditions, and thus the test suite can be further improved by including additional tests and more experimental data. The tests were developed using data and models for rat hippocampal CA1 pyramidal cells. However, most of the tests are directly applicable to or can be adapted for other cell types if the necessary experimental data are available; examples of this will be presented in later sections. HippoUnit is implemented in the Python programming language, and is based on the SciUnit  [19]  framework for testing scientific models. The current version of HippoUnit is capable of handling single cell models implemented in the NEURON simulator, provided that they do not apply any runtime modification, do not have a built-in graphical user interface, and do not automatically perform simulations.
Meeting these conditions may require some modifications in the published code of the model. Once such a "standalone" version of the model is available, the tests of HippoUnit can be run by adapting and using the example Jupyter notebooks described in S1 Appendix, without any further coding required from the user. In principle, neural models developed using other software tools can also be tested by HippoUnit; however, this requires the re-implementation by the user of the interface functions that allow HippoUnit to run the necessary simulations and record their output (see the Methods section for more details). In the current tests of HippoUnit, once all the necessary simulations have been One convenient way of running a test on a model is to use an interactive computational notebook, such as the Jupyter Notebook  [55] , which enables the combination of program codes to be run (we used Python code to access the functionality of HippoUnit), the resulting outputs (e.
g. figures, tables, text) and commentary or explanatory text in a single document. Therefore, we demonstrate the usage of HippoUnit through this method (See S1 Appendix and https://github. com/KaliLab/HippoUnit_demo). Comparison of the behavior of rat hippocampal CA1 pyramidal cell models selected from the literature We selected six different publications containing models of hippocampal CA1 pyramidal cells whose implementations for the NEURON simulator were available in the ModelDB database. Our aim was to compare the behavior of every model to the experimental target data using the tests of HippoUnit, which also allowed us to compare the models to each other, and to test their generalization performance in paradigms that they were not originally designed to capture. These models differ in their complexity regarding the number and types of ion channels that they contain, and they were built for different purposes.
Somatic Features Test Using the Somatic Features Test of HippoUnit, we compared the behavior of the models to both patch clamp recordings (patch clamp dataset) and sharp electrode recordings (sharp electrode dataset). After performing a review of the relevant literature, we conclude that the patch clamp dataset is in good agreement with experimental observations available in the literature (see Table  1  in Methods), and will be used as a representative example in this study. The two datasets used in this study (sharp electrode dataset, patch clamp dataset) differ not only in the recording technique, but also in the simulation protocol. In the sharp electrode recordings, the cells received 400 ms-long depolarizing and hyperpolarizing current injections, using amplitudes of 0. 2, 0. 4, 0. 6, 0.
8 and 1. 0 nA in both directions. In the patch clamp recordings, both the depolarizing and the hyperpolarizing current injections were 300 ms long and 0. 05, 0. 1, 0. 15, 0. 2, 0. 25 nA in amplitude. As each of the tested models apparently used experimental data obtained from patch clamp recordings as a reference, here we show the detailed results of the test on the models when their output was compared to the features extracted from the patch clamp data (we will return to the comparison between the two datasets near the end of this section). During these recordings the cells were stimulated with relatively low amplitude current injections. Some of the examined models ( did not fire even for the highest amplitude tested. Some other models started to fire for higher current intensities than it was observed experimentally.
To quantify the difference between the experimental dataset and the simulated output of the models, these were compared using the feature-based error function (Z-Score)  Besides enabling the comparison of different models regarding how well they match a particular dataset, the tests of HippoUnit also allow one to determine the match between a particular model and several datasets of the same type. As experimental results can be heavily influenced by recording conditions and protocols, and also depend on factors such as the strain, age, and sex of the animal, it is important to find out whether the same model can simultaneously capture the outcome of different experiments, and if not, how closely it is able to match the different datasets. As a practically relevant example, we looked at how well the various published models that we were testing captured a different experimental dataset that also contained current clamp recordings from rat CA1 PCs, but which was obtained using sharp electrodes rather than the whole-cell patch clamp technique  [3].
We therefore evaluated all the models with the Somatic Features Test of HippoUnit using both datasets, and then compared the results. described When we simply compared the raw outputs of the test for each model evaluated using the two different data sets (Fig 5A ) we identified two factors that substantially bias the results. First, we found that the standard deviation values for the features extracted from the two datasets are very different in magnitude; more specifically, the patch clamp recording data set had much lower standard deviation values for most of the features. This results in relatively higher feature scores achieved by the models, as the difference of the model output from the experimental features is given in the unit of the experimental standard deviation. The other source of bias is that the two datasets were recorded not only using different recording methods -patch clamp and sharp electrode -but (partly as a consequence) also using different protocols (current amplitudes, current duration etc.
), and therefore provide different sets of features. As an important example, voltage traces with and without action potentials clearly provide different types of features. Also note that the same electrophysiological parameter can often be extracted from multiple voltage traces, and these are all treated as separate features in the test, so a difference in the number of recorded traces automatically leads to a difference in the set of features. Consequently, the models are not compared to exactly the same set of features in the two cases. Mainly as a result of these two confounding factors, comparison of the raw scores of the models for the two data sets (Fig 5A ) appears to indicate that most models fit the dataset obtained from sharp electrode recordings better, even though these models were typically built mostly based on patch clamp data.
compared to the number of features attempted to be evaluated (successfully extracted from the data set). Note that while in the recalculated final scores (B) only those eFEL features were taken into account that could be extracted from both datasets, they are extracted for different current step amplitudes, which accounts for the difference in the number of observation features for the two datasets. To overcome these issues and make unbiased comparisons of the models to the two datasets, the feature scores and the final scores were recalculated in the following way (Fig It is worth noting that one biasing factor still remains in the last comparison: as it has already been mentioned, not all the observation features can be evaluated for each of the models, especially when they are compared to the patch clamp data set, which uses smaller currents.
The model files that were used to test the models described above, the detailed validation results (all the output files of HippoUnit), and the Jupyter Notebooks that show how to run the tests of HippoUnit on these models are available in the following Github repository: https://github. com/KaliLab/HippoUnit_demo. Application of HippoUnit to models built using automated parameter optimization within the Human Brain Project Besides enabling a detailed comparison of published models, HippoUnit can also be used to monitor the performance of new models at various stages of model development. Here, we illustrate this by showing how we have used HippoUnit within the HBP to systematically validate detailed multi-compartmental models of hippocampal neurons developed using multi-objective parameter optimization methods implemented by the open source Blue Brain Python Optimization Library (BluePyOpt  [16] ).
To this end, we extended HippoUnit to allow it to handle the output of optimization performed by BluePyOpt (see Methods). Models of CA1 pyramidal cells were optimized using target feature data extracted from the same sharp electrode dataset  [3]  that was also one of the datasets used by the Somatic Features Test of HippoUnit. However, while during validation all the eFEL features that could be successfully extracted from the data are considered, only a subset of these features was used in the optimization (mostly those that describe the rate and timing of the spikes; e. g. , the different inter-spike interval (ISI), time to last/first spike, mean frequency features). In addition, sharp electrode measurements were also available for several types of interneuron in the hippocampal CA1 region, and models of these interneurons were also constructed using similar automated methods  [3].
All the models that were tested and compared in this study (including the CA1 pyramidal cell models from the literature and the BluePyOpt optimized CA1 pyramidal cells and interneurons of the HBP) have been registered and are available in the Model Catalog of the Validation Framework with their locations in the CSCS storage linked to them. In addition to the modifications that were needed to make the models compatible with testing with HippoUnit (described in the section "Methods -Models from literature"), the versions of the models uploaded to the CSCS container also contain an __init__. py file. This file implements a python class that inherits all the functions of the ModelLoader class of HippoUnit without modification. Its role is to make the validation of these models via the Framework more straightforward by defining and setting the parameters of the ModelLoader 57 class (such as the path to the HOC and NMODL files, the name of the section lists, etc.