dataesr/scientific-paragraphs-categorization huggingface dataset:
In the field of neuroimaging, voxels are used as the typical representation of data, while the available representation in tractography is in the form of streamlines (i. e. , sets of 3D points in space). Figure  2  is a sketch of both representation. Several similarity measures exist to compare voxel-wise segmentations, for example, Dice score. Most of them have an equivalent formulation to compare sets of streamlines. However, resulting values can widely vary as the spatial distribution is not the same for both representations. Some measures related to streamlines require the datasets to be exactly the same, for example, Dice score, as streamline reconstructions are sets of discrete points with floating point coordinates and not discrete grids like 3D images. For this reason, comparison of streamlines is more challenging and datasets that do not originate from the same source distance in millimeters is often the only available solution  (Garyfallidis et al.
We do not want to promote a given dissection protocol but emphasize the fact that any new protocol, Five independent tractograms and their associated structural/diffusion images were used, each was triplicated (total of 15). One was untouched, one was flipped in the X-axis (left/right), and one was translated. This was done to ensure that the participants were not aware they were performing reproducibility tasks. The symmetry of the segmentation plan (no difference between hemispheres) and lack of absolute frame of reference (coordinates) allowed these operations. Then, all datasets were randomly named so the tasks could be performed blindly for each participant. Participants were not aware of the presence of duplicated datasets. Five tractotrams were used to obtain stable averages, duplicated datasets were used to score the intra-rater agreement and the multiple participants to evaluate interrater agreement.
The decision to separate participants in two groups was made to generate additional data about reproducibility in real-life conditions. Figure  3  shows an overview of the study design. To evaluate intra-rater reproducibility of rater #1, each triplicate was used to compute reproducibility measures. Meaning that 5 (A-B-C-D-E) × 3 (1-2-3) values were averaged to obtain the intra-rater "reproducibility score" of a single rater. To evaluate inter-rater reproducibility of rater #1, triplicates were fused and compared to all other raters to obtain a reproducibility measure. Meaning that 5 (A-B-C-D-E) × N (raters) values were averaged to obtain a single inter-rater "reproducibility score. " To evaluate the reproducibility of rater #1 against the gold standard, the fused triplicates were also used.
| DWI datasets, processing, and tractography to make sure sufficient density and spatial coverage were achieved. The decision to use high-quality data was to maximize the quality of the tractogram so it would not be a limiting factor for the segmentation tasks. Moreover, it is now more and more common in clinical research to reach resolution as high as 1. 5 mm isotropic with multishell schemes, such dataset generates tractograms on a similar quality range as the HCP. The CSD model was also used for bundle-specific tractography (BST) to further improve density and spatial coverage of the bundle of interest  (Rheault et al. , 2019). This was to ensure that the full extent of the PyT was reconstructed and to ensure not to have criticisms from our experts in neuroanatomy complaining of a lack of fanning  (Pujol et al.
And these results were fused to a conventional whole brain CSD probabilistic tractogram. The resulting tractogram provided to the participants is indistinguishable from a conventional one. The rationale for this decision to use a more efficient seeding/tracking method to fill-up the full spatial extent of the PyT, as opposed to generating 10-50 millions of streamlines. To accommodate all participants and the wide range of computer performance, tractograms were compressed using a 0. 2 mm tolerance error  (Presseau, Jodoin, Houde, & Descoteaux, 2015; Rheault, Houde, & Descoteaux, 2017)  and commissural streamlines were removed and datasets split into hemispheres. Each hemisphere (of each subject) had approximately 500,000 streamlines.
| Dissection plan and instructions Each participant received their randomly named datasets, a document containing instructions for the segmentation and a general overview of a segmentation as an example (see Supplementary Information). The segmentation task consisted in 15 segmentations of the pyramidal tract (left and right). The rationale behind the decision to focus on this PyT bundle was first, that a well-defined and well-known pathway was desired. Second, a dissection plan made of small and large inclusion and exclusion regions was desired. Finally, that the general shape was intuitive so the participants with no background in neuroanatomy could perform the tasks. Segmentation involved using three WM inclusion ROIs (internal capsule, midbrain, and medulla oblongata) and two exclusion ROIs (one plane anterior to the precentral gyrus and one plane posterior to the postcentral gyrus).
The detailed segmentation plan is available in the Supporting Information  (Chenot et al. , 2019). Participants had to perform the segmentation plans, following the instructions as closely as possible. The dataset order was provided in a spreadsheet file. Participants had to choose between two software; Trackvis  (Wang et al. , 2007)  or MI-Brain  (Rheault et al. , 2016). This decision was made to guarantee that the data received from all participants was compatible with the analysis. Metadata such as date, starting time, and duration had to be noted in the spreadsheet file. Upon completion, the participants had to send back F I G U R E 3 Representation of the study design showing N participants, each received five HCP datasets (listed and color coded) which were replicated three times (original, flipped, translated).
| Bundles analysis Once returned by all participants, datasets were de-randomized to match triplicates across participants. The duplicates (flipped and translated) were reverted back to their native space and all datasets (images and tractograms) were warped to a common space (MNI152c 2009 nonlinear symmetrical) using the Ants registration library  (Avants, Epstein, Grossman, & Gee, 2008; Fonov et al. , 2011)  to simplify the analysis. With all datasets having a uniform naming convention and in a common space, the intra-rater and inter-rater reproducibility can be assessed.
| Intra-rater and inter-rater Each participant performed the same tasks on each triplicate. The goal of this triplication is to evaluate intra-rater reproducibility. Since all participants had access to the same datasets, inter-rater reproducibility can be assessed too. Computing the average value from all pairwise combinations provides an estimate of the agreement between multiple segmentations of a same bundle. The deviation can also provide insights about the consistency of these segmentations. Measurement values can be between 0 and 1, such as Dice and Jaccard  (Dice, 1945) , meaning they are independent of the size. Figure  4  shows bundles and how to interpret these measures. Pearson's correlation coefficient obtained from density maps provides insight into the statistical relationship and spatial agreement between two segmentations  (Hyde & Jesmanowicz, 2012).
| Gold standard When multiple raters provide segmentations from an identical dataset, it is of interest to produce a gold standard. For a voxel representation, a probability map can be constructed, where each voxel value represents the number of raters that counted the voxel as part of their segmentation  (Frisoni et al. , 2015; Iglesias & Sabuncu, 2015; Langerak, van der  Heide, Kotte, Berendsen, & Pluim, 2015;  Pipitone et al. , 2014). This can be normalized and then thresholded to obtain a binary mask representing whether or not the voxel was segmented by enough rater. A threshold above 0. 5 is often referred to as a majority vote. The same logic can be applied to streamlines, each streamline can be assigned a value based on the number of raters that considered it part of their segmentation.
This can be seen in Figure  5   To produce our gold standard, a majority vote approach was used from the segmentations of the experts group, as their knowledge of anatomy was needed to represent an average version of the bundle of interest. In the context of this work, the gold standard is actually the average segmentation from experts. For simplicity, the expression gold standard was used as it is the best approximation of what can/could be achieved by our group of experts. The vote was set at 6 out of 11 and each of the five datasets got its own left and right gold standard. Since the representation at hand is streamlines (which can be converted to voxels), a streamline-wise and a voxel-wise gold standard was created. A majority vote approach is not necessarily optimal, but in the context where experts could not collaborate beforehand or after, this approach is adequate to obtain an average representation of the segmentation.
| Inter-rater evaluation To minimize the influence of intra-rater reproducibility during the evaluation of inter-rater reproducibility, the triplicate datasets were fused into a single bundle. This was performed to approximate the results as if participant segmentations had no intra-rater variability. This leads to an underestimation of inter-rater variability but necessary to separate sources of variability later in the analysis. Voxel-wise Dice coefficient is on average higher between experts than between nonexperts, at 0. 62 and 0. 67, respectively, while the streamline-wise Dice coefficient is much lower at 0. 11 and 0. 18. The median density correlation is higher between experts at 0. 88 while nonexperts are at 0. 71. The IQR is higher for the nonexperts group, meaning that the similarity among nonexperts is not only lower but widely varies.
0, the Youden score is almost equal to sensitivity. All three measures take into account the true negatives, which far outweigh the true positives, in our datasets, for this reason they were removed from Figure  10  and shown only in the Supporting Information. Sensitivity is much lower at 0. 78 and 0. 82 for experts and nonexperts respectively, as both groups partially capture the gold standard. Precision is higher for experts (0. 92) than for nonexperts (0. 78), meaning that experts were providing segmentations approximately the same size as the gold standard while nonexperts were providing much bigger segmentations (that generally encompass the gold standard). This explains the higher sensitivity and lower specificity of nonexperts. The Kappa and Dice score is lower for experts at 0. 83 and 0.
It is important to consider that the ratio of true negatives to true positives is not the same for both representations (voxels vs. streamlines). The computation of inter-rater reproducibility was performed using the fused triplicate to minimize the influence of intra-rater reproducibility. The approach to fuse the triplicate is an approximation, fusing more than three segmentations from the same datasets would be necessary to perfectly evaluate intra-rater reproducibility. However, the five datasets used for this study represent sufficiently similar tasks to consider our approximation adequate for this work. Preliminary analysis showed low correlation values, between a participant "score" for intra-rater reproducibility and inter-rater reproducibility. Correlation was between 0. 2 and 0. 4 for all measures, this indicates that there is no clear link between the reproducibility of a participant's own segmentations and the agreement with other participants.
| Evaluation of protocols This work illustrates that intra-rater and inter-rater agreement is far from perfect even when following a strict and "simple" segmentation protocol. The intra-rater and inter-rater agreement must be taken into account when researchers compare bundles obtained from manual segmentations. When human expertise is required for a project, it is crucial that people involved in the manual segmentation process evaluate their own reproducibility, even if they have sufficient neuroanatomy knowledge and extensive experience in manual segmentation. This measure of error could increase the threshold for statistical significance. In such case, either more datasets will be needed or a better protocol for segmentation needs to be designed  (Boccardi et al. , 2015; Gwet, 2012). The similarity between both groups indicates that with the right protocol, it would be possible to train people without anatomical background to perform tasks with results and quality similar to experts.
Most measures used are simplified to have easily interpretable results. The previous example shows the importance of using more than one type of measurements to obtain a complete picture of the reproducibility. "Reproducibility scores" are likely to vary with the project and the bundle of interest. This needs to be addressed as a community. The discrepancy between protocol quality, reproducibility, and conclusion put forward in the literature can be problematic. For binary measures (accuracy and specificity), scores were both above 95% as it is easy to discard true negatives and consequently did not provide much insight. Similarly to the curse of dimensionality in machine learning  (Ceotto, Tantardini, & Aspuru-Guzik, 2011; Verleysen & François, 2005) , our datasets typically contain millions of voxels (or streamlines), of which only a few thousands true positives are considered during segmentation.
Thus, the vast majority of true negatives are rapidly discarded resulting in both accuracy and specificity almost reaching 100%. Sensitivity provides more information, as true positives are more difficult to get, since they are rarer in the tractograms (few thousands out of millions)  (Maier-Hein et al. , 2017). This needs to be taken into account using precision, as in some cases, strict segmentation is encouraged because false positives are more problematic than false negatives. Streamline-wise measures show lower agreement, meaning that reproducible results are likely more difficult to achieve with the streamline representation. More complex measures need to be designed to fully capture the complexity of tractography datasets and compare them, even across datasets or for longitudinal studies.
However, this is in line with our main message, which is that every project involving manual segmentation should come with its own reproducibility assessment. Hence, changing tractography algorithm would require a re-evaluation of the reproducibility is considered as part of important future work. For example, the project "TractEm"  (Bayrak et al. , 2019)  featured a framework to obtain 61 bundle of interests from deterministic tractography and report some voxel-wise measures for intra-rater and inter-rater reproducibility. However, this protocol is likely optimized for specific datasets (BLSA, HCP). It also requires registration and tractograms must be generated with DSI-Studio (deterministic) only. Any deviation from this protocol would likely change the reported reproducibility measures.
Using the same dataset and tractography algorithm, but increasing or decreasing the number of streamlines, variation in step-size or angular threshold could also change the reproducibility "scores. " Another anatomical definition of the PyT having that definition taught to participants in person instead of a simple PDF document or dissections of another BOI would likely lead to different reproducibility "scores. " Other dataset could come with their own challenges, for example, infants or aging population, where finding anatomical landmarks could be harder and thus lead to lower reproducibility. Such trend can be observed in numerous other studies where investigation of the same bundle, different bundles or when different algorithmic choice leads to a wide variety of reported reproducibility "scores"  (Colon-Perez et al.
| Impact on analysis If variability needs to be minimized further than the defined protocol, a simple recommendation is to have a single rater performs each task multiple times or multiple raters perform each task multiple times (or a subset of tasks). This way, it is guaranteed that each dataset is segmented more than once, decreasing the error risk. Regardless of the decision made, it is of great importance to quantify the reproducibility of manual segmentation of raters involved in the project before doing any statistics or group comparisons. This could drastically change the statistical significance of results. As of now, to the best of our knowledge, human variability and errors are rarely considered. Measurements such as volume and streamlines count can take into account the measurement error (i.
In a very thorough longitudinal large-scale project across centers involving manual segmentation, it would be desirable to acknowledge the variability across timepoints, across scanners, and across rater into the analysis. The extension to other bundles of interest or other segmentation plans is not trivial and the only conclusion that stands is that agreement is never 100% and that a unique measure is not enough to represent the whole picture for tractography segmentation. The desire to simplify measures or have only one value to describe quality or reproducibility of segmentations needs to be discouraged. The nature of our datasets makes this task much more complex to interpret than 2D or 3D images, and it is imperative that the field comes to understand and agree on measures to report. This is more relevant than ever as the field grows and now that open science is becoming more popular and reproducibility studies are encouraged.
| Future work Future work includes the creation of a database containing bundle segmentations and metadata from participants that will be available online so further analysis can be done. This metadata could help explain the variability, similar trend has been observed for most types of measurements, medical images, or tractography segmentation  (Boccardi et al. , 2011; Bürgel et al. , 2009; Gwet, 2012). As for now, a preliminary upload of the participants segmentation is available on Zenodo (https://doi. org/10. 5281/zenodo. 2547024), which will be updated. In this work, metadata was not used to evaluate duration as a variable influencing reproducibility. Investigating the relationship between variability and duration of a task or looking for bias (interhemispheric or software influence) could be of interest for future research.
including a new tractography algorithm and another bundle of interest would require a new assessment. A second contribution of this work is to investigate overlap, similarity measures, and gold standard comparison designed for tractography. Development of easily interpretable measures for bundle comparison is necessary for large datasets. Overall, the voxel representation is significantly more reproducible than the streamline representation. The voxel representation is better suited for analysis of tractography datasets (e. g. , reporting volume instead of streamline count). More details about these different representations and voxel/ streamline-wise measures will be detailed in Sections 2 and 3. 2 | METHOD2. 1 | Study designTwenty-four participants were recruited and divided into two groups: experts and nonexperts.
The division was based on their neuroanatomical educational background. Participants working as researchers or PhD students in neuroanatomy, neurology or with extended experience in the field performing "virtual dissection" as well as neurosurgeons were part of the experts group (11 participants). The nonexperts group was composed of MSc, PhD student or PostDoc in neuroimaging, but without any formal education in neuroanatomy (13 participants). All participants had knowledge of dMRI tractography in general as well as the concept of manual segmentations of tractography datasets. Participation was voluntary and anonymous, recruitment was done individually and participants from various laboratories in Europe and the United States were solicited. The study was performed according to the guidelines of the Internal Review Board of the Centre Hospitalier Universitaire de Sherbrooke (CHUS).
datasets and other variables points to the importance our proposed framework for evaluation, as a step forward. It is of importance to reiterate that the intention of this study is not to propose/enforce a processing pipeline for tractography and/or propose a new set of rules for PyT segmentation. The diversity of reported values for reproducibility shows that clearly defined processing and segmentation protocol is necessary in this field. All data and metadata used in this work are now publicly available (https://doi. org/10. 5281/zenodo. 2547024) in the hope to stimulate discussions and more evaluations in the future for other bundles and protocols. Better reproducibility of results is needed and goes hand-in-hand with the open science movement. A collaborative effort to evaluate and quantify human variability is needed.
e. variability of decisions). Our main goal was to decide between the two listed strategies that can solve the task: repeating rewarded sequences or choosing randomly. Therefore, we momentarily put aside the question of learning speed and only considered the model behavior after convergence. α was set to 0. 1 in these simulations. Hyperparameters were selected through random search  30  (see ranges listed in Supplementary Table  1 ). The model was run for 2. 10 6 iterations for each parameter set. The fitness score with respect to mice average data at each session was calculated as follows: fitness ¼ 1 À D session ð4Þ with D session ¼ 1 2 Ŝ À S þ Ĉ À C À ð5Þ where S and C are the average success rate and complexity in mice respectively and Ŝ and Ĉ the model success rate and complexityall the four ∊ [0, 1].
A sample code for the model fitting procedure is publicly available at https:// zenodo. org/record/2564854#. Xe07NB-YUpg (see ref. 31  ). Markov chain analysis. Markov chain analysis allows to mathematically describe the dynamic behavior of the system, i. e. transitions from one state to another, in probabilistic terms. A process is a first-order Markov chain (or more simply Markovian) if the transition probability from state A to a state B depends only on the current state A and not on the previous ones. Put differently, the current state contains all the information that could influence the realization of the next state. A classical way to demonstrate that a process is Markovian is to show that the sequence cannot be described by a zeroth-order process, i. e. that P(B|A) ≠ P(B), and that the second-order probability is not required to describe the state transitions, i.
All statistical tests were applied using the scipy. stats Python module. They were all two-sided except Mann-Whitney. p > 0. 05 was considered to be statistically nonsignificant. In all Figures: error bars represent 95% confidence intervals. *p < 0. 05, **p < 0. 01, ***p < 0. 001. n. s. , not significant at p > 0. 05  33. Reporting summary. Further information on research design is available in the Nature Research Reporting Summary linked to this article.
Acknowledgements This work was supported by the  Centre National de la Recherche Scientifique   CNRS UMR 8246 et UMR 7222 , the  Labex SMART  ( ANR-11-LABX-65 ) supported by  French  state funds managed by the  ANR  within the  Investissements d'Avenir programme  under reference  ANR-11-IDEX-0004-02 , the  Foundation for Medical Research (FRM, Equipe FRM   EQU201903007961  to P. F),  ANR  ( ANR-16  Nicostress to PF), the  French National Cancer Institute  Grant  TABAC-16-022  (to P. F. ). P. F. team is part of the  École des Neurosciences de Paris Ile-de-France RTRA network and member of LabEx Bio-Psy.