dataesr/scientific-paragraphs-categorization huggingface dataset:
Among the open-source packages, EEGLAB  (Delorme and Makeig, 2004) , Brainstorm  (Tadel et al. , 2011) , MNE  (Gramfort et al. , 2014) , FieldTrip  (Oostenveld et al. , 2011)  and Automagic  (Pedroni et al. , 2019)  are the most commonly used. Each toolbox has its own way to organize and format the data, to implement functions and to define their arguments, parameters, optimal and default values. Another important difference between tools resides in the availability of the desired preprocessing steps as well as the parameters that can be accessed for each step. Here we investigate how sensitive the results of EEG analyses are to variations in software packages when using the same dataset and aligned preprocessing methods. To this aim, we reanalyzed data (N = 500) from a recent study by Williams and colleagues  (Williams et al.
, 2021)  and reproduced the study using three software packages to quantify the observed differences in the final results. Our objective was first to reproduce the main figures of  (Williams et al. , 2021)  by replicating the original preprocessing pipeline used within each software tool. We compare three of the most commonly used open-source Matlab toolboxes: Brainstorm (first release in 2000, 2559 citations as of 11/10/2022 according to Google Scholar), EEGLAB (first release in 2004, 18373 citations) and FieldTrip (first release in 1999, 7448 citations) in order to achieve two main objectives: 1) Study whether the main findings of the original paperincluding ERP waveform as well as effect size estimates related to selected ERP featurescould be reproduced within each software package. 2) Quantify variations observed across software packages.
Dataset We used the dataset previously analyzed in  (Williams et al. , 2021) , and publicly available at www. osf. io/65x4v/. In brief, this dataset comprises data from 500 undergraduate healthy students (341 females, 154 males, mean age = 21. 71 years old, 440 right handed, 53 left handed) recruited by the University of Victoria. These participants were selected amongst a total of 637 subjects as they had provided signals with a high data quality. The study was approved by the University of Victoria's Human Research Ethics Board and all participants provided written informed consent before any data acquisition. We chose to reproduce this study by Williams and colleagues for two main reasons. First the availability of the raw data and of the preprocessing and analysis scripts made it possible for us to recompute the original results to serve as a reference for our subsequent analyses.
Original preprocessing pipeline The preprocessing pipeline adapted by the reference paper was performed in Matlab using scripts available at www. osf. io/65x4v/(the main file is named 'RewardProcessing_Preprocessing. m'), where some functions have EEGLAB dependencies. Briefly, the pipeline consists of processing data twice wherein the first pass was used to identify noisy or damaged electrodes, and the second pass was done to process data. The steps (Fig. 1B ) are performed as follows: • Reduce the number of electrodes to 32 electrodes (for all data that were collected with a 64 electrode EEG system). • First processing pass -Detect artifactual channels: Practically, the detection of artifactual channels can be approached in different ways. Among these strategies, Williams and colleagues chose to mark as 'bad', the channels that provided a high trial rejection rate.
Comparison across toolboxes We selected three of the most widely used software packages available to preprocess EEGs and reproduced the reference preprocessing pipeline in each. All code to reproduce the preprocessing pipelines is available at: http s://github. com/Inria-Empenn/EEG_preprocessing (released on Zenodo, doi: 10. 5281/zenodo. 6918329) and more details are provided below on the algorithms and parameters chosen in each software package.
EEGLAB. The EEGLAB preprocessing script was assembled and run for all the 500 subjects as follows: • Load the data using pop_loadbv. m • Reduce data into 32 channels using pop_select. m • Automatically detect the noisy channels with the substeps detailed in 'First processing pass -Detect artifactual channels' of section 2. 2. 1 using EEGLAB functions. As these substeps were replicated in the second processing pass (as pointed in section 2. 2. 1), the names of the used functions as well as the parameters selected are listed in the following. • Re-reference the signals to linked mastoid electrodes as performed in 'Second preprocessing pass -Re-reference data to linked mastoids' using prop_reref. m. (a)  We used shared EEG data from  (Williams et al. , 2021)  with 500 participants performing a simple gambling task of six blocks composed of 20 trials.
(b) This dataset was then preprocessed using the different software tools: Reference (using the code published with the original paper), EEGLAB, Brainstorm and FieldTrip. The preprocessing steps to be performed in each tool included: reduction to 32 electrodes, reference to average, automatic detection of bad electrodes, band-pass filtering (0. 1-30 Hz), interpolation of bad channels, segmentation into time-locked epochs (from -500 to 1300 ms around the feedback stimulus) and removal of artifactual trials (identified with 10 μV/ms gradient and 100 μV maximum-minimum criteria). (c) The preprocessed signals derived from the four preprocessing codes were used to reproduce the reference statistics and validate the hypotheses. A quantitative comparison between the resulting signals was also conducted in terms of signal features (peak latency, mean peak, maximum peak and base-to peak features -please refer to materials and methods section for more details).
Brainstorm. The list of steps used to perform the preprocessing in Brainstorm are as follows: • Detect the noisy channels with the substeps detailed in 'First processing pass -Detect artifactual channels' of section 2. 2. 1 using Brainstorm functions. As these substeps were replicated in the second processing pass (as pointed in section 2. 2. 1), the names of the used functions as well as the parameters selected are listed in the following. • Re-reference to linked mastoids (using TP9 and TP10) as performed in 'Second preprocessing pass -Re-reference data to linked mastoids' using process_eegref • Apply the notch filter at 60 Hz using process_notch as performed in 'Second preprocessing pass -Apply notch filter'. • Apply a band-pass filter between 0. 1 and 30 Hz as performed in 'Second preprocessing pass -Apply a band-pass filter', with a linear phase FIR filter using the process process_bandpass.
The filter order calculated was 18128 determined by the means of the Kaiser method. • Interpolate the detected noisy channels as performed in 'Second preprocessing pass -Interpolate the detected artifactual channels', using an interpolation of the neighbors weighted by distance method with process_eeg_interpbad. • Segment data into time-locked epochs as performed in 'Second preprocessing pass -Extract the time-locked events', using process_import_data_event. • Apply baseline correction as in 'Second preprocessing pass -Baseline correction', using the baseline period from -200 ms to 0 using process_baseline. • Detect and reject the bad trials using a peak to peak of 100 μV using process_detectbad, as in 'Second preprocessing pass -Reject trials'. • Compute the ERPs as in 'Second preprocessing pass -Compute the ERPs of gain and loss conditions', using process_average The analysis was conducted using brainstorm version 22.
FieldTrip. FieldTrip toolbox is not a software with a user interface, but rather a collection of functions. Thus, a Matlab script, in which a sequence of FieldTrip functions are called, is considered as an analysis protocol in FieldTrip. Each of the functions of the toolbox takes as input the data that was produced by the previous function. To allow a function to implement a specific algorithm, particular parameters can be specified via a configuration structure cfg. Here, we used the major functions ft_preprocessing, ft_artifact_clip, ft_channelrepair, ft_redefinetrial, ft_rejectartifact and ft_timelockanalysis. More precisely, the FieldTrip preprocessing script was assembled and run for all the 500 subjects as follows: • Bad channels were detected with the substeps detailed in 'First processing pass -Detect artifactual channels' of section 2.
Code availability Codes supporting the results of this study are available at https://gi thub. com/Inria-Empenn/EEG_preprocessing (released on Zenodo, doi: 10. 5281/zenodo. 6918329). All the preprocessing codes were written in Matlab (  2018 )). The visualizations of ERP waveforms (Fig. 2 ) and the quantitative features (Fig. 3 ) were done in R (R Core Team, 2020). Seaborn was used to illustrate the comparisons between the software distribution of the quantitative measures (Fig. 4 ), and the similarity matrix between software tools (Fig. 5 ). Other visualizations and statistical assessments were conducted using Matlab.
The same conclusions regarding the effect size related to the maximum and base to peak measures of the difference ERP, and the mean measure of the ERPs evoked by conditions were derived. In Fig. 2 , we illustrate the ERP waveforms at electrode FCz reflecting the reward positivity, obtained when running the preprocessing code of the reference paper, EEGLAB, Brainstorm and Fieldtrip. The grand averaged ERPs shown in Fig. 2  were obtained after averaging all the clean epochs (kept after the artifactual trials removal step) of all subjects. Changes in the amplitude of the two peaks were noticed for the ERP of gain and loss conditions generated by EEGLAB, Brainstorm and Fieldtrip, when compared to the reference results. An important variability in the number of remained trials/subjects obtained after the preprocessing was observed between the software tools: the number of subjects with clean data was N = 497 for the reference preprocessing, N = 453 for EEGLAB, N = 414 for Brainstorm code, and N = 441 for FieldTrip code.
1 ± 2. 1 μV). The same findings can be observed for the loss waveforms. One can also notice that, for all software tools, the gain waveform elicited higher amplitude than the loss waveform, the loss between 0 ms and 450 ms, whereas the opposite occurs between 450 ms and 1000 ms. According to the grand averaged difference, the reward positivity peaked at a latency of 310 ms for all the different software tools. The peak voltage is increased in Fieldtrip (6. 6 μV ± 2. 1) and decreased in EEGLAB (4. 1 ± 1 μV), Brainstorm (3. 4 ± 1. 1 μV) compared to the reference (4. 8 ± 0. 9 μV). Looking at the quantitative measures, results show good consistency Fig. 2. ERP waveforms at electrode FCz illustrating the reward positivity after preprocessing by: the reference code, EEGLAB, Brainstorm and FieldTrip.
Comparison across software A significant statistical difference was observed between FieldTrip and all the other tools in terms of mean peak amplitude of the gain ERP (Fig. 4 ). EEGLAB showed significant statistical differences with all the other tools in terms of mean peak voltage of the loss ERP. Regarding the features derived from the difference ERP (maximum peak, difference peak, base-to-peak and peak location), no statistical differences were found between software tools. Fig. 5  illustrates the similarity matrix between the gain and loss ERPs generated by the different software tools when taking into account all the EEG channels. For each participant, the similarity between the preprocessed ERPs obtained from two different software tools was calculated using Pearson's correlation averaged across all channels (see materials and methods).
Between the three Matlab toolboxes, FieldTrip reached the highest similarity with the reference pipeline for both conditions (0. 65 ± 0. 22 for gain; 0. 69 ± 0. 18 for loss; 0. 62 ± 0. 16 for the difference), followed by Brainstorm (0. 61 ± 0. 36 for gain; 0. 56 ± 0. 24 for loss; 0. 43 ± 0. 21 for the difference) then EEGLAB (0. 51 ± 0. 26 for gain; 0. 47 ± 0. 22 for loss; 0. 38 ± 0. 21 for the difference). Between the three tested tools, the highest similarity is observed between Brainstorm and FieldTrip (0. 56 ± 0. 29 for gain, and 0. 5 ± 0. 34 for loss; 0. 39 ± 0. 28 for the difference). All the reported correlations are significant with p-value lower than 0. 01. In addition, we investigated where and when the ERPs were statistically different by plotting the thresholded statistical map (time x channels).
To do this, we quantified the statistical difference between the subjects' distribution of ERPs obtained from the reference pipeline and each of the tested tools, at each time sample and each channel using cluster-based permutation test (see materials and methods). In line with the previous findings, Fig. 6  highlights that EEGLAB shows the highest statistical differences compared to the reference results. One can also remark that EEGLAB statistical differences are distributed along the time axis starting from 200 ms to 1000 ms after the stimulus. The major statistical differences between FieldTrip and the reference results were revealed between 400 ms and 600 ms, at some EEG channels adequately. While both the gain and loss conditions showed important differences for all three software packages compared to the reference paper, there were only limited areas of significant differences in the reward positivity (i.
Discussion A large range of techniques and tools are now available to process a single EEG dataset. This high analytical flexibility, reflected by the large number of choices made during the data preprocessing and analysis workflow, can be problematic as it can yield variability in research outcomes. Therefore, it is important to particularly understand the impact of the preprocessing methods, software package, software version and even the operating system on the reproducibility of the final research outcome of a study. Here, we were interested in exploring the impact of the preprocessing software on the ERP derived from EEG data of 500 participants performing a simple gambling task as originally published by  (Williams et al. , 2021). The degree of agreement across software packages was good in terms of peak latencies and the general profile of ERP waveforms.
However, it is noteworthy to clarify that we do not consider that the reference results obtained by the original script are better than those obtained by the other software tools in terms of the quality of the preprocessed signals. The objective of the current study is not to favor any software tool over another or to recommend the 'best' preprocessing tool, but rather, to illuminate and quantify differences Table  1  Mean, maximum, base to peak and the effect size of the reward positivity (the difference ERP) for the reference paper as well as the three studied software packages EEGLAB Brainstorm and FieldTrip. The reported mean (in μV), standard deviation (in μV) and Cohen's d values were computed across subjects. This table is reproduced from Table  1  (first 3 rows) reported in  (Williams et al.
, 2021). 11. 20 μV]   5. 12 μV 2. 99 [2. 34, 3. 20]  Base to peak 13. 49 μV   [12. 75   μV,14. 25 μV]   5. 07 μV 3. 76 [3. 22, 3. 95]  Table  2  Effect size of the gain and loss ERP, using the meak peak measure, for all software. The reported mean (in μV), standard deviation (in μV) and Cohen's d values were computed across subjects. This table is reproduced from Table  2  (first two rows) reported in  (Williams et al. , 2021). that can be generated by different software tools on the same database The variations observed across tools can be related to several factors that are implicated in the preprocessing steps applied in each software. Results are discussed hereafter.
For instance, the band-pass filter cannot be configured in EEGLAB and Brainstorm to have the same type (Butterworth) and order used in the original paper. In addition, the gradient criteria adapted by the original study to detect the bad epochs is not supported by any of the tested software. EEG software packages such as EEGLAB and Brainstorm can choose to restrict the range of parameters that can be freely set by users in order to help practitioners by limiting the choice they have to make to perform their analysis. Among the multiple influencing factors, the filter choice has a substantial impact on the resultant preprocessed signals. Conceptually, as no ideal filter exists, each filter (with the variation of type, order …) affects the temporal structure of EEG signals in both amplitude and phase  (Rousselet, 2012; Vanrullen, 2011; Widmann and Schröger, 2012).
S9A  that the filtered signal obtained using Brainstorm and EEGLAB shows higher peak-to-peak amplitudes compared to the filtered signals obtained using the reference and the FieldTrip filters. Fig. S9B  reveals that while EEGLAB and Brainstorm directly drop the power of undesired frequencies (>30 Hz), the Butterworth filter used in the reference and FieldTrip scripts gradually attenuate the power of these amplitudes. The impact of the filters is directly reflected by the number of subjects and trials kept after trial rejection, as this latter is mainly based on the peak-to-peak criterion. This may explain the reason why Brainstorm has the lowest number of trials and subjects, followed by EEGLAB, FieldTrip and the reference tools (see Table  S5 ). To better understand the effect of the filter type selected by  (Williams et al.
, 2021)  on the results, we filtered the raw signals of the 500 participants using the four filters: the Butterworth filter from the reference paper, EEGLAB FIR filter, Brainstorm FIR filter, and FieldTrip Butterworth filter. Then, for each participant, we assessed the correlation between the filtered signals derived between each pair of tools. Fig. S14A  shows the matrix reporting the correlation values averaged across all participants. FieldTrip reached the highest correlation (r = 0. 99 ± 0. 0007) with the filtered signals generated by the reference filter, followed by EEGLAB (r = 0. 88 ± 0. 06) then Brainstorm (r = 0. 71 ± Fig. 6. The statistical maps showing the differences between the results of each software tool and the reference at each millisecond and channel. 0. 12). One important remark is that the correlation between EEGLAB and Brainstorm (that use the same type of filter -FIR), is lower than that obtained between the reference/FieldTrip and Brainstorm.
This means that the effect of the filter on the signal amplitude also exists even when using similar types of filters showing comparable responses (Fig. S8 ). However, the observed impact on continuous EEG signals does not necessarily imply a similar impact on ERP-derived signals. Therefore, we also explored whether the use of the reference filter has led to the major differences in the ERP amplitudes or not. We thus compared the results obtained by the reference script with the Butterworth filter to those obtained using the same script with only the filter replaced by the FIR filter (as designed by EEGLAB). In Fig. S10 , we show the results obtained after preprocessing following the same channel detection method reported in  (Williams et al. , 2021). By using a simple flat channel detection method, we obtained the results illustrated in Fig.
In this study, we explored the independent effects of three of the factors that may affect the reproducibility of the preprocessed ERP: i-the filtering method, ii-the trial rejection method and, iii-the channel detection method. To explore this effect, the same data were set as inputs for the different methods, and the results were statistically compared using correlation measures (see supplementary information for more details about this analysis). When interpreting the results illustrated in Fig. S14 , one could expect that the difference in the ERP obtained between EEGLAB and the reference is mostly due to the difference in the rejected number of trials, and the bad channels detected. In addition, the difference between Brainstorm and the reference is mostly due to the filter effect. As FieldTrip showed the highest correlations with the reference results when exploring the effects of the filtering, trial rejection and bad channel rejection methods, the derived ERPs were the most comparable to those obtained by the reference script as demonstrated in Figs.
, 2021). In a sample of 132 ERP papers  ( Šoškić et al. , 2021) , reveals that the number of software tools used to perform the EEG analysis stages (from the presentation of stimulus to the statistical assessment) ranged from 8 to 17 options, and that such methodological decisions can induce substantial variability in the reported results, ultimately hindering research replicability. Using fMRI, many studies have quantified the impacts of the analysis software  (Bowring et al. , 2019; Li et al. , 2021) , the software version  (Gronenschild et al. , 2012)  and the operating system  (Glatard et al. , 2015; Gronenschild et al. , 2012)  on results conducted on a single dataset. In the current study, we focused on examining whether it is possible to reproduce the same ERP results after preprocessing data with different software tools.
An interesting future prospect would be testing the consistency of results when varying these factors. To validate the hypotheses supported by the reference paper exploring the same dataset, we compared the results generated by each software to those originally published by  (Williams et al. , 2021). The cross-software discrepancies were also quantified. Another issue that may be of great interest to be investigated is to evaluate the feasibility of each preprocessing tool in generating reliable signals with good data quality. This could be done by comparing results to ground-truth data generated ideally by a computational model of electrophysiological signals such as neural-mass models  (Bensaid et al. , 2019)  or multivariate autoregressive models  (Haufe and Ewald, 2019). In order to conduct the comparative analysis between the results generated by the different software tools, we used several quantification metrics to measure the consistency/discrepancy of ERP waveforms and their related characteristics.
Results show that the mean peak SME was the best (the lowest) for FieldTrip, followed by the reference, then EEGLAB then Brainstorm. A one-way ANOVA revealed that there was not a statistically significant difference in SME between the four tested tool with F(3,497) = 1. 07, and p-value = 0. 37 (greater than 0. 05). These findings demonstrate that even with the difference in the number of accepted trials used in averaging ERPs, the data quality (measured by mean peak SME) was statistically similar. A possible interpretation of this result is that both gain and loss waveforms have a clear peak, on which noise in the data had low impact on the surrounding mean voltage score. Among the available preprocessing tools used in EEG studies, we selected three of the most commonly used open-source software tools.
In each software, we tried to follow, as much as possible, the same preprocessing workflow of  (Williams et al. , 2021)  using the provided software functions. This led us to exclude other interesting packages that conduct a fully automatic preprocessing such as automagic  (Pedroni et al. , 2019) , the Harvard Automated Preprocessing Pipeline for EEG (HAPPE)  (Gabard-Durnam et al. , 2018)  and the Batch Electroencephalography Automated Processing Platform (BEAPP)  (Levin et al. , 2018)  toolboxes. In other words, our inability to control or modify the inclusion and the order of the various preprocessing steps impedes these toolboxes to respect the same preprocessing pipeline we were trying to reproduce. Besides Matlab, it would be interesting to investigate and systematically quantify the differences of results generated by the MNE-python package  (Gramfort et al.