dataesr/scientific-paragraphs-categorization huggingface dataset:
Variability of reported results A set of mixed-effects logistic regression models identified several analytical variables and image features that were associated with reported outcomes (Extended Data Table  3c ). The strongest factor was spatial smoothness; higher estimated smoothness of the unthresholded statistical maps (estimated using the FMRIB Software Library (FSL) smoothest function) was associated with a greater likelihood of significant outcomes (P < 0. 001, delta pseudo-R 2 = 0. 04; mean full width at half-maximum, 9. 69 mm, range 2. 50-21. 28 mm across teams). Notably, although the estimated smoothness was related to the width of the applied smoothing kernel (r = 0. 71; median applied smoothing 5 mm, range 0-9 mm across teams), the applied smoothing value itself was not significantly related to positive outcomes in a separate analysis, suggesting that the relevant smoothness arose from analytical steps beyond explicit smoothing (such as modelling of head motion; P = 0.
014). An effect on outcomes was also found for the software package used (P = 0. 004, delta pseudo-R 2 = 0. 04; n = 23 (SPM), n = 21 (FSL), n = 7 (AFNI) and n = 13 (other software package))-with FSL being associated with a higher likelihood of significant results across all hypotheses compared to SPM; odds ratio = 6. 69)-and for the effect of different methods of multiple test correction (P = 0. 024, delta pseudo-R 2 = 0. 02: n = 48 (parametric), n = 14 (nonparametric), n = 2 (other)), with parametric correction methods resulting in higher rates of detection than nonparametric methods. No significant effect was detected for the use of standardized preprocessed data versus custom preprocessing pipelines (48% of included teams used fMRIPrep; P = 0. 132) or for the modelling of head motion parameters (used by 73% of the teams; P = 0.
Variability of unthresholded statistical maps Analysis of the correlation between unthresholded z-statistic maps across teams showed that for each hypothesis, a large cluster of teams had statistical maps that were strongly positively correlated with one another (Fig. 2 , Extended Data Fig. 2 ). The mean Spearman correlation between all pairs of unthresholded maps (Extended Data Table  4b ) was moderate (mean correlation range 0. 18-0. 52 across hypotheses), with higher correlations within the main cluster of analysis teams (range 0. 44-0. 85 across hypotheses). An analysis of voxelwise heterogeneity across unthresholded maps (equivalent to tau-squared) demonstrated that inter-team variability was large-in many cases several times the variability expected across different datasets (Extended Data Fig.
Discussion The analysis of a single fMRI dataset by 70 independent analysis teams, all of whom used different analysis pipelines, revealed substantial variability in reported binary results, with high levels of disagreement across teams for most of the tested hypotheses. For every hypothesis, at least four different analysis pipelines could be found that were used in practice by research groups in the field and resulted in a significant outcome. Our findings highlight the fact that it is hard to estimate the reproducibility of single studies that are performed using a single analysis pipeline. Notably, analyses of the underlying statistical parametric maps on which the hypothesis tests were based revealed greater consistency than would be expected from those inferences, and significant consensus in activated regions across teams was observed using meta-analysis.
Second, public sharing of data and analysis code should become common practice, to enable others to run their own analysis with the same data or to validate the code used. These practices, combined with the use of preregistration  16  or registered reports  17  , will reduce researcher degrees of freedom but would not prevent analytical variability, as demonstrated here; however, they would ensure that the effects of variability can be assessed. All of the data and code used in the current study are publicly available with a fully reproducible execution environment for all figures and results. We believe that this can serve as an example for future studies. Foremost, we propose that complex datasets should be analysed using several analysis pipelines, and preferably by more than one research team.
Moreover, although the present investigation was limited to the analysis of a single fMRI dataset, it seems highly likely that similar variability will be present for other fields of research in which the data are high dimensional and the analysis workflows are complex and varied. The multiverse approach combined with meta-analysis is suggested as a promising solution. Notably, transparent scientific projects that involve community-wide self-assessment-such as this one-are definitive evidence of the awareness of researchers of reproducibility concerns, and the desire to assess their effect and improve practices accordingly (for additional discussion see Supplementary Discussion).
fMRI dataset To test the variability of neuroimaging results across analysis pipelines used in practice in research laboratories, we distributed a single fMRI dataset to independent analysis groups from around the world, requesting them to test nine predefined hypotheses. The full dataset is publicly available in the Brain Imaging Data Structure (BIDS)  23  on OpenNeuro (https://doi. org/10. 18112/openneuro. ds001734. v1. 0. 4) and is described in detail in a Data Descriptor  1. In brief, the fMRI dataset consisted of data from 108 participants who performed a mixed gambles task, which is often used to study decision-making under risk. In this task, participants are asked on each trial to accept or reject a presented prospect. The prospects consist of an equal 50% chance of either gaining a given amount of money or losing another, similar or different, amount of money.
Participants were divided into two groups: in the 'equal indifference' group (n = 54) the potential losses were half the size of the potential gains 10 (reflecting the 'loss aversion' phenomenon, in which people tend to be more sensitive to losses than to equal-sized gains  24  ); and in the 'equal range' group (n = 54) the potential losses and the potential gains were taken from the same scale  11, 12. The two groups were used to resolve inconsistencies of previous published results. The dataset was distributed to the teams via Globus (https://www. globus. org/). The distributed dataset included raw data of 108 participants (n = 54 for each experimental group), as well as the same data after preprocessing with fMRIPrep v. 1. 1. 4 (RRID: SCR_016216)  13. The fMRIPrep preprocessing mainly included brain extraction, spatial normalization, surface reconstruction, head motion estimation and susceptibility distortion correction.
Both the raw and the preprocessed datasets underwent quality assurance (described in detail in the Data Descriptor 1 ). MRI data collection was approved by the Helsinki committee at Sheba Tel Hashomer Medical Center and the ethics committee at Tel Aviv University, and all participants gave written informed consent (as described in the Data Descriptor of this dataset 1 ). The Board for Ethical Questions in Science at the University of Innsbruck approved the data collection in the prediction markets, and certified that the project complied with all requirements of the ethical principles and guidelines of good scientific practice. The Stanford University Institutional Review Board (IRB) determined that the analysis of the submitted team results did not meet the definition of human subject research, and thus no further IRB review was required.
Recruitment of and instructions to analysis teams We recruited analysis teams via social media, mainly Twitter and Facebook, as well as during the 2018 annual meeting of the Society for Neuroeconomics. Ninety-seven teams registered to participate in the study. Each team consisted of up to three members. To ensure independent analyses across teams, and to prevent influencing the subsequent prediction markets, all team members signed an electronic nondisclosure agreement that they would not release, publicize or discuss their results with anyone until the end of the study. All team members of 82 teams signed the nondisclosure form. They were offered co-authorship on the present publication in return for their participation. Analysis teams were provided with access to the full dataset. They were asked to freely analyse the data with their usual analysis pipeline to test the nine hypotheses and report a binary decision for each hypothesis on whether it was significantly supported on the basis of a whole-brain analysis.
Although the hypotheses were region-specific, we clearly requested a whole-brain analysis result to avoid the need of teams to create and share masks of ROIs. Each team also filled in a full report of the analysis methods used (following the guidelines of the Committee on Best Practices in Data Analysis and Sharing; COBIDAS  25  ) and created a collection on NeuroVault  15  (RRID: SCR_003806) with one unthresholded and one thresholded statistical map for each hypothesis, on which their decisions were based (teams could optionally include additional maps in their collection; see Extended Data Table  3a  for links for collections). For each result (that is, the binary decision on whether a given hypothesis was supported by the data or not), teams further reported how confident they were in this result and how similar they thought their result was to the results of the other teams (each measure was an integer between 1 (not at all) to 10 (extremely)).
These measures are presented in Extended Data Tables  1, 2. To measure the variability of results in an ecological manner, instructions to the analysis teams were minimized and the teams were asked to perform the analysis as they usually would in their own laboratory and to report the binary decision on the basis of their own criteria. Seventy of the 82 teams submitted their results and reports by the final deadline (15 March 2019; overall teams were given up to 100 days, varying based on the date they joined, to complete and report their analysis). The dataset, reports and collections were kept private until the end of the study and closure of the prediction markets. To avoid identification of the teams, each team was provided with a unique random four-character ID. Overall, 180 participants were part of NARPS analysis teams.
Factors related to analytical variability To explore the factors related to the variability in results across teams, the reports of all teams were manually annotated to create a table describing the methods used by each team. Code for all analyses of the reports and statistical maps submitted by the analysis teams is openly shared in GitHub (https://github. com/poldrack/narps). Analyses reported in this manuscript were performed using code release v. 2. 0. 3 (https://doi. org/10. 5281/zenodo. 3709273). We performed exploratory analyses of the relation between the reported hypothesis outcomes and several analytical choices and image features using mixed-effects Article logistic regression models implemented in R, with the lme4 package  26. The factors included in the model were: hypothesis number, estimated smoothness (based on the smoothest function in FSL), use of standardized preprocessing, software package, method of correction for multiple comparisons and modelling of head movement.
The teams were modelled as a random effect. One team submitted results that were not based on a whole-brain analysis as requested, and therefore their data were excluded from all analyses. Inferences using logistic regression models were confirmed using nonparametric bootstrap analysis, resampling data team-wise to maintain random effect structure. For the continuous or binary regressors (smoothness, movement modelling and use of fMRIPrep data), we computed bootstrap confidence intervals and, as an approximate hypothesis test, tested whether the confidence interval includes zero. For the factorial variables (hypothesis, software package and multiple testing method), this was not possible because there is not a single coefficient for the factor; in addition, for software package and multiple testing methods, some bootstrap samples did not contain all values of the factor.
A separate mean statistical map was then created for the teams in each cluster (see Fig. 2 , Extended Data Fig. 2 ). Drivers of map similarity were further assessed by modelling the median correlation distance of each team from the average pattern as a function of several analysis decisions (for example, smoothing, whether or not the data preprocessed with fMRIPrep were used, and so on). To assess the effect of variability in thresholding methods and anatomical definitions across teams, unthresholded z maps for each team were thresholded using a common approach. The z maps for each team were translated to P values, which were then thresholded using two approaches: a heuristic correction (known to be liberal  32  ), and a voxelwise FDR correction. Note that it was not possible to compute the commonly used familywise error correction using Gaussian random field theory because residual smoothness was not available for each team.
We then identified whether there were any suprathreshold voxels within the appropriate anatomical ROI for each hypothesis. The ROIs for the ventral striatum and amygdala were defined anatomically on the basis of the Harvard-Oxford anatomical atlas. As there is no anatomical definition for the ventromedial prefrontal cortex, we defined the region using a conjunction of anatomical regions (including all anatomical regions in the Harvard-Oxford atlas that overlap with the ventromedial portion of the prefrontal cortex) and a meta-analytical map obtained from https://neurosynth. org/ (ref. 33 ) for the search term "ventromedial prefrontal". An image-based meta-analysis was used to quantify the evidence for each hypothesis across analysis teams (Extended Data Fig. 3b ), accounting for the lack of independence due to the use of a common dataset across teams.
Prediction markets The second main goal of NARPS was to test the degree to which researchers in the field can predict results, using prediction markets  [2] [3] [4] [5] 34. We  nvited team members (researchers that were members of one of the analysis teams) and non-team members (researchers that were neither members of any of the analysis teams nor members of the NARPS research group) to participate in a prediction market  2, 35  to measure peer beliefs about the fraction of teams reporting significant whole-brain-corrected results for each of the nine hypotheses. The prediction markets were conducted 1. 5 months after all teams had submitted their analysis of the fMRI dataset. Thus, team members had information about the results of their specific team, but not about the results of any other team.
Similar to previous studies  [2] [3] [4] [5]  , participants in the prediction markets were provided with monetary endowments (100 tokens, worth US$50) and traded on the outcome of the hypotheses through a dedicated online market platform. Each hypothesis constitutes one asset in the market, with asset prices predicting the fraction of teams reporting significant whole-brain-corrected results for the corresponding ex-ante hypothesis examined by the analysis teams using the same dataset. Trading on the prediction markets was incentivized, that is, traders were paid on the basis of their performance in the markets. Recruitment. For the non-team members prediction market, we invited participants via social media (mainly Facebook and Twitter) and emails. The invitation contained a link to an online form on the NARPS website (https://www.
narps. info/) where participants could sign up using their email address. Participants for the team members prediction market were invited, after all teams submitted their results, by an email that directed them to an independent registration form (with identical form fields), to separate participants for the two prediction markets already at the time of registration. Note that team members were not aware to start with that they would be invited to participate in a separate prediction market after they had analysed the data. The decision to implement a second market, consisting of traders with partial information about the fundamental values (that is, the team members) was made after the teams obtained access to the fMRI dataset. Thus, team members were only invited to participate in the market after all teams had submitted their analysis results.
See Extended Data Table  5c  for data about trading volume on the prediction markets. Of the participants, 10. 2% did not work in academia (but hold a PhD), 34. 2% were PhD students, 43. 3% were post-doctoral researchers or assistant professors, 7. 5% were lecturers or associate professors and 4. 8% were full professors. 27. 8% of the participants were female. The average time spent in academia after obtaining the PhD was 4. 1 years. Most of the participants lived in Europe (46. 3%) and North America (46. 3%). Preregistration. All analyses of the prediction markets data reported were preregistered at https://osf. io/pqeb6/. The preregistration was ) that reported significant activations in their thresholded statistical map, for each hypothesis (labelled H1-H9), thresholded at 10% (that is, voxels with no colour were significant in fewer than 10% of teams).
+ or -refers to the direction of effect; gain or loss refers to the effect being tested; and equal indifference (EI) or equal range (ER) refers to the group being examined or compared. Hypotheses 1 and 3, as well as hypotheses 2 and 4, share the same statistical maps as they relate to the same contrast and experimental group but different regions (see Extended Data Table  1 ). Images can be viewed at https://identifiers. org/neurovault. collection:6047. reported by the analysis teams; row colours denote cluster membership. Maps are thresholded at an uncorrected value of z > 2 for visualization. Unthresholded maps for hypotheses 2 and 4 are identical (as they both relate to the same contrast and group but different regions), and the colours represent reported results for hypothesis 2. For hypotheses 1 and 3, see Fig.
L. was supported by the  Vienna Science and Technology Fund  ( WWTF VRG13-007 ) and  Austrian Science Fund  ( FWF P 32686 ); A. B. L. V. was supported by the  Vienna Science and Technology Fund  ( WWTF VRG13-007 ); L. Z. was supported by the  Vienna Science and Technology Fund  ( WWTF VRG13-007 ), the  National Natural Science Foundation of China  (no. 71801110 ),  MOE (Ministry of Education in China) Project of Humanities and Social Sciences  (no. 18YJC630268 ) and  China Postdoctoral Science Foundation  (no. 2018M633270 ); D. P. is currently supported by the  Deutsche Forschungsgemeinschaft (DFG, German Research Foundation ) under Germany's  Excellence Strategy 'Science of Intelligence'  ( EXC 2002/1 , project number  390523135 ); P. H. was supported in part by funding provided by  Brain Canada , in partnership with  Health Canada , for the  Canadian Open Neuroscience Platform initiative ; J.
-B. P. was partially funded by the  NIH (NIH-NIBIB   P41 EB019936  (ReproNim),  NIH-NIMH   R01 MH083320  (CANDIShare) and  NIH   RF1 MH120021 (NIDM) ) and the  National Institute Of Mental Health  of the  NIH  under award number  R01MH096906  (Neurosynth), as well as the  Canada First Research Excellence Fund , awarded to  McGill University for the Healthy Brains for Healthy Lives initiative  and the  Brain Canada Foundation  with support from  Health Canada ; S. B. E. was supported by the  European Union 's  Horizon 2020 Research and Innovation Programme  under grant agreement no. 785907  ( HBP SGA2 ); G. M. was supported by the  Max Planck Society ;  S. Heunis  has received funding from the  Dutch foundation LSH-TKI  (grant  LSHM16053-SGF ); J. F. G. M. was supported by a  Graduate Research Fellowship  from the  NSF  and  T32 Predoctoral Fellowship  from the  NIH ; B.
nih. gov);D. M. N. was supported by  NIH   IRP  project number  ZICMH002960 ; C. F. C. was supported by the  Tianqiao and Chrissy Center for Social and Decision Neuroscience Center Leadership Chair ; R. G. B. was supported by the  Max Planck Society ;  R. M. W. J. B. was supported by the  Max Planck Society ; M. B. , O. C. and R. G. were supported by the  Belgian Excellence of Science program  ( EOS  project  30991544 ) from the  FNRS-Belgium ; O. C. is a research associate at the  FRS-FNRS of Belgium ; A. D. L. was supported by grant  R4195  " Repimpact" of EraNET Neuron ; Q. S. was funded by grant no. 71971199 , 71602175  and  71942004  from the  National Natural Science Foundation of China  and no. 16YJC630103  of the  Ministry of Education of Humanities  and Social Science; and T. E. N.
Data availability The full fMRI dataset is publicly available on OpenNeuro (https://doi. org/10. 18112/openneuro. ds001734. v1. 0. 4) and is described in detail in a Data Descriptor  1. The results reported by all teams are presented in Extended Data Table  2. A table describing the methods used by the analysis teams is available with the analysis code. NeuroVault collections containing the submitted statistical maps are available via the links provided in Extended Data Table  3a. Source data for Figs. 1, 2 are provided with the paper. Readers may obtain access to the data and run the full analysis stream on the team submissions by following the directions at https://github. com/poldrack/narps/tree/master/ImageAnalyses. Access to the raw data requires specifying a URL for the dataset, which is: https://zenodo.
Code availability Code for all analyses of the reports and statistical maps submitted by the analysis teams is openly shared in GitHub (https://github. com/ poldrack/narps). Image-analysis code was implemented within a Docker container, with software versions pinned for reproducible execution (https://hub. docker. com/r/poldrack/narps-analysis/tags). Python code was automatically tested for quality using the flake8 static analysis tool and the codacy. com code quality assessment tool, and the results of the image-analysis workflow were validated using simulated data. The image-analysis code was independently reviewed by an expert who was not involved in writing the original code. Prediction market analyses were performed using R v. 3. 6. 1; packages were installed using the checkpoint package, which reproducibly installs all package versions as of a specified date (13 August 2019).
We processed daily ocean-color satellite observations to construct a monthly climatology of phytoplankton pigment concentrations in the Senegalo-Mauritanian region. Our proposed new method primarily consists of associating, in well-identified clusters, similar pixels in terms of oceancolor parameters and in situ pigment concentrations taken from a global ocean database. The association is carried out using a new self-organizing map (2S-SOM). Its major advantage is allowing the specificity of the optical properties of the water to be taken into account by adding specific weights to the different ocean-color parameters and the in situ measurements. In the retrieval phase, the pigment concentration of a pixel is estimated by taking the pigment concentration values associated with the 2S-SOM cluster presenting the oceancolor satellite spectral measurements that are the closest to those of the pixel under study according to some distance.