dataesr/scientific-paragraphs-categorization huggingface dataset:
This suggests that defensible, but subjective, analytic choices can lead to highly variable quantitative effect size estimates. The disturbing implication is that if only one team had obtained the dataset and presented their preferred analysis, the scientific conclusion drawn could have been anything from major racial disparities in red cards to equal outcomes. Subsequent crowd initiatives have likewise revealed divergent results across independent scientific teams  (Bastiaansen, Kunkels, & Blaauw, 2020; Botvinik-Nezer et al. , 2020). Relying on fMRI data from 108 research participants who performed a version of a decision-making task involving risk,  Botvinik-Nezer et al. (2020)  recruited 70 research teams to test nine hypotheses (e. g. , "Positive parametric effect of gains in the vmPFC").
Analysts were asked whether each hypothesis was supported overall (yes/no) in their analysis of the dataset. No two teams used the same approach, and only 1 of 9 hypotheses received support (i. e. , a "yes" response) across the large majority of teams (Hypothesis 5, with 84. 3% support). Three hypotheses were associated with nearlyuniform null results across analysts (94. 3% non-significant findings), while for the remaining five hypotheses between 21. 4% and 37. 1% of teams reported statistically significant support. At the same time, metaanalysis revealed significant convergence across analysis teams in terms of the activated brain regions they each identified. In another recent crowd project,  Bastiaansen et al. (2020)  recruited 12 analysis teams with expertise in event sampling methods to analyze individual time-series data from a single clinical patient for the purposes of identifying treatment targets.
The experience sampling dataset used in  Bastiaansen et al. (2020)  was based on a set of standardized questionnaire items, with variability in results attributable to data preprocessing, statistical techniques, and software packages. Although different analysts clustered items differently, they did not employ fundamentally different approaches to conceptualizing and measuring variables like depression and anxiety. In contrast, in the present initiative crowdsourcing the analysis of a complex dataset on gender and professional status in group meetings, conceptualization and operationalization of key variables (e. g. , social status) was left unconstrained and up to individual researchers. This approach is arguably closer to the ambiguity researchers typically confront when approaching a complicated dataset, and may lead to even greater heterogeneity of methods and results than seen previously.
The dataset for this project included over three million words and thousands of pieces of dialogue from an invitation-only online forum for scientific debates (see Supplement 1 for a detailed overview and htt ps://osf. io/u9zs7/ for the dataset). Consider the simple and straightforward hypothesis that high status scientists tend to speak more during such group meetings. An analyst might choose to operationalize professional status using dataset variables such as citation counts, h-index, i10-index, job title, rankings of current university, rankings of doctoral institution, years since PhD, or some combination of the above. She might also decide to focus on professional status within a field, subfield, or among participants in an individual conversation, and use this to predict how actively the person participated in the meeting.
Likewise, verbosity might be operationalized in different ways, among these number of words contributed, or number of comments made. The overall project featured a pilot phase to generate and select hypotheses, and also carry out initial analyses testing these hypotheses (see Supplements 2 and 3 for detailed reports). To help generate and evaluate ideas, a crowd of scientists recruited online were provided with an overview of the dataset (variables and data structure) and asked to propose research hypotheses that might be tested with it. The crowd then voted on which ideas should be selected for systematic testing (Supplement 2). Subsequently, a small number of research teams (a subset of this crowd) used the dataset to test the final set of eleven hypotheses. As reported in Supplement 3, the quantitative results of these pilot analyses proved remarkably dispersed across teams, with little convergence in outcomes for any of the scientific predictions.
The primary study reported in the present manuscript reduced the number of hypotheses from eleven to two characterized by positive evaluations in the selection survey (Supplement 2) and divergent results in the pilot analyses (Supplement 3). We focused on two hypotheses from the pilot with especially dispersed outcomes across analysts in order to pursue our goal of understanding the sources of such variability. To this end, we asked analysts to use an online platform we developed called DataExplained to articulate the reasoning underlying each of their analytic decisions as they made them (further details on how the platform works are provided in the Methods section, in  Feldman, 2018, Staub, 2017, and in Supplement 9). The stated reasons were then subjected to a qualitative analysis based on the General Inductive Approach  (Thomas, 2006).
DataExplained offers a novel form of scientific transparency, in that it documents analytic paths being taken and not taken in real time and provides this output in addition to the traditional research analytic outputs. Both of the research ideas selected for crowdsourced testing were previously explored in the managerial and psychological literatures on gender, status, and group dynamics  (Brescoll, 2011; Inzlicht & Ben-Zeev, 2000; Schmid Mast, 2001 , 2002; Spencer, Logel, & Davies, 2016). Hypothesis 1 posits that "A woman's tendency to participate actively in a conversation correlates positively with the number of females in the discussion. " Hypothesis 2 predicts that "Higher status participants are more verbose than are lower status participants. " Our project examined whether independent analysts would arrive at similar analyses and statistical results using the same dataset to address these questions.
Analysts had to choose the dataset variables they would use to capture the independent and dependent variables (e. g. , whether to measure status with academic citations or job rank), determine their unit of analysis (e. g. , commentators vs. conversations), decide what covariates to include, and which type of regression or other measure of association to use. In the Boba multiverse, we crossed as many choice as possible and was reasonable, and examined the implications for the final estimates for both Hypotheses 1 and 2.
Dataset The dataset included 3,856,202 words of text in 7,975 comments from the online academic forum Edge  (Lazer et al. , 2009). As described by Edge's founders, its purpose is: "To arrive at the edge of the world's knowledge, seek out the most complex and sophisticated minds, put them in a room together, and have them ask each other the questions they are asking themselves" (http://edge. org). The group discussions spanned almost two decades  (1996) (1997) (1998) (1999) (2000) (2001) (2002) (2003) (2004) (2005) (2006) (2007) (2008) (2009) (2010) (2011) (2012) (2013) (2014)  and included 728 contributors, 128 of them female. The dataset contained 150 variables related to the conversation, its contributors, or the textual level of the transcript (Supplement 1). New attributes not provided on the website were manually collected by browsing CVs, university or personal web-pages, Google Scholar pages, and professional networking websites, and added to the dataset.
An anonymized version of the dataset for the project is available at: https://osf. io/u9zs7/. The dataset is structured as follows: each row in the dataset presents one comment made by one contributor to one conversation. Each row contained variables for comment id, conversation id, and contributor id. Each comment contributed to only one conversation. A comment consisted of at least one character, and most comments consisted of several words and sentences. A new comment was created when a contributor wrote at least one character that was submitted to the forum. A conversation started when a contributor wrote a new comment that did not respond to a previous comment. Conversations consisted of two or more comments that were posted sequentially by at least one contributor. A contributor was one person who posted at least one comment to one or more conversations.
Recruitment and initial survey of analysts Data analysts were recruited via open calls on social media platforms including Twitter, Facebook, forums of psychology interest groups, and R (R Core Team, 2018) mailing lists (see Supplement 4 for the project advertisements). In total, 49 scholars submitted analyses for this crowdsourcing initiative, of which 23 scholars completed 37 sufficiently detailed analysis reports (one report per hypothesis) and provided reproducible code suitable for inclusion. Notably, difficulties in reproducing analyses from the reported statistics  (Bergh, Sharp, Aguinis, & Li, 2017) , as well as the original data and code are common  (Chang & Li, in press; Hardwicke et al. , 2018; McCullough, McGeary, & Harrison, 2006; Stockemer, Koehler, & Lentz, 2018; Stodden, Seiler, & Ma, 2018) , even under the most favorable of circumstances as with pre-registered reports  (Obels, Lakens, Coles, Gottfried, & Green, in press).
Eight of the remaining analyses, from six analysts, were flagged by sub-teams of research assistants and independent statisticians as containing errors. See below and Supplement 7 and 8 for further details on the error and reproducibility checks, and the results of the excluded analyses. The overall rate of problems identified is not surprising since scientific errors are quite common  (Bakker & Wicherts, 2011; Bergh et al. , 2017; Rohrer et al. , in press ). The exclusions for errors left a total of 29 analyses, N = 14 for Hypothesis 1 and N = 15 for Hypothesis 2, which were conducted by 19 analysts, as the focus of this primary project report. The quantitative analyses below focus on these 29 results from 19 analysts. Prior to receiving the dataset, analysts completed a pre-survey of their disciplinary background and expertise, and a set of demographic measures (see Supplement 5 for the complete pre-survey items and htt ps://osf.
io/y9fq4/ for the data). At the time of the project, participating analysts were on average 31. 2 years of age (SD = 7. 2), and included 15 men and 4 women. Seven resided in the United States, five in European countries, and the rest in Australia, Brazil, New Zealand, Pakistan, Russia, Singapore, and South Korea. Three were professors, one was a post-doctoral researcher, six were doctoral students, four held another academic position (e. g. , data analyst), and five were not affiliated with an academic institution. The participating analysts selfreported an average of 6. 5 years of experience in data analysis (SD = 5. 5). A substantial minority indicated that they performed data analysis on a daily basis (7 analysts, 37%), while the rest performed data analysis a few times a week (3 analysts, 16%), once a week (4 analysts, 21%), once every two weeks (1 analyst, 5%), or less (4 analysts, 21%).
Analyses using the DataExplained platform We designed an online platform called DataExplained that supports transparent data analysis reporting in real time. The platform records all executed source code and prompts analysts to comment on their code and analytical thinking steps. DataExplained is based on RStudio Server (https://www. rstudio. com/products/rstudio-server/), a data analysis platform that allows users to conduct analyses remotely via a web browser based on the familiar RStudio interface. In addition to the online RStudio environment, we implemented features that enabled us to track all executed commands along with the analysts' detailed explanations for every step of the executed analysis. The procedure was as follows. First, the participants were provided access to the platform, where they executed their data analysis using the RStudio user web-interface.
During their analysis, every executed command (i. e. , log) was recorded. Recording all executed commands (including commands executed but not necessarily found in the final code) is useful, as such logs might reveal information that affected the analysts' decisions but are not reflected in the final script. Whenever the participants believed that a series of logs could be described as a selfexplanatory block, or when a certain number of logs was produced, they were asked to describe their rationales and thoughts about the underlying code. The dataset was available in the environment of DataExplained only. Use of this platform essentially involves conducting analyses in R with added transparency features. We included a number of elements to capture the workflow of analysts. In particular, once the analysts reached a certain number of executed commands, we prompted them to explain the goals and reasoning underlying the relevant code, as well as alternative approaches they rejected.
This way, each block was annotated with descriptive properties which reflect the rationales and reasoning of the analyst's actions within a block. Analysts were able to navigate through their analysis history, by restoring the state of the RStudio workspace at any given point a block was created. These features helped the analysts to recall the considerations during their analysis, even if the corresponding portion of code was no longer in the final script. Finally, DataExplained provided analysts with an overview of all blocks that they created and asked them to graphically model the workflow representing the evolution of the analysis. Initially, each analyst was presented with a straight chain of blocks, ordered by their execution. The analysts were then asked to restructure the workflow such that it better reflected their actual process.
Independent assessment of analysis quality Finally, two teams of research assistants and statisticians carefully reviewed each analyst's approach for errors and ensured they could independently reproduce the results (see Supplements 7 and 8 and htt ps://osf. io/n5q3c/). These error-checks involved a two-step process. First, three research assistants from The European School of Management and Technology (ESMT) conducted an initial review and error check. These three RAs were graduate students in computational neuroscience, public policy, and economics and were selected for their strong data analysis backgrounds. They had advanced knowledge of statistics and econometrics and were skilled in R, Python, Matlab, and Stata. Two of the ESMT research assistants coded each analysis for potential errors, and if they found any discussed this with each other to clarify whether they agreed on an analytical choice being an error or not.
Operationalization of number of women in discussion Effect size Notes. This table includes analyses not flagged as having clear errors by independent reviewers. This table includes the original effect sizes reported by the analysts, which are not directly comparable to one another. * In the online article, the column includes hyperlinks for each analyst's error checks and raw code 1 Random intercept for conversation ID; random intercept and slope for contributor ID 2 Random intercept for conversation ID postdoctoral researcher, and professor) reviewed these error checks and individual analyses, again examining whether the code by each analyst contained any serious errors. The error check documents are publicly posted at https://osf. io/n5q3c/. In the end the ESMT and Tilburg subteams converged on a subset of analyses that were deemed as containing errors.
Variability in analytic approaches and conclusions We set out to identify the extent of heterogeneity in researchers' choices of analytic methods, and the impact of this heterogeneity on the conclusions drawn about research questions regarding gender and professional status in group meetings. We found that the participating analysts employed a wide array of statistical techniques, covariates, and operationalizations of key theoretical variables such as professional status and verbosity (see https://osf. io/n5q3c/ for the code for each individual analyst). As summarized in Tables 1. 1-1. 3, different analysts operationalized variables in various ways: for example, Analysts 3, 10, and 17 operationalized verbosity as the number of words contributed in a comment, Analyst 5 operationalized verbosity as the number of conversations participated in, and Analysts 1, 7, and 14 operationalized verbosity as the number of characters in comments, among other approaches.
This method transforms individual p-values of test statistics to zscores, assuming that the sampling distribution of the test statistic is approximately normally distributed, resulting in random variables with a variance of 1. It is crucial to realize that the analysts' z-statistics are a function of the effect size, the number of independent observations in the analysis, as well as the selected statistical technique and their statistical properties (e. g. , statistical power, in case of a true nonzero effect). As the three aforementioned factors are all affected by the analysts' selected analysis, and all analysts use the same dataset, differences in z-scores still reflect differences in the consequences of analysts' choices. Regarding the normality assumption of the z-scores, note that most parameters in models correspond to linear combinations of the data.
For example, Analyst 1 for H1 drew on a sample of 5,443 observations. The sample sizes for all other analyses are reported in these tables. As most statistics are well approximated by a normal distribution for the number of observations considered by the analysts, we believe that the normal approximation works rather well in this application. The z-scores of individual results were obtained using different methods. In some cases the z-scores could be directly retrieved from the output of the analyst, but in the majority of the cases z-scores were Notes. This table includes analyses not flagged as having clear errors by independent reviewers. This table includes the original effect sizes reported by the analysts, which are not directly comparable to one another. * In the online article, the column includes hyperlinks for each analyst's error checks and raw code 1 Random intercept for contributor ID; random intercept and slope for conversation ID 2 Random intercepts for conversation ID and contributor ID 3 Random intercept for whether the conversation was written / transcribed computed using the p-value of the test statistic (using the quantile normal distribution in R).
In one case where a p-value was not presented by the analyst we ran our own code in R to retrieve it (i. e. , cor. test(data2 $TendencyToParticipate, data2$UniqueFemaleContributors, method-="kendall"). Sometimes a large t-value was provided in combination with its df and a p-value < 0. 001. In those cases, the exact p-value was first calculated using R, and then transformed to a z-score (e. g. , t(100) = 10 is transformed to z = 8. 306 by qnorm(pt(10,100, lower. tail = FALSE), lower. tail = FALSE)). As t-values could be very large or p-values very small, we sometimes had to use the log. p argument to obtain zvalues (e. g. , t(7000) = 100 results in -3,110. 64 using pt(100,7000, lower. tail = FALSE,log. p = TRUE), which yields z = 78. 81 using qnorm (-3,110. 64,lower. tail = FALSE,log. p = TRUE).
Weick, Sutcliffe, and Obstfeld (2005)  define sensemaking as "the ongoing retrospective development of plausible images that rationalize what people are doing" (p. 409). Through DataExplained, we are able to observe the roadmap of different analytical alternatives and justifications for decisions in much greater detail than ever before. To better understand the sensemaking process underlying these analytic decisions, we relied on a qualitative research approach. A project sub-team of qualitative researchers analyzed the descriptive text explaining in detail every step undertaken by individual analysts throughout their data analyses as well as the source-code corresponding to each step. By asking analysts to explain their decisions and considered alternatives to the executed code, we obtained a rich dataset capturing their various workflows.
This is especially useful due to the exploratory element of data analysis, where researchers often experiment with data prior to deciding on how to proceed. Indeed, graphic representations of the analysts' R-codes show that the analyses were often iterative, seemingly lacked a clear direction at times and instead included several explorative loops which help analysts make sense of the data over time. The relatively unstructured nature of the R-codes provided did not facilitate quantitative numeric or quantitative text analyses. Instead we decided to use the General Inductive Approach  (Thomas, 2006)  because this allowed us to analyze the R-code from the bottom up, subjecting each line of code to an iterative, qualitative analysis. This qualitative approach helped us understand how analysts made sense of the data and the factors guiding their decision-making processes.
After the coders refined a set of codes, they developed an initial description of the meaning of each code along with a memoa short description explaining the code and elaborating on when it should be applied. Eventually, the codes from different coders were merged and discussed as a group. All codes as well as their memos were aggregated together into a code book, provided in Supplement 9 (see also  Feldman, 2018, and Staub, 2017). The coders then iteratively kept refining and reevaluating the codebook until the process reached a well-established and shared understanding of all the codes (see Figure  5 ). A detailed report of this bottom-up qualitative analyses of the annotated code from DataExplained is provided in  Feldman (2018) ,  Staub (2017) , and in Supplement 9. Our analytical approach was bottom-up in that we qualitatively analyzed individual blocks of code.
Specifically, we closely read the analysts' blocks of code, as well as their responses to open questions about their analytical choices such as: "Why did you choose your option?". Following the General Inductive Approach  (Thomas, 2006)  we identified meaningful units in these responses and assigned different labels to these meaningful units. For example, if an Fig. 3. Dispersion of z-scores corresponding to estimates of independent analysts using the same dataset to test Hypothesis 1 ("A woman's tendency to participate actively in the conversation correlates positively with the number of females in the discussion"), together with some details on each specification. Note that there is a break in the y-axis of the figure to incorporate the extreme z-score of Analyst 4. analyst responded "I experimented with both, but will ultimately use the non-transformed data for reporting; diagnostic plots did not improve much with transformations, and interpretability was reduced", we assigned the label "exploratory" to this response.
) using the final set of 31 codes grouped into 10 categories and 4 metacategories. These codings led to a proposed model of the data analyst's reasoning process and workflow (Figure  6 ). The model seeks to capture the iterative interplay between understandings of the dataset and hypotheses to be tested, the analyst's knowledge and beliefs, the actions and methods actually performed during the analysis, and insights gained. As researchers conduct data analyses, they obtain intermediate results. These results are almost always interpretative in their nature and often stem from personal understanding and beliefs, which often vary across individuals. Data analysis is an iterative process, and intermediate output plays a key role in deciding which path to further follow. The data by itself can influence an analyst's beliefs, which as a consequence Fig.