Introducing Big Team Science Initiatives (BTSIs) to Organizational Researchers
Creators: name: , Joshua.Cogswell@Nicholls.Edu
name: Cobb, Haley R.
name: Castille, Christopher Michael
name: Kreamer, Liana
Publisher: name: Open Science Framework
Year: 2023
Subjects: subject: Social and Behavioral Sciences

description: Below is a brief description and overview of this project TITLE: Would IO psychology benefit from a big team science initiative (BTSI)? A systematic review and synthesis of the BTSI literature and recommendations for IO psychology ABSTRACT: Big team science is a method involving a relatively large number of collaborators dispersed across locations, institutions, disciplines, cultures, and continents. These scientists pool resources to solve field-wide problems (Forscherer et al., 2022). Although popular across various fields of science, very little discussion has focused on the promise of big team science for applied social sciences, such as industrial-organizational psychology, and more specifically the barriers, risks, challenges, and opportunities of this method for field research. Indeed, there is no such initiative in IO psychology to our knowledge. Therefore, we systematically reviewed the big team science initiative (BTSI) literature at large, using this scholarship as a basis for determining the viability and relevance of such initiatives for IO psychology. Given that 1000s of papers are produced by BTSIs across the sciences, we focused only on summaries or critiques of these initiatives, distilling insights that are pertinent to IO psychology and related disciplines (e.g., management). We then theorize about the potential impact of BTSIs on IO psychology research and practice (e.g., consensus generation, closing the science practice gap). METHOD To make our systematic review of BTSIs both authoritative and comprehensive, we will target distilling authors’ sensemaking and sense-giving about the causal impacts of BTSIs as represented in summaries or critiques included in their manuscripts. For our purposes, sensemaking and sensegiving refer to the explicit or implicit causal connections authors make about BTSIs. For example, claiming or suggesting that BTSIs help scholars overcome replication challenges and challenges to generalizability (Forscher et al., 2020) describes a causal claim linking BTSIs to overcoming challenges associated with replicability and generalizability, or BTSI → overcoming (i) replication challenges and (ii) challenges to generalizability. In other words, articles containing explicit causal connections linking BTSIs to outcomes that matter for a scientific discipline (e.g., replicability, generalizability) as suggested by their title, abstract, primary, and/or secondary headings will be included in our systematic review. For those articles surviving initial screening (those that explicitly describe the impacts of BTSIs), a more in-depth analysis using grounded theory will be used to draw implied connections linking BTSIs to outcomes. For instance, suppose an author claims that BTSIs help scholars adopt open science practices (e.g., Moshontz et al., 2019), or in other words BTSIs → promoting open science practices in a discipline. How do BTSIs have this impact? Well, they may do so by helping scholars enact distinct individual practices that are of modest importance in isolation (e.g., writing reproducible analytical code, pre-registering analyses) but, when enacted collectively in a coordinated fashion, help scholars learn new behaviors that bleed into their professional practice. Such connections are ones that we the authors will make using grounded theory (Glaser &amp; Strauss, 1968). Search Strategy. Using the Discovery Service provided by EBSCO, we used the following search terms: "big team science" OR TX "big-team science" OR ("team science" AND "open science" ) OR TX ("crowdsourcing" AND "collaboration" AND "teams" AND "metascience"). We also used Google Scholar and searched for “big team science”. Overall, our search strategy yielded 321 articles that we are currently subjecting to inclusion/exclusion criteria. Inclusion/Exclusion Criteria. We have one criterion for inclusion in our review: sources that include a summary, evaluation, or critique of at least one BTSI or multiple BTSIs or the idea of big team science itself (e.g., big team science, crowdsourcing, meta-science). Our specific measure is whether a title, abstract, primary heading, or secondary heading contains our key search terms. In other words, the decision to include a source is determined by the existence of explicit (and we hope substantive) commentary on BTSIs, regardless of the form of the source (e.g., primary study, review, commentary). Regarding exclusion criteria, we have two: (1) unpublished sources (e.g., pre-prints) and (2) studies (primary or otherwise) that make cursory mention of BTSIs in only one or two sentences. For instance, if an article only notes that BTSIs should be used more frequently, it was excluded. However, we did count the presence of such cases (as well as positive, neutral or negative mentions). Consistent with best practices, to ensure the trustworthiness and dependability of our findings for our systematic review, we will examine the inter-rater reliability of two trained independent coders using Cohen’s kappa. The threshold of .75 will be used to indicate excellent reliability. Critiquing the BTSI Literature from an IO Perspective. In synthesizing the BTSI literature for IO psychology, we leveraged articles that met our inclusion criterion (e.g., summaries and critiques of BTSI initiatives). As our goal is to distill insights on executing BTSIs and translate this knowledge for application in IO psychology (particularly field settings), we supplemented our systematic review with a cursory review of the literature on field-wide limitations (e.g., most research is cross-sectional and limited by method bias; see Brutus et al., 2010), incompatibilities between open science practices and IO psychology (e.g., Guzzo et al., 2022), and the practical relevance of IO psychology research (e.g., most research in our journals does not reflect topic areas that are important for practitioners; see Rynes, 2012). We then used the insights from these articles to frame our synthesis of the BTSI literature for IO psychology. Our aim was to identify areas where academics and practitioners possessed complementary skillsets for collaborating via a BTSI, coordinating the thoughtful application of open science practices (OSPs) via a BTSI (e.g., born open data may not be realistic), and meaningfully addressing concerns held by all parties. Extracting and Synthesizing the Literature. Our main interest in this literature review was in distilling authors’ sensemaking and sense-giving about the causal impacts of BTSIs, which we contextualize for IO psychology. For instance, Forscher et al. (2020) – a frequently cited paper in this area – points out that BTSIs allow scholars to pool resources, thereby overcoming five key challenges in psychology: (i) replicability, (ii) generalizability, (ii) strategy selection, (iv) inferential reproducibility, and (v) computation reproducibility. Addressing any single challenge places a cost on different stakeholders (e.g., researchers, participants, and platform maintainers). They suggest that BTSIs are well suited to addressing all of these challenges. To summarize this sense-giving with a causal diagram, we suggest that BTSIs → Overcoming Challenges in Psychology Research. To contextualize this causal diagram for IO psychology, we conducted a more cursory review of the literature and (i) presented evidence that these challenges are present in IO psychology and/or (ii) identified narrow relevant domains of study within IO psychology that struggle to overcome this challenge. For example, in a review of self-reported limitations in IO psychology, Brutus et al. (2010) note external validity/generalizability as one of the most commonly noted limitations in top-tier IO psychology research. More recently, the SCORE initiative supported by DARPA estimated that applied psychology published in a top-tier journal (i.e., Journal of Applied Psychology) had an estimated replicability of ~50% (Youyou et al., 2023). As such, we contextualize this causal diagram (BTSIs → Challenges in Psychology Research) by providing evidence that these challenges (particularly replicability and generalizability) are salient in IO psychology specifically. To capture this sensemaking perspective, we adopted a grounded theory approach. Grounded theory is concerned with discovering, rather than testing, theory and aims to explore phenomena when there is little empirical or, rather, quantitative evidence (Glaser &amp; Strauss, 1968), such as the use of BTSIs in IO psychology. It is also often referred to as the “constant comparative” method where, throughout data analysis, themes and categories emerge and are constantly compared to one another as well as back to the extant literature on the topic (Strauss &amp; Corbin, 1990). First, during the data extraction phase, the team of coders will decide which text is relevant for extracting, extract the content, and then openly code that content. The content will be viewed line-by-line and labels will be used to represent the phenomenon (Cowan &amp; Fox, 2015). Next, during axial coding, the team of coders will group together the codes developed during open coding to identify larger concepts. This will involve making connections between a larger concept and narrower concepts (Strauss &amp; Cobin, 1990). We will ask questions such as: “How are these codes different”, “How are these codes the same to one another?”, and “If they are different, how so?” (Cowan &amp; Fox, 2015). During this phase, the team will also discuss areas of disagreement and discrepancy among our emergent themes. Axial coding will be completed when the research team is satisfied with the summary themes. Lastly, we will engage in selective coding to establish causal connections between themes and build the model or theory through analysis. The summary model will be distributed to key stakeholders including (i) leaders from BTSI initiatives and then to (ii) both IO psychology practitioners and academicians. Distribution to BTSI Leaders. Regarding distribution to BTSI leaders, these individuals will be interviewed and asked to comment on our summary model (does it adequately describe BTSIs from their perspective of the merits/drawbacks of BTSIs they have experience with?). Additionally, as the BTSI literature paints a quite positive picture of BTSIs, we will also ask leaders to share details of BTSIs that are either unsuccessful (a ‘failed start’), or successful but are no longer very actively used among either academicians or practitioners (e.g., the Profit Impact of Market Strategy), or simply not yet successful. Regarding this latter category, the Advancement of Replications Initiative in Management (ARIM) encourages replication studies among doctoral students in strategic management. However, at the time of our writing, this initiative has not yet produced a publication. Any insights will be used to refine our model, which will be summarized and presented to IO academicians and practitioners. Distribution to IO Academicians and Practitioners. As BTSIs require the coordinated use of multiple open science practices (e.g., data-sharing arrangements, commitment to theory testing that is relevant across organizations, and use of unconventional dissemination practices such as registered reports), a BTSI in IO psychology would seem practically impossible (e.g., Guzzo et al., 2022). For the viability of a BTSI in IO psychology to be properly tested, the perspective of academicians and practitioners in IO psychology must be brought into our analysis. While such parties often operate in distinct professional networks (Rynes et al., 2012), there may be research areas that represent overlapping interests (e.g., compensation policies, see Banks et al., 2016). Regarding distribution to IO psychology academicians and practitioners, we will send a survey containing a summary of our results. We will ask both parties to comment on our proposal (e.g., does it address field-wide limitations, and overcome incompatibilities in applying open science to our field). A final set of focus groups will be conducted with representatives of practitioners and academics to validate and stabilize our claims about the merits of a BTSI for IO psychology. This process, therefore, ensures the trustworthiness of our findings. Key References Banks, G. C., Pollack, J. M., Bochantin, J. E., Kirkman, B. L., Whelpley, C. E., &amp; O’Boyle, E. H. (2016). Management’s Science–Practice Gap: A Grand Challenge for All Stakeholders. Academy of Management Journal, 59(6), 2205–2231. https://doi.org/10.5465/amj.2015.0728 Brutus, S., Gill, H., &amp; Duniewicz, K. (2010). State of the science in industrial and organizational psychology: A review of self-reported limitations. Personnel Psychology, 63(4), 907–936. https://doi.org/10.1111/j.1744-6570.2010.01192.x Cowan, R. L., &amp; Fox, S. 2015. Being pushed and pulled: A model of US HR professionals’ roles in bullying situations. Personnel Review, 44: 119–139. Forscher, P. S., Wagenmakers, E.-J., Coles, N. A., Silan, M. A., Dutra, N., Basnight-Brown, D., &amp; IJzerman, H. (2022). The Benefits, Barriers, and Risks of Big-Team Science. Perspectives on Psychological Science : A Journal of the Association for Psychological Science, 17456916221082970. MEDLINE. https://doi.org/10.1177/17456916221082970 Glaser, B. G., &amp; Strauss, A. L. (1968). The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company. Guzzo, R., Schneider, B., &amp; Nalbantian, H. (2022). Open Science, Closed Doors: The Perils and Potential of Open Science for Research in Practice. Industrial and Organizational Psychology: Perspectives on Science and Practice. Moshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., Grahe, J. E., McCarthy, R. J., Musser, E. D., Antfolk, J., Castille, C. M., Evans, T. R., Fiedler, S., Flake, J. K., Forero, D. A., Janssen, S. M. J., Keene, J. R., Protzko, J., Aczel, B., … Chartier, C. R. (2018). The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network. Advances in Methods and Practices in Psychological Science, 1(4), 501–515. https://doi.org/10.1177/2515245918797607 Rynes-Weller, S. L. (2012). The research-practice gap in I/O psychology and related fields: Challenges and potential solutions. In S. W. J. Kozlowski (Ed.), The Oxford Handbook of Organizational Psychology (pp. 409–452). Strauss, A., &amp; Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage Publications, Inc. Youyou, W., Yang, Y., &amp; Uzzi, B. (2023). A discipline-wide investigation of the replicability of Psychology papers over the past two decades. Proceedings of the National Academy of Sciences, 120(6), e2208863120. https://doi.org/10.1073/pnas.2208863120