# Best Practices of Interdisciplinary Communication in Complex Systems Research

Camille Masselot, Liubov Tupikina, Marc Santolini

<figure><img src="image_1.png" /></figure>

## To cite this version:

Camille Masselot, Liubov Tupikina, Marc Santolini. Best Practices of Interdisciplinary Communication in Complex Systems Research. German Ministry for Science and Education. 2022. [hal-04166081](https://hal.science/hal-04166081)

HAL Id: hal-04166081

https://hal.science/hal-04166081

Submitted on 19 Jul 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Best Practices of Interdisciplinary Communication in Complex Systems Research

Report as part of the BMBF projekt „Krisenkomplex“

Camille Masselot¹,², Liubov Tupikina², Marc Santolini¹,²,³

1. Just One Giant Lab

2. Learning Planet Institute, F-75004 Paris, France

3. Université Paris Cité, Inserm, System Engineering and Evolution Dynamics, F-75004 Paris, France

## Table of contents

1. Preliminaries 0

1.1. Research Task 1

1.2. Prerequisites 2

1.2.1. Background and rationale 2

1.2.2. Objectives 4

1.2.3. Assumptions 4

2. Methods and Implementation 4

2.1. Study design and methods 4

2.2. State of the Art 5

3. Results 6

3.1. The OpenCovid19 context 6

The OpenCovid19 community 6

Structuration of OpenCovid19 6

Communication within OpenCovid19 13

Mode of co-production within OpenCovid19 14

3.2. OpenCovid19 workflow 15

3.3. Emergent coordination 15

3.3.1. Facilitate the onboarding and orientation of participants—participative architecture 15

Acknowledge the motivation of future community members and clearly communicate the core value and vision of your initiative 15

Facilitate triaging and navigation of your initiative 17

Ease your onboarding process 17

3.3.2. Continuously embedding and improving predictability for participants 18

Structure your community around one or multiple clear and reliable core goals 18

Be transparent on how your community is structured and governed, and on how decisions are made 18
<table><tr><td>Find the balance between agility and rigidity in your initiative's structure</td><td>18</td></tr><tr><td>Systematize your communication process</td><td>19</td></tr><tr><td>Adapt your structure and process to the rhythm of the activity of your projects</td><td>19</td></tr><tr><td>3.3.3. Collective building of common understanding</td><td>19</td></tr><tr><td>Choose an inclusive communication tools, and strategy</td><td>19</td></tr><tr><td>Use participatory approaches to empower your community members</td><td>20</td></tr><tr><td>3.3.4. Gradual self-accountability</td><td>21</td></tr><tr><td>Anticipate risks, offer mediation and mitigation of conflicts</td><td>21</td></tr><tr><td>Identify, document and share how decisions are taken and by whom</td><td>21</td></tr><tr><td>3.3.5. Continuous experimentation</td><td>22</td></tr><tr><td>Get to know your community and its needs</td><td>22</td></tr><tr><td>Design and implement a continuous monitoring and evaluation strategy</td><td>22</td></tr><tr><td>Design a feedback and moderation mechanism</td><td>23</td></tr><tr><td>3.4. Open knowledge exchange</td><td>23</td></tr><tr><td>3.4.1. Multidisciplinary knowledge integration</td><td>23</td></tr><tr><td>Invest in interpersonal connections and create bonds</td><td>23</td></tr><tr><td>3.4.2. Engaging in exchanging efforts and not committing directly to the entire project</td><td>23</td></tr><tr><td>3.4.3. Extreme transparency in knowledge produced and knowledge capitalization</td><td>24</td></tr><tr><td>3.4.4. Facilitated and open translation/ appropriation of the results</td><td>24</td></tr><tr><td>II.Usability and Application</td><td>24</td></tr><tr><td><strong>References</strong></td><td><strong>24</strong></td></tr><tr><td><strong>Supplementaries</strong></td><td><strong>31</strong></td></tr></table>
# 1. Preliminaries

This document contains the description of research and requirements of the research task "Best Practices of Interdisciplinary Communication in Complex Systems Research". The research project is part of the "Krisenkomplex" project funded by the German Ministry for Science and Education (BMBF). In this project, the TU Dresden and the Chair for Network Dynamics of Prof. Timme aim to contribute to a sustainable crisis management of socio-economic and ecological challenges by advancing complex systems research toward application-oriented basic research. The simultaneous and opposing impact of publicly accessible mobility on ecological sustainability and spreading of and resilience against pandemics serves as an example of these challenges in the project.

The research task is part of a set of complementary projects to research and collect best practices and workflows from international experts in complex systems research and related topics. The goal is the design of a conceptual rapid-response workflow that supports fast and flexible reaction to crises with complex systems modeling and research. The individual tasks focus on data management, documentation, communication and collaboration, risk analysis, as well as agile processes.

The research task supports this goal by contributing a better understanding of the requirements and potential solutions to organizing and fostering communication in complex systems research and outreach and explanation to non-scientific partners. Firstly, the research task aims to understand best practices and workflows for communications in scientific project teams and beyond specific project boundaries as well as tools and events to improve interdisciplinary exchange. Secondly, the research task aims to understand best practices and methods to explain and transfer methods, results, and insights of complex systems research to non-scientific partners, in particular political and societal decision-makers.

## 1.1. Research Task

In this report, we provide an account on the management and coordination practices that have been implemented within the context of the Just One Giant Lab interdisciplinary, large-scale open research programmes. We used a mixed method approach that tackles the research tasks described in the table below.

<table><thead><tr><td>Work package</td><td>Description</td><td>Fraction workload</td></tr></thead><tbody><tr><td>WP10</td><td>Analysis of the requirements for a successful communication structure in interdisciplinary research teams<ul><li>Definition and description of the requirements for efficient scientific exchange in interdisciplinary project groups</li><li>Definition and description of the requirements for efficient scientific exchange in interdisciplinary project groups</li></ul></td><td>1/6</td></tr></tbody></table>
<table><tr><td></td><td>exchange in interdisciplinary complex systems research beyond specific project across perspectives and methods used by different actors<br/>- Description of typical challenges with respect to the requirements of scientific exchange in interdisciplinary complex systems research</td><td></td></tr><tr><td>WP11</td><td>Compilation and prototypical description of suitable communication structures and strategies<br/>- Collection and discussion of tools and strategies to enable and improve scientific communication in large interdisciplinary project-teams based on the requirements developed in WP10<br/>- Discussion of strategies to foster participation and enhance scientific exchange across boundaries of individual disciplines and perspectives</td><td>1/3</td></tr><tr><td>WP20</td><td>Analysis of the requirements for a successful communication of complex systems research to non-scientific partners</td><td>1/6</td></tr><tr><td>WP21</td><td>Compilation and prototypical description of suitable communication structures and strategies</td><td>1/3</td></tr></table>

## 1.2. Prerequisites

### 1.2.1. Background and rationale

As the world faces a rise in the number of complex challenges that threaten the resilience of our economic, environmental and social systems, we observe a shift towards more collaboration and openness in the way science and innovation is performed (Burgelman et al., 2019; Elliott & Resnik, 2019; Nosek et al., 2015), bringing closer governments, civil society, and the private sector. Examples of this include the efforts made to accelerate society's progress towards Sustainable Development Goals (SDG) (Fraisl et al., 2020) and the fight against pandemics such as COVID-19 (Auffray et al., 2020). In response to such complex, multi-scale problems, the practice of science is changing. Research demonstrates that the intensity and the diversity of collaboration positively affect the quality (Liao, 2011) and productivity (Juan et al., 2012) of research while positively impacting the knowledge integration from participants (Polk, 2015). Likewise, participant transdisciplinarity (Leavy, 2011) seems critical to generating innovative outcomes (Stokols, 2006) and dealing with complex real world problems (Klein, 2008).

Such mechanisms are often at play in the field of citizen science (also often referred to as “crowd science” or “networked science”), promising to transform the knowledge generation landscape by tapping into networks of non-academic citizens (Franzoni & Sauermann, 2012; Nielsen, 2014) in a new social contract for this kind of research (Vayena et al., 2016). Accordingly, multiple approaches to create and develop knowledge and solutions to complex
challenges have slowly entered the mainstream. Hackathons, challenge-based approaches, and the participation of citizens in science have been flourishing over the last two decades (Bosman et al., 2017), especially within the natural sciences (Hecker et al., 2018) and more recently, medical sciences, public health and population-health research (Katapally, 2019; Rowbotham et al., 2019). (Angelidis et al., 2016; Kienzler & Fontanesi, 2017; Ramatowski et al., 2017). Such initiatives are not without pitfalls: they suffer, by design, from the lack of paths to sustainability for the projects they launch (“Are Civic Hackathons Stupid?,” 2013). In response to such criticisms, there are increasing efforts, such as “Make the Breast Pump not Suck” hackathon and “Trans\*H4CK”, to improve hackathon methodology by working directly with affected communities (D’Ignazio & Klein, 2020). Several initiatives such as MIT collaborative design studio provide insights on hackathon methods (Costanza-Chock, 2020) to facilitate better hackathons (Day et al., 2017; DePasse et al., 2014). More recently, multiple entities have engaged in organizing hackathons to address the COVID-19 crisis (Pan-European Matchathon, n.d.; #WirVsVirus - #WirVSVirus, n.d.). Challenge-based approaches, providing frameworks for learning while solving real-world issues, have also been on the rise in global health, incentivizing collaboration across a broad range of disciplines and backgrounds (Pan et al., 2017). For example, the potential of participative models to address complex questions, and the power of contests to offer a structure that catalyzes this work, has been exhibited by the “Epidemium” initiative on cancer epidemiology (Benchoufi et al., 2018a).

Despite the numerous tools and technologies created to facilitate collaboration and communication in citizen science projects, challenges remain. These include the issue of the complementarity, coherence and diffusion of these initiatives (Bosman et al., 2017) to efficiently address international policies and local needs, as often the local adoption of hackathon solutions remains low (“Are Civic Hackathons Stupid?, ” 2013).

At a more fundamental level, large-scale collaborations introduce a set of unique challenges, from communication to coordination, which, if left unaddressed, could jeopardize the completion and the success of the project. Prompted by the prevalence of the phenomenon, multiple studies have explored how team size (Wu et al., 2019), composition (Guimerà et al., 2005; Palla et al., 2007; Uzzi et al., 2012; Woolley et al., 2010), organization (Barkoczi & Galesic, 2016; Klug & Bagrow, 2016; Uzzi & Spiro, 2005) or dynamics (Riedl & Woolley, 2016) determine team survival and performance. Such studies rely on network theory and dynamical systems frameworks (Palla et al., 2005; Scharnhorst et al., 2012), allowing in particular to unravel hidden community workflows and information exchange within topological communities (Lehmann, 2019). In addition, the sociology of science has offered numerous in-depth insights from case studies (Owen-Smith, 2001), revealing the factors that influence the way a group of individuals work together (Latour & Woolgar, 1986). This has led to the extraction of four organizational challenges of particular importance for coordinating citizen science projects: the division of tasks into simple, well-defined entities, the allocation of tasks to different actors matching their area of expertise, the provision of rewards to motivate contributions, and the provision of information to enable successful collaboration and decision making (Nosek et al., 2015; Puranam et al., 2014).
Facing these challenges, the individuals or organizations running large scale networked projects have a limited understanding of the types of coordination that are needed when multidisciplinary contributions exist (Dickel & Franzen, 2016). In this report, we aim to document best practices that we have observed from our experience running large-scale, interdisciplinary crowd science projects during the 2019-2021 period to address complex and global challenges within the collaborative platform Just One Giant Lab (JOGL). JOGL is a collaborative open platform aimed at designing a virtual, open, and distributed research institute to develop solutions in order to meet the Sustainable Development Goals. Since March 2020, a community of 4,000+ individuals has self-organized within JOGL to develop low-cost, open-source solutions to mitigate the COVID19 crisis.

### 1.2.2. Objectives

Our main objective is to distillate learnings from 2 years of working with distributed, international, interdisciplinary communities. We aim to provide a clearer picture of the coordination, communication, governance and reward systems that have been implemented, and discuss their relevance to the case of a Complex Systems Hub.

### 1.2.3. Assumptions

Some concepts that we use in this report can have different interpretations, therefore we provide a few definitions in order to clarify our language:

* **Community:** we refer to a group of individuals with an emotional connection or/and identification between members of the group. If group members do not have an identity or emotional affiliation with one another and don't share a group identity, then there is no feeling of community. Examples can be found with intentional communities (see the Conscious Coliving Community Facilitation playbook<fnref n="1" />). In the scientific context, such groups constitute “**epistemic communities**”, identifying through their shared interest towards a certain topic.

* **Crisis:** is any event or period that either will or might lead to an unstable and dangerous situation affecting an individual, group, or all of society.

* **Interdisciplinarity collaboration** - refers to a collaboration taking place between scientists coming from different disciplines of science (*Open Innovation*, *Open Science*, *Open to the World*, 2016).

This report is aimed at interdisciplinary, epistemic, open communities. We assume that the community is predefined (colleagues, members of an organization) and constitutes an open system (individuals can enter and leave the system). Individuals in this community have various backgrounds (scientific and non-scientific) and interests, and do not necessarily know each other beforehand. Yet, they are tied by the same goal, project, or organization. We do not restrict our scope to the case of individual motivation driving actions (as is often the case in citizen science), and assume that there is an incentive structure for contributing

<footnote n="1">¹ https://www.consciouscoliving.com/the-community-facilitation-handbook/</footnote>
to tasks (e.g. monetary or authorship). We finally assume that individuals speak at least one
language in common and use digital tools to communicate.

## 2. Methods and Implementation

### 2.1. Study design and methods

This report provides a retrospective study of the OpenCovid19 program on JOGL. Case studies are often used in the fields of business, policy, and medical sciences in order to investigate complex issues or interventions in real-life settings (Crowe et al., 2011). We leverage this approach to discuss the barriers and facilitators for implementing such interdisciplinary programs to inform future decisions.

To document the requirements for a successful communication structure in interdisciplinary research teams, we focus on the OpenCovid19 community. In particular, we build on findings from Olga Koshagina in her qualitative study on the OpenCovid19 program (Kokshagina, 2021) as well as recommendations elaborated by the JOGL coordination team in the playbook "How to build and manage Open Communities" (Mains & JOGL community, 2020).

The qualitative study of (Kokshagina, 2021) relies on data collected through a virtual community ethnographic observation (in particular community calls) combined with secondary data collection on the platform and connected tools that the community used, including Slack, Facebook, Twitter, and Medium over 5 months (March 2020–August 2020). The analysis then relies on a thematic coding procedure to identify, analyze, and report emerging themes from the data. The report from (Mains & JOGL community, 2020) relies on semi-structured interviews of community members of OpenCovid19 and was collaboratively edited by members of the community.

The JOGL community playbook is a useful resource to help navigate how these insights translate into community building guidelines (Mains & JOGL community, 2020). This guide can be found in an open repository at the following address: https://app.jogl.io/project/510.

In order to complement the findings extracted from the OpenCovid19 case study, we performed an exploratory literature review and integrated learnings from other open research programmes that have been run within the JOGL framework (Benchoufi et al., 2018b; Masselot et al., 2021).

### 2.2. State of the Art

The OpenCovid19 programme that we focus on in this report falls within the "Crowd Science" (Franzoni & Sauermann, 2014) and open innovation (Kokshagina, 2021) frameworks. The focus on "Grand Challenges" is particularly useful to situate the present report. As summarized by (Kokshagina, 2021):

> George et al. define grand challenges as global problems that are only effectively addressed through a coordinated and collaborative effort (George et al., 2016). Grand challenges are subject to enormous complexity because of the many interactions and nonlinear relations,
and the outcomes and problem spaces are highly uncertain (Ferraro et al., 2015). Working on grand challenges involves collaboration on complex questions that require multidisciplinary contributions (Sun et al., 2020). Participants generally belong to different communities with different practices and approaches. Therefore, the collaboration needs to evolve and includes new contributors who may introduce new skills and data into an open and complex research design (Van Rijnsoever and Hessels, 2011; König et al., 2013). [...] Grand challenges usually require going far beyond the span of formal organizational control where clear hierarchy is absent, *making typical top-down approaches to management ineffective in resolving these problems*. Owing to the limits of top-down approaches, academics turn to alternative forms of organizing where partners and communities work together to drive their R&D efforts (Chesbrough et al., 2006; Ahn et al., 2019).

Beyond qualitative insights, a large-scale quantitative analysis was recently conducted across 32 initiatives from 6 continents to extract standard modes of co-production to address complex sustainability challenges (Chambers et al., 2021). The authors exhibited six modes of co-production: (1) researching solutions; (2) empowering voices; (3) brokering power; (4) reframing power; (5) navigating differences and (6) reframing agency. They then scored the different approaches in terms of activating different outcomes, from network building to inclusion, risk management or knowledge production. This study highlights how different methods can promote different goals, and can therefore guide the design of co-production networks beyond the case study presented within this report.

# 3. Results

## 3.1. The OpenCovid19 context

### The OpenCovid19 community

On March 1, 2020, a community of biologists, developers, and data scientists launched the OpenCovid19 project on the JOGL collaborative open platform. JOGL aims at designing a virtual, open, and distributed research institute to develop solutions in order to meet the Sustainable Development Goals (SDGs) defined by the United Nations. Within only 1 month, there were over 60,000 visitors from 183 countries, including 3,000 active contributors generating more than 90 projects ranging from mask designs to low-cost ventilator prototypes and cough-classification artificial intelligence apps. The community spanned several areas: hardware and software development, data analysis, biology, project management or education among others (Figure 1).
<figure><img src="image_3.png" /><figcaption>Figure 1. Main areas of focus of the OpenCOVID19 community in JOGL: hardware and software development, data analysis, biology, project management, education and others.</figcaption></figure>

## Structure of OpenCovid19

<caption>Table 1: Summary statistics on the OpenCovid19 challenges (Kokshagina, 2021).</caption>

<table><caption>Table 1. Overview of challenges and projects</caption><thead><tr><td>Challenge</td><td>Description</td><td>Number of projects submitted</td><td>Number of participants</td><td>Number of needs</td><td>Number of claps</td></tr></thead><tbody><tr><td>Covid-19 diagnostic and detection</td><td>Research and develop open-source and low-cost solutions to detect Covid-19</td><td>27</td><td>566</td><td>43</td><td>17</td></tr><tr><td>Evaluation and validation of open-source solutions</td><td>Evaluate and validate the best open-source designs capable of helping us fight the Covid-19 pandemic</td><td>12</td><td>154</td><td>16</td><td>6</td></tr><tr><td>Covid-19 treatments and therapeutics</td><td>Research and develop open-source and low-cost solutions to treat Covid-19</td><td>12</td><td>674</td><td>16</td><td>13</td></tr><tr><td>Covid-19 prevention</td><td>Research and develop open-source and low-cost solutions to prevent the spread of Covid-19</td><td>39</td><td>274</td><td>61</td><td>18</td></tr><tr><td>Data analysis and simulation</td><td>Research and develop data collection methods, data analyses, models, and simulations to describe and forecast the Covid-19 pandemic</td><td>16</td><td>104</td><td>37</td><td>9</td></tr></tbody></table>

The JOGL team organized the program across five challenges. Table 1 provides an overview of the number of projects submitted, members, needs indicated, and claps received on the JOGL platform. The program's objective was to develop alternative designs for the equipment, to test and manufacture them by building a community of partners, and to share resources in order to provide complimentary R&D support.
As such, OpenCovid19 presents a case of extreme open crowdsourcing: solvers collaborate openly and can join at any time. JOGL provides a platform of full disclosure where results are documented openly and shared with anyone. This case is therefore relevant to study the coordination of extreme crowdsourcing for grand challenges.

OpenCovid19 operates within an open research programme coordinated by JOGL. The architecture of JOGL programmes is shown in Figure 2 and further detailed in (Masselot et al., 2021). In order for the community to focus on project development and implementation, JOGL focuses its activity on tool deployment (JOGL platform and Slack workspace), event organization (community calls, challenges), and community coordination (facilitating onboarding and collaborations), and financial support via micro-grants allocated by community peer-review.

<figure><img src="image_4.png" /><figcaption>**Figure 2: The architecture of a JOGL program**</figcaption></figure>

People could join as project leads or participants and contribute to projects, interest groups or working groups.

Along with the platform, a Slack workspace was provided for fostering community communications. This allowed for an “emergent coordination” to happen based on the communication needs of the community (see Results). The network of Slack channels highlights the structure of the OpenCovid19 community (see Figure 4).

The main components underlying the OpenCovid19 community structure is summarized in table 1 below:
<table><caption>Table 1: OpenCovid19 components</caption><thead><tr><th>JOGL PLATFORM<br>-- infrastructure and leadership to fuel OpenCovid19 Community</th><th>OPENCOVID19 STEWARDS<br>-- groups of community members providing community-wide support</th><th>OPENCOVID19 TEAMS<br>-- groups of community members addressing program challenges</th><th>OPENCOVID19 COMMUNITY<br>-- collective of individuals focused on open source solutions related to covid19</th></tr></thead><tbody><tr><td><strong>Coordination/Strategy</strong><ul><li>Ensure that all individuals and groups are receiving the support they need and are adhering to the initiative missions and expectations</li></ul><strong>Program -- Open Covid19 Initiative</strong><ul><li>Oversees grant allocation, partnerships and creates Challenges</li></ul><strong>Challenges</strong><ul><li>Global needs expressed by JOGL or the Open Covid19 Community</li></ul></td><td><strong>Biosafety Advisory Board</strong><ul><li>Panel of community experts providing projects with guidance regarding biosafety and biosecurity</li></ul><strong>Steering Committee</strong><ul><li>JOGL team members and OpenCovid19 community ambassadors establish and resolve short and long term cross-functional goals of community</li></ul><strong>Working Groups</strong><ul><li>Collective of community members organized around community-wide objectives relative to team needs (eg Governance, Deployment, Communications, Project Review etc.)</li><li>Maintain channel on OpenCovid19 Slack</li><li>Elect Ambassador to communicate needs, opportunities, successes and cross-functional requests to and from the community at large</li></ul></td><td><strong>Projects</strong><ul><li>Collective of community members explicitly addressing JOGL challenge/s</li><li>Complete project profile on JOGL platform</li><li>Maintain channel on OpenCovid19 Slack</li><li>Elect Project Manager to track results and keep team focused</li><li>Elect Ambassador to communicate needs, opportunities, successes and cross-functional requests to and from the community at large</li></ul><strong>Interest Groups</strong><ul><li>Panels of community experts committed to advise and design solutions for challenge/s (Software, Hardware, Design, Research, Students, Public Health, etc.)</li><li>Maintain channel on OpenCovid19 Slack</li><li>Elect Ambassador to communicate needs, opportunities,</li></ul></td><td><strong>Community Members</strong><ul><li>Complete intake survey</li><li>Complete JOGL profile</li><li>Join OpenCovid19 Slack</li><li>Submit post about experience and expertise to #prgm-introductions</li><li>Join project team, interest group or working group</li></ul><strong>Project Managers = project leaders</strong><ul><li>Accountable for tracking results and keeping teams focused</li></ul><strong>Community Ambassadors</strong><ul><li>Primary point of contact on teams, communicating needs, opportunities, successes and cross-functional requests to and from the community at large</li><li>Reporting project needs and updates to program, challenge, level groups</li></ul></td></tr></tbody></table>
<table><tr><td></td><td></td><td>successes and cross-functional requests to and from the community at large</td><td></td></tr></table>

We then describe how these components operate within the community, by disentangling what is done, by whom, and where:

<table><caption>**Table 2: OpenCovid19 community structure**</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Program level</th><th colspan="3">Community / Project</th></tr><tr><th>Challenges</th><th>Projects</th><th>Individuals</th></tr></thead><tbody><tr><td>What happens?</td><td>Coordination: systems, processes and tools are designed and implemented so that all individuals and groups in the community are receiving the support they need and are adhering to the initiative missions and expectations. (e.g. the onboarding process)<br>Strategy: decisions are being taken that determine the initiative's growth and trajectory. (e.g. the social media strategy.)</td><td>Collaboration takes place between project teams working on different solutions to the same challenge (e.g. an innovative face mask and a new type of hand sanitizer are solutions that both address the "Prevention" challenge).</td><td>Solutions to covid19-related challenges are developed within teams.</td><td>A human is choosing to join the OpenCovid19 initiative.</td></tr><tr><td>Who participates and how?</td><td>Any member of the community can participate at the program-level.<br>Ways to get involved:<br>- Join a WG: participate in the decision-making on how the initiative is being structured.<br>- Be a project ambassador: represent your project team's experience, and your project's programmatic needs to the appropriate program-level workgroup so that the opencovid19 initiative can work better for you.</td><td>- Project teams are sharing updates, pain points, and best practices</td><td>- Project members:<br>→ Build solutions<br>→ Recruit new talent and source tools and infrastructure needed for their solutions to be successful<br>→ Can report on project updates during Global Community Calls<br>- Project leads:<br>→ Lead the project internally within the project's Slack channel and on the JOGL platform<br>→ Are the technical experts on the project's solution<br>- Project Ambassadors:<br>→ Report on project needs to appropriate program-level WG<br>→ Are most aware of the project team's structure, needs, pain points</td><td>Anyone can participate, by:<br>- adding value to a project<br>- recruiting new talent for a project<br>- sourcing technology for a project<br>- helping coordinate the initiative<br>- observing, browsing, and learning</td></tr></tbody></table>
<table><tr><td>Where does it happen?</td><td>Purely on Slack</td><td>Slack:<br/>- Instant communication between individuals working on challenge-related projects<br/><br/>JOGL:<br/>- List of all projects within each challenge</td><td>Slack:<br/>- Instant communication between team members<br/>- Each project has at least 1 dedicated channel<br/><br/>JOGL:<br/>- Projects are listed<br/>- Collaborators are linked to project pages<br/>- Project needs are listed</td><td>Slack:<br/>- Each individual has a profile<br/>- Individuals join channels according to the role they want to play<br/><br/>JOGL:<br/>- Each individual has a profile listing the expertise / skill / technology they can offer<br/>- Individuals browse project needs and contact projects to contribute</td></tr><tr><td>What are the working groups / workgroups?</td><td>Program-level WG:<br/>- Communications<br/>- Community-Management<br/>- Funding + project review<br/>- Governance (aka Steering Committee)<br/>- Metastudy<br/>- Student engagement<br/>- Dev<br/>- Design<br/><br/>Each WG has at least 1 dedicated channel.</td><td colspan="3">2 types of workgroups:<br/>- A project team working on a specific covid19 solution is a WG<br/>- A channel that focuses on a specific technology that is applicable to multiple covid19 solutions is a WG</td></tr></table>

As the community scaled, we saw a stratification emerge into working groups and interest groups that allowed to both improve the community coordination as well as create spaces for subcommunities to thrive:

<table><caption>**Table 3: Working groups and their objectives**</caption><thead><tr><td>Working groups</td><td>Objectives</td></tr></thead><tbody><tr><td>Coordination</td><td><ul><li>Receive needs and priorities from the field by relevant authorities and community organizers</li><li>Review project propositions by community members and provide feedbacks for improvement</li><li>Converge toward commonly accepted solutions that targets best meet the needs and priorities defined</li><li>Offer resources to promising initiatives</li><li>Help validate tools and methodologies that has been developed through the program</li><li>Share, disseminate, and help the implementation of validated solutions with the world</li></ul></td></tr><tr><td>Governance</td><td><ul><li>Provide rapid guidance to the opencovid19 initiative growing community</li><li>Provide framework and clarity of organization for [new] members</li><li>Establish guiding principles for community building and workflow</li></ul></td></tr><tr><td>Learnings</td><td><ul><li>Identify what works and what doesn't to create and sustain a open science community powered by volunteers working on the creation and development of</li></ul></td></tr></tbody></table>
<table><caption>Table 4: Interest groups and their objectives</caption><tr><td></td><td><ul><li>open-source projects</li><li>Describe the Opencovid19 program framework: including its design, governance/coordination, community, major features</li><li>Present the Opencovid19 program outputs: its community, individual successes, projects, commons</li><li>Assess the link between the program framework and the outputs</li><li>Identify the strengths and weaknesses of the program for open science project creation and development and creation and maintenance of an open science community</li><li>Assess the satisfaction of the participants regarding the opencovid19 initiative</li><li>Showing case studies of success stories and failures of projects within OpenCovid19.</li><li>Equip people to be more successful in their own experience within open-source communities.</li></ul></td></tr><tr><td>Communication</td><td><strong>External communication</strong><ul><li>Ensure that we properly communicate with the greater public the aims of the initiative and the details of specific projects</li><li>Manage any relevant social media accounts, postings, and platforms, including but not limited to: Facebook, Twitter, LinkedIn, Instagram, JOGL (project/challenge/program page content is communicated clearly)</li><li>Ensures that all branding and messaging regarding the initiative is consistent and of high quality</li><li>Keep track of medias talking about the initiative</li><li>Develop resources to manage press interactions</li></ul><strong>Internal communication</strong><ul><li>Communicate program updates on the announcement channel.</li><li>Ensure that the communication of the program and projects are unified and effective</li><li>Elaborate and implement communication processes on internal communication channel for community management</li><li>Promote respect of the program code of conduct and report or infrangment of the code of conduct</li></ul></td></tr><tr><td>Project Management</td><td>Work on the facilitation of project management by providing guidelines, tools, templates.<br>A place to coordinate capacity-building workshops and webinars. These virtual sessions will feature experts from the community and help community members develop skills for individual, team, and project success.</td></tr><tr><td>Project review</td><td>Review projects in need of microgrants to determine and allocate proper funding</td></tr><tr><td>Meta Study/Analytics Workgroup</td><td>Group interested in analyzing the workings of this initiative as a whole, producing a "science of science" study to help future large-scale international scientific collaborations</td></tr></table>

<table><thead><tr><td>Interest groups</td><td>Objectives</td></tr></thead></table>
<table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Software Workgroup</td><td>(Some thoughts on what might go here are collected in OpenCOVID19 Software Workgroup - Initial Thoughts)</td></tr><tr><td>Hardware</td><td>Group focused on assisting with any hardware needs on the program, challenge, and project levels. Also works on developing more projects/opportunities for software developers/engineers and related people.</td></tr><tr><td>Design</td><td>Interest in Design</td></tr><tr><td>Research</td><td>Interest in Research</td></tr><tr><td>Logistics</td><td>This is a channel to help each other get access to needed reagents and equipment--post requests for reagents, links/docs compiling reagent sources, etc</td></tr><tr><td>Students</td><td>A space to engage students/early-career members. If you are part of a project that you believe will benefit from student involvement, post them here! In the future, there may also be projects and collaborations launched by this channel as well.</td></tr><tr><td>Chicago</td><td>People in Chicago</td></tr><tr><td>OpenCovid19-africa</td><td>Applying the JOGL OpenCovid19 approach in the African context</td></tr><tr><td>OpenCovid19-global-south</td><td>Applicability and reproducibility of OpenCovid19 projects in low resource countries of the Global South</td></tr><tr><td>OpenCovid19-spanish</td><td>Spanish channel for easy communication</td></tr><tr><td>Funding opportunities</td><td>Interest in founding opportunities</td></tr></tbody></table>

## Communication within OpenCovid19

In terms of information circulation, we had two specific objectives:

- **External**: keep the public informed on what projects are doing within the OpenCovid19 initiative

- **Internal**: highlight needs and avoid redundancy by creating synergies ("designing serendipity")

To meet these objectives, the communication strategy focused on several tools (Table 5), processes (Table 6), and events (Table 7) described below.

<caption>**Table 5: Tools used for the communication strategy within OpenCovid19**</caption>
<table><caption>Table 6: Communication pathways within OpenCovid19</caption><thead><tr><td>Channel</td><td>Communication purpose</td><td>Audience</td></tr></thead><tbody><tr><td>Slack</td><td><ul><li>Gather all internal communication of the community</li></ul></td><td>OpenCovid19 community</td></tr><tr><td>JOGL Platform</td><td><ul><li>Share the needs of projects</li><li>Document the progress of projects</li><li>Open access of protocols, results, and data of projects</li><li>Allow for independent matching between community members</li><li>Provide sufficient documentation for peer-review of projects</li></ul></td><td>JOGL community</td></tr><tr><td>Social media</td><td><ul><li>Share success stories</li><li>Inform members of the community of events</li><li>Recruit potential contributors</li></ul></td><td>Potential contributors</td></tr><tr><td>Press</td><td><ul><li>Inform decision makers and the general public</li><li>Give visibility</li><li>Disseminate findings</li><li>Share success stories</li></ul></td><td>General public</td></tr></tbody></table>

<table><caption>Table 7: Types of events run within OpenCovid19</caption><thead><tr><td>From... to ....</td><td>Information</td><td>Process</td></tr></thead><tbody><tr><td>governance body →<br/>governance bodies</td><td>Decision taken which involve the implication of another governance bodie</td><td>Via: Slack<br/>Documents on Drive</td></tr><tr><td>governance body →<br/>individuals</td><td>Minutes of meetings<br/>Decisions at program level<br/>General announcement: partnership</td><td>Via: Slack<br/>As soon as possible</td></tr><tr><td>Individual →<br/>governance body</td><td>Suggestions for organization, community management, features, etc<br/>Update on project stage of advancement<br/>Report of bugs on JOGL<br/>Report of code of conduct infringement<br/>Raise concerns towards a governance body</td><td>Forms<br/>Slack<br/>JOGL platform</td></tr><tr><td>individuals →<br/>individuals</td><td>Anything</td><td>Via: Slack, JOGL<br/>platform</td></tr></tbody></table>

<table><thead><tr><td>Event</td><td>Objectif</td><td>Audience; number of attendees; frequency</td><td>Agenda</td></tr></thead><tbody><tr><td>Global Community calls</td><td><ul><li>Update the community on program activities</li><li>Create synergies among projects (match needs and resources)</li></ul></td><td><ul><li>OpenCovid19 community</li><li>General public interested in joining</li></ul></td><td><ul><li>JOGL update</li><li>Work group updates</li><li>Project updates: status of advancement, needs</li><li>Open discussion</li></ul></td></tr><tr><td>JOGL Live</td><td><ul><li>Provide targeted support to one project</li><li>Foster the</li></ul></td><td><ul><li>OpenCovid19 project</li><li>Relevant community members</li></ul></td><td><ul><li>Round of introduction</li><li>Presentation of the projects and needs</li></ul></td></tr></tbody></table>
<table><tr><td></td><td>sustainability of the project by connecting with partners and experts</td><td><ul><li>Experts</li><li>Partners</li></ul></td><td><ul><li>Discussion on how to best meet the project needs</li></ul></td></tr><tr><td>Onboarding call</td><td><ul><li>Present OpenCovid19 to new members</li><li>Facilitate onboarding</li></ul></td><td>JOGL coordinator, new members; on demand</td><td><ul><li>Onboarding video</li><li>Presentation of the tools &amp; process</li><li>Q&amp;A</li></ul></td></tr></table>

## Mode of co-production within OpenCovid19

We can frame this initiative according to the framework provided by (Chambers et al., 2021). Chambers et al. highlight in particular two key motivations for co-producing: to more effectively solve predefined problems and to reframe problems. OpenCovid19 belongs to both categories, as it aims at providing solutions to the COVID19 crisis while coping with the constantly evolving needs generated by the crisis. Leveraging the categories produced by Chambers et al. (Figure 2), we can situate OpenCovid19 as being “Researching solutions”, putting an emphasis on producing knowledge and solving problems by fostering direct agency rather than reframing paradigms.

<figure><img src="image_5.png" /><figcaption>**Figure 3: Six modes of co-production.** Figure taken from (Chambers et al., 2021), highlighting the 6 general modes of co-production, situating OpenCovid19 within the mode "1. Researching solution".</figcaption></figure>
## 3.2. OpenCovid19 workflow

In the next sections we elaborate key learnings for best practices based on the OpenCovid19 programme workflow. We divide it into two parts: describing the "emergent" coordination structure, and the communication and open knowledge exchange:

**Principles underlying Emergent Coordination:**

* Facilitating the onboarding and orientation of voluntary participants—participative architecture

* Continuously embedding and improving predictability for participants

* Collective building of common understanding

* Gradual self-accountability

* Continuous experimentation

**Principles underlying Open Knowledge Exchange:**

* Multidisciplinary knowledge integration

* Engaging in exchanging efforts and not committing directly to the entire project

* Extreme transparency in knowledge produced and knowledge capitalization

* Facilitated and open translation/ appropriation of the results

## 3.3. Emergent coordination

### 3.3.1. Facilitate the onboarding and orientation of participants—participative architecture

Due to the open nature of the OpenCovid19 ecosystem, new individuals are continuously joining the initiative and are confronted with an existing and large structure within which they need to be able to navigate. During their journey, they will enter new projects, groups or communities within that initiative. Their proper onboarding and orientation is therefore key to efficiently integrate them within the initiative. The onboarding process should be seamless, designed to reduce friction and be as fast as possible to minimize the chances of people to drop-out. We develop below recommendations for optimizing the onboarding process.

**Acknowledge the motivation of future community members and clearly communicate the core value and vision of your initiative**

**Understanding the motivations** of individuals joining an initiative is key to setting up the right type of communication process. Individuals join communities based on shared interest or based on social bonds. The design of the community then affects the type of bonds created within it (Ren et al., 2007). OpenCovid19 is an example of an interest-based community. In OpenCovid19, people joined because they felt the urge to solve the COVID19 crisis, and most people didn't know each other before.

Motivations are different across individuals and can conflict with the core values of the
initiative. It is therefore critical to express the **core values of the initiative** early enough in order to provide a motivation beacon for individuals who join the initiative. In OpenCovid19, we co-elaborated the values of the initiative with community members and formalized them in a code of conduct (see Annex, Code of Conduct). By contributing to the initiative, you agree to act in good faith to maintain an environment of inclusion, safety, and open dialogue. We repeated these principles at the beginning of each global community call. The code of conduct was made visible to all, especially to the newcomers during the onboarding process.

## Facilitate triaging and navigation of your initiative

It is possible to leverage motivational aspects and shared values at the entry point of an initiative by using a mix of self-triaging and point-of-contact triaging.

Triaging was done at the very first step when someone entered, focusing on what they have to offer and what they need. While triaging can be active and involve the community coordinators, self-triaging can be fostered by providing participants with a triage diagram (see Annex, Triaging Diagram). On the online JOGL platform, two features were designed to foster self-triaging. First, the activity status of ongoing projects was useful to guide participants in places where other participants were active. Second, project needs could be explored by the type of skills required as well as date added, allowing to guide participants towards projects where they could best contribute.

Yet, self-triaging can be limited, and it is crucial to have a human **point of contact for triaging**, a person that participants can refer to if they are lost. We observed that the identification of such a role led to an increased engagement rate of newcomers. Similarly, at the level of individual team-based projects, it is important to specify one specific person that has an overview of the project needs and can interface with the rest of the community and the coordination team.

## Ease your onboarding process

In order to facilitate the onboarding process, we did three things. First, we created an onboarding web page including: a summary of active projects and their needs, the latest announcements from the core team, and an overview of active challenges. We provided static information in the form of a "Get started" and "Learn more" buttons, leading to pages explaining the background of the initiative and the typical participant trajectory within the community. Participants could also access a page showing the registered members, with an overview of their skills, resources, interests in particular Sustainable Development Goals (SDGs) and involvement in projects.

Second, we elaborated an onboarding document (see Annex, Onboarding document) that was available at the forefront of all entry points: the JOGL platform and Slack.

Finally we set up automatic channels for people to join in order to facilitate their onboarding. In addition, people were asked to introduce themselves and explain what they can contribute to directly in the introduction channel. This allowed the coordination team to follow up by linking them to relevant people or resources. On demand, coordinators would run 15 minutes one-on-one **interviews with members** to clarify what their expectations are and help them
navigate the initiative. When the demand was too high to have individual conversation, the coordination team organized weekly onboarding calls.

One should be careful about the consistency of the people and processes associated with the onboarding. We tried to implement several roles such as greeters, guides, moderators, and moderators (see Annex. call for community leads) but we had very limited engagement. We hypothesized that the mid-term commitment discouraged people from taking on these roles. To increase engagement, especially in the case of punctual needs (a rush during a crisis), we recommend to gamify the process by setting numerical objectives (eg, "match 10 people to relevant projects"). In addition, we note that students mentioned that such roles were a good opportunity for them to "contribute" when their technical skills were not serving projects or when they wanted to experience leadership and management positions.

### 3.3.2. Continuously embedding and improving predictability for participants

#### Structure your community around one or multiple clear and reliable core goals

It is important to clearly define "successes" for the initiative. Some communities use an emotional projection approach for defining success: "what does success looks like?", "what does it sound like?", "what does it feel like?". This is a good exercise to test your different personas (typical users), since success may sound, look, and feel different depending on the members of your initiative. You may not find a consensus on what success is, but you can try to cover and highlight the diversity of successes to the members of your community. For OpenCovid19, one "visible" success was the design of solutions and their implementation ("what does it look like?"). Volunteers in the initiative reported that they enjoyed contributing to something bigger than themselves ("what does it sound like?") and felt that they were doing something useful ("what does it feel like?").

#### Be transparent on how your community is structured and governed, and on how decisions are made

The organizational structure should be made clear to allow individuals to thrive, and also have a built-in feedback structure to be alert for the needs of the individuals and various parts of your community. It is particularly important to have a **clear and stable leadership** that provides a long-term scaffold to the community. Finally, engaging your members in designing and maintaining your community structure can help in creating a shared vision.

To increase transparency in OpenCovid19, members of the community created a **governance working group** that was open for everyone to join in or leave. This group used diverse modes of information sharing: written on Slack for rapid updates and on Google Docs for more permanent storage, and oral during the weekly calls of the governance group. Most importantly, one should be vigilant that the group is not operating in silos. For this, one needs to make sure that all the relevant stakeholders are informed and consulted. In our case, we provided extensive documentation on Google Drive and held topical conversations and subsequent decisions at different meetings to give all community members time to read and contribute.
## Find the balance between agility and rigidity in your initiative's structure

In order to foster feedback between the community evolution and the organizational structure, we implemented practices for ensuring both rigidity and agility of our structure. For enhancing clarity, we created a **nomenclature on Slack channel names** for gaining in clarity. We asked projects to define a **project lead** who would be the focal point for communications. Finally, we formalized **roles and accountability** in governance documents that were made accessible to all, clearly identifying coordination teams and providing guides for navigating the community.

The Slack nomenclature was the following:

* **#prgm-xx**: all channels related to program-wide discussions (working groups, interest groups, boards, coordination)

* **#chlg-xx**: all channels related to challenge-wide discussions

* **#proj-xx**: all channels pertaining to a specific project

Within this rigid framework, the community was invited to take leadership in changing the structure of the initiative and make the initiative “agile”. For this, we used the following concepts:

* **Minimum viable bureaucracy**

* **Do-ocracy**, typical of open-source ecosystems, promoting individual leadership in taking actions. For example, anybody could create Slack channels or initiate a task-force when needs arise.

* **Holacracy and sociocracy**

Tools such as Holaspirit or Glassfrog, often used to implement holacracy, can help visualize and navigate such a distributed structure

## Systematize your communication process

In the context of OpenCovid19, the use of public channels on Slack was essential in order to monitor that communication flows well across the community. Creating a nomenclature for the channels helped ensure they would fit a clear purpose. Meta-analyses of social interactions on public channels can then reveal communication barriers: for example the need for an improved onboarding method in Figure 4.
<figure><img src="image_6.png" /><figcaption>**Figure 4: Slack interaction network** from the week of 1st April 2020, showing public channels as purple nodes with active individuals connected to them. The yellow circle indicates an onboarding bottleneck where new members posting in the introduction channels are not connected to the rest of the network, highlighting a need for a more efficient onboarding methodology that was then implemented. The core of the network is constituted by the coordination team from JOGL HQ members and community leaders, and connects the various subcommunities working on software development, hardware development and biological projects.</figcaption></figure>

### Adapt your structure and process to the rhythm of the activity of your projects

Like living organisms, organizations require energy to sustain their activity. The maintenance of an activity or a sub-group can be costly for the organization, which will need to adapt to be more efficient. In our case, we followed up with active members who left the initiative in order to inquire about the reason for their disengagement, and discuss possible mitigation strategies.

### 3.3.3. Collective building of common understanding

#### Choose an inclusive communication tools, and strategy

The communication strategy is of prime importance to ensure everyone has access to the right information to navigate the initiative and create meaningful collaboration.
ensure that all voices can be heard, and avoid that personality differences build into structural communication biases. In general, communication tools should be limited in terms of number but cover the entire needs of your members, with each tool having one defined purpose. When choosing tools for community interactions, one should consider:

- The existing tool usage habits in your community (Teams, Slack, Mattermost...)

- The geographic disparities: take into account time zones when planning on global meetings. You may want to plan two sessions of meetings (most common is 9am CET and 6pm CET) in order to cover all timezones.

- The literacy level

- The diversity of mindsets: some individuals need time to reflect before deciding, or may need to think about the question before giving an answer

## Use participatory approaches to empower your community members

Community participation allows for a greater engagement of the members in the overall project (World Health Organization, 2021). In the citizen science context, the involvement of citizens in the design of research projects demonstrated an increased relevance of the outcomes. In our case, we leveraged weekly global community calls to give participants the opportunity to voice their needs and contribute to the decision making.

In an interdisciplinary community it is important to ensure that all voices can be heard at all times. This applies to the process of making decisions, to how to run meetings.

Simple rules about meetings:

- Be careful about who speaks first, who doesn't speak, and who decides who speaks

- Never bring up a topic, discuss and decide in the same meeting

- Share the agenda in advance so that people have time to prepare, suggest changes or communicate important information if they can't attend

- Give people time: if you want to brainstorm a topic, make sure you give enough time to people to think individually before putting thoughts collectively

There is a thin balance to find between being inclusive, horizontal, and overloading people with too many things to give their opinion on.

Below we list the questions and answers that drove our process when reflecting on consultation of the community member in decision making. We encourage you to reflect on these.

<table><thead><tr><td>Questions</td><td>Answers for OpenCovid19</td></tr></thead><tbody><tr><td>On which occasions the community should NOT be consulted?</td><td><ul><li>When the biosafety board decides to exclude a project following their procedures</li><li>When a person is going through mediation process because of infringement of the code of conduct</li></ul></td></tr><tr><td>On which occasions the community MUST be</td><td><ul><li>When decision taken can affect in one way or another the work and collaboration of the community</li></ul></td></tr></tbody></table>
<table><tr><td>consulted?</td><td><ul><li>For strategy documents and major guiding principles</li><li>For elections/appointment of people in the governance bodies</li></ul></td></tr><tr><td>How to run a consultation?<br/>Tool, deadline, etc</td><td><ul><li>This should be decided by the team asking for consultation</li><li>Team should take into account the results of consultation, and be genuinely open to constructive feedback</li></ul></td></tr></table>

### 3.3.4. Gradual self-accountability

#### Anticipate risks, offer mediation and mitigation of conflicts

Don't assume your members will successfully self-regulate or self-organize. In addition, in open science projects involving scientific experiment it is crucial to connect with appropriate ethical committees and regulatory bodies.

The OpenCovid19 initiative :

* Compliance with RGPD was the responsibility of JOGL

* At the very early stage, the community self organized to build a bio-safety board, composed of volunteers and experts

* The coordination team set up a mitigation of conflict and report systems available to all in the code of conduct which has been continuously adapted after each case

#### Identify, document and share how decisions are taken and by whom

Decisions related to the structuring of the community and its communication processes requires a governance model that should involve experts in management or technical boards. It is particularly important to ensure that you have a buy-in of the community members about the individuals represented in the technical boards.

In our case, we kept regular contacts with projects by conducting one-on-one meetings with project leads to detect barriers, and report back at a higher level. Setting early mechanisms of accountability and decision making is particularly important to avoid unilateral decisions that go against the opinion of the community.

The weekly survey shared with projects had a question to mention the project needs, and a spreadsheet for anonymously reporting structural or organizational issues was shared among members of the coordination team. We tried to create an ambassador group composed of representatives of each project to have periodic joint feedback, but it appears simpler for them to have a focal point to contact in case of anything at the coordination level.

### 3.3.5. Continuous experimentation

#### Get to know your community and its needs

In order to map the community profiles and motivations, we conducted an initial survey on new registered participants asking for their background, skills, and location. We then sent a
weekly form to project leads to collect updates on their project status and needs. This was streamlined by having a "needs" form on the JOGL platform.

## Design and implement a continuous monitoring and evaluation strategy

In order to adapt the organization strategy, you need to be able to "know your community": its activity, diversity, collaboration practices, outputs and roadblocks. To collect insights, it is important to implement a continuous **evaluation framework**. Evaluation is the action of systematic process to judge merit, worth or significance by combining evidence and values (Home | Better Evaluation, n.d.), and is sometimes referred to as project review, assessment, impact analysis, social impact analysis, appreciative inquiry, or cost-benefit assessment. **Monitoring the activity, needs, and interactions within your community** allows you to collect data for your evaluation. This serves as a check of whether you are on track to reach your goals and take corrective actions or adapt and adjust your objectives. A "Monitoring, Evaluation, Accountability and Learning (MEAL) Officer" is now a standard position within the NGO sector.

Evaluation methodologies can be found on https://www.betterevaluation.org/. Several tools can be useful for monitoring and evaluation. Survey tools encompass Redcap, Google Forms, or Microsoft Access. Using Slack as a main communication channel (or other tools with an API) helps in tracking the communication activity. Yet, digital traces don't encompass the full reality of social interactions, and direct feedback from community members is often more informative when you wish to assess dynamics of collaboration or touch on power-related issues. In our case, we implemented a MEAL working group at OpenCovid19 that conducted semi-structured interviews on lessons learnt as well as satisfaction interviews. We produced an internal report to the coordination team which led to corrective actions, and a Community Playbook for dissemination to the external audience (Mains & JOGL community, 2020). It is to note that depending on the nature of your community (volunteer or not) the corrective actions you take will not be the same.

## Design a feedback and moderation mechanism

Feedbacks can range from desires of the community members for improvement, roadblocks to the implementation of certain actions, or reports of harassment or discrimination. It is important to have different reporting methods in order to ensure that people can provide such feedback irrespective of their physical or mental disabilities, language, gender, etc. For this, it is important to communicate on the moderation process: how are reports handled? With which delay should one expect an answer? Identifying an "ethic" specialist or board can be useful, as well as providing moderation mechanisms in the public code of conduct, and clarifying the consequences. Finally, the moderation team should systematically acknowledge receipt of a report.
## 3.4. Open knowledge exchange

### 3.4.1. Multidisciplinary knowledge integration

Recent work in the science of team science has demonstrated that teams showing diverse individuals with more equal conversational input displayed higher ability to solve complex problems, a property known as "collective intelligence" (Woolley et al., 2010). Within OpenCovid19, we did not have any selection criteria on the background and did not organize the teams or projects based on diversity. Rather we tried to match the right profiles to projects based on the expressed needs.

### Invest in interpersonal connections and create bonds

Interpersonal connection is the top reason why people stay engaged in an initiative (Kraut et al., 2011). Yet the serendipitous encounter at the coffee machine isn't possible online. Thus, it must be architected by the community organizers. In our case, a set of people held the role of coordinators. Participants could go to them for any reason: they knew where to find the information. This helped prevent individuals from reinventing the wheel or to disengage because of being lost in the large quantity of information. Coordinators also helped by matching individuals to other relevant individuals. Connecting directly to individuals and not to a group helped craft a personalized introduction and lower the energy barrier to create a bond.

### 3.4.2. Engaging in exchanging efforts and not committing directly to the entire project

The coordination team attempted to create synergies and collaborations instead of promoting separate ideas in order to reduce the overall effort. At the triaging level, coordinators encouraged people who joined to identify projects where their skills match. During the on-boarding process, the coordination team would specifically link people to project leads and provide an updated status of on-going projects and needs on JOGL.

In addition, we organized weekly global community calls to synchronize the community where all newcomers were invited. Projects would quickly present their status, updates and needs. As a consequence, other projects would exchange knowledge and practice. We also organized longer "Open JOGL" online sessions addressing the specific needs of a project.

### 3.4.3. Facilitated and open translation/appropriation of the results

Appropriation of the results allows for empowerment of the community members. Because they gain knowledge, or skills, it allows them to reflect differently on their practice. Learning from failures is equally important, and we kept an excel spreadsheet of failures, or struggles.
References

Angelidis, P., Berman, L., Casas-Perez, M. de la L., Celi, L. A., Dafoulas, G. E., Dagan, A., Escobar, B., Lopez, D. M., Noguez, J., Osorio-Valencia, J. S., Otine, C., Paik, K., Rojas-Potosi, L., Symeonidis, A. L., & Winkler, E. (2016). The hackathon model to spur innovation around global mHealth. *Journal of Medical Engineering & Technology, 40*(7-8), 392-399. https://doi.org/10.1080/03091902.2016.1213903

Are Civic Hackathons Stupid? (2013, July 5). *Bloomberg.COM*.
https://www.bloomberg.com/news/articles/2013-07-05/are-civic-hackathons-stupid

Auffray, C., Balling, R., Blomberg, N., Bonaldo, M. C., Boutron, B., Brahmachari, S., Bréchot, C., Cesario, A., Chen, S.-J., Clément, K., Danilenko, D., Di Meglio, A., Gelemanović, A., Goble, C., Gojobori, T., Goldman, J. D., Goldman, M., Guo, Y.-K., Heath, J., ...
Wouters, E. (2020). COVID-19 and beyond: A call for action and audacious solidarity to all the citizens and nations, it is humanity's fight. *F1000Research, 9*, 1130.
https://doi.org/10.12688/f1000research.26098.1

Barkoczi, D., & Galesic, M. (2016). Social learning strategies modify the effect of network structure on group performance. *Nature Communications, 7*(1), 13109.
https://doi.org/10.1038/ncomms13109

Benchoufi, M., Fournier, M., Magrez, D., Macaux, G., Barué, V., Mansilla Sanchez, A., de Fresnoye, O., Fillaudeau, R., Tauvel-Mocquet, O., Chalabi, N., Petit-Nivard, J. F., Blondel, L., Santolini, M., & Ben Hadj Yahia, B. (2018a). Epidemium: A multidisciplinary community to tackle cancer using big and open data. *Journal of Clinical Oncology, 36*(15_suppl), e13604-e13604.
https://doi.org/10.1200/JCO.2018.36.15_suppl.e13604

Benchoufi, M., Fournier, M., Magrez, D., Macaux, G., Barué, V., Mansilla Sanchez, A., de Fresnoye, O., Fillaudeau, R., Tauvel-Mocquet, O., Chalabi, N., Petit-Nivard, J. F.,
Blondel, L., Santolini, M., & Ben Hadj Yahia, B. (2018b). *Epidemium: A multidisciplinary community to tackle cancer using big and open data*. *Journal of Clinical Oncology*, 36(15_suppl), e13604–e13604.

https://doi.org/10.1200/JCO.2018.36.15_suppl.e13604

Bosman, J., Bruno, I., Chapman, C., Greshake Tzovaras, B., Jacobs, N., Kramer, B., Martone, M., Murphy, F. L. M., O'Donnell, D. P., Bar-Sinai, M., Hagstrom, S., Utley, J. A., & Veksler, L. (2017). *The Scholarly Commons—Principles and practices to guide research communication* [Preprint]. Open Science Framework.

https://doi.org/10.31219/osf.io/6c2xt

Burgelman, J.-C., Pascu, C., Szkuta, K., Von Schomberg, R., Karalopoulos, A., Repanas, K., & Schouppe, M. (2019). *Open Science, Open Data, and Open Scholarship: European Policies to Make Science Fit for the Twenty-First Century*. *Frontiers in Big Data*, 2, 43. https://doi.org/10.3389/fdata.2019.00043

Chambers, J. M., Wyborn, C., Ryan, M. E., Reid, R. S., Riechers, M., Serban, A., Bennett, N. J., Cvitanovic, C., Fernández-Giménez, M. E., Galvin, K. A., Goldstein, B. E., Klenk, N. L., Tengö, M., Brennan, R., Cockburn, J. J., Hill, R., Munera, C., Nel, J. L., Österblom, H., ... Pickering, T. (2021). Six modes of co-production for sustainability.

*Nature Sustainability*, 1–14. https://doi.org/10.1038/s41893-021-00755-x

Costanza-Chock, S. (2020). *Design justice: Community-led practices to build the worlds we need*. The MIT Press.

Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case study approach. *BMC Medical Research Methodology*, 11(1), 100.

https://doi.org/10.1186/1471-2288-11-100

Day, K., Humphrey, G., & Cockcroft, S. (2017). How do the design features of health hackathons contribute to participatory medicine? *Australasian Journal of Information Systems*, 21. https://doi.org/10.3127/ajis.v21i0.1383
DePasse, J. W., Carroll, R., Ippolito, A., Yost, A., Santorino, D., Chu, Z., & Olson, K. R.

(2014). Less noise, more hacking: How to deploy principles from MIT's hacking medicine to accelerate health care. *International Journal of Technology Assessment in Health Care, 30*(3), 260–264. https://doi.org/10.1017/S0266462314000324

Dickel, S., & Franzen, M. (2016). The “Problem of Extension” revisited: New modes of digital participation in science. *Journal of Science Communication, 15*(1), A06.

https://doi.org/10.22323/2.15010206

D'Ignazio, C., & Klein, L. F. (2020). *Data feminism*. The MIT Press.

Elliott, K. C., & Resnik, D. B. (2019). Making Open Science Work for Science and Society.

*Environmental Health Perspectives, 127*(7), 075002.

https://doi.org/10.1289/EHP4808

Fraisl, D., Campbell, J., See, L., Wehn, U., Wardlaw, J., Gold, M., Moorthy, I., Arias, R.,

Piera, J., Oliver, J. L., Masó, J., Penker, M., & Fritz, S. (2020). Mapping citizen science contributions to the UN sustainable development goals. *Sustainability Science*. https://doi.org/10.1007/s11625-020-00833-7

Franzoni, C., & Sauermann, H. (2012). Crowd Science: The Organization of Scientific

Research in Open Collaborative Projects. SSRN Electronic Journal.

https://doi.org/10.2139/ssrn.2167538

Franzoni, C., & Sauermann, H. (2014). Crowd science: The organization of scientific

research in open collaborative projects. *Research Policy, 43*(1), 1–20.

https://doi.org/10.1016/j.respol.2013.07.005

Guimerà, R., Uzzi, B., Spiro, J., & Nunes Amaral, L. A. (2005). Team Assembly Mechanisms

Determine Collaboration Network Structure and Team Performance. *Science (New York, N.Y.)*, 308(5722), 697–702. https://doi.org/10.1126/science.1106340

Hecker, S., Haklay, M., Bowser, A., Makuch, Z., Vogel, J., & Bonn, A. (2018). *Citizen science*

innovation in open science, society and policy. UCL Press.
Home | Better Evaluation. (n.d.). Retrieved December 22, 2021, from
https://www.betterevaluation.org/en

Juan, A. A., Daradoumis, T., Roca, M., Grasman, S. E., & Faulin, J. (Eds.). (2012).
*Collaborative and Distributed E-Research: Innovations in Technologies, Strategies
and Applications*. IGI Global. https://doi.org/10.4018/978-1-4666-0125-3

Katapally, T. R. (2019). The SMART Framework: Integration of Citizen Science, Community-Based Participatory Research, and Systems Science for Population Health Science in the Digital Age. *JMIR MHealth and UHealth, 7*(8), e14056.
https://doi.org/10.2196/14056

Kienzler, H., & Fontanesi, C. (2017). Learning through inquiry: A Global Health Hackathon.
*Teaching in Higher Education, 22*(2), 129–142.
https://doi.org/10.1080/13562517.2016.1221805

Klein, J. T. (2008). Evaluation of interdisciplinary and transdisciplinary research: A literature review. *American Journal of Preventive Medicine, 35*(2), S116–S123.

Klug, M., & Bagrow, J. P. (2016). Understanding the group dynamics and success of teams.
*Royal Society Open Science, 3*(4), 160007. https://doi.org/10.1098/rsos.160007

Kokshagina, O. (2021). Open Covid-19: Organizing an extreme crowdsourcing campaign to tackle grand challenges. *R&D Management, radm*.12470.
https://doi.org/10.1111/radm.12470

Kraut, R. E., Resnick, P., & Kiesler, S. (2011). *Building successful online communities: Evidence-based social design*. MIT Press.

Latour, B., & Woolgar, S. (1986). *Laboratory Life: The Construction of Scientific Facts*. Princeton University Press. https://doi.org/10.2307/j.ctt32bbxc

Leavy, P. (2011). *Essentials of transdisciplinary research: Using problem-centered methodologies*. Left Coast Press.

Lehmann, S. (2019). Fundamental Structures in Dynamic Communication Networks.
ArXiv:1907.09966 [Physics]. http://arxiv.org/abs/1907.09966

Liao, C. H. (2011). How to improve research quality? Examining the impacts of collaboration intensity and member diversity in collaboration networks. *Scientometrics*, 86(3), 747–761. https://doi.org/10.1007/s11192-010-0309-2

Mains, A., & JOGL community. (2020). *How to build and manage open communities* (p. 23).
https://jogl-prod.s3.eu-west-3.amazonaws.com/sn78h6wbfmekq6bqbnjdiyig8gtc?response-content-disposition=inline%3B%20filename%3D%22JOGL%20playbook%20-%20updated%2021.15.2020.pdf%22%3B%20filename%2A%3DUTF-8%27%27JOGL%2520playbook%2520-%2520updated%252021.15.2020.pdf&response-content-type=application%2Fpdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVMSP7HGJPWU56E7B%2F20211222%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Date=20211222T141134Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&t&X-Amz-Signature=e4937fb9b755720601f7d00442d26136d91db5090508c59af2e936d30378f1cc

Masselot, C. M., Greshake Tzovaras, B., Graham, C. L. B., Jeyaram, R., Finnegan, G., Vitali, I., Landrain, T. E., & Santolini, M. (2021). Co-Immune: A case study on open innovation for vaccination hesitancy and access. *MedRxiv*, 2021.03.29.20248781.
https://doi.org/10.1101/2021.03.29.20248781

Nielsen, M. A. (2014). *Reinventing discovery: The new era of networked science*.

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., ... Yarkoni, T. (2015). Promoting an open research culture. *Science*, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374

*Open innovation, open science, open to the world: A vision for Europe*. (2016). Publications Office. https://data.europa.eu/doi/10.2777/061652
Owen-Smith, J. (2001). Managing Laboratory Work through Skepticism: Processes of Evaluation and Control. *American Sociological Review*, 66(3), 427–452.

https://doi.org/10.2307/3088887

Palla, G., Barabási, A.-L., & Vicsek, T. (2007). Quantifying social group evolution. *Nature*, 446(7136), 664–667. https://doi.org/10.1038/nature05670

Palla, G., Derényi, I., Farkas, I., & Vicsek, T. (2005). Uncovering the overlapping community structure of complex networks in nature and society. *Nature*, 435(7043), 814–818.

https://doi.org/10.1038/nature03607

Pan, S. W., Stein, G., Bayus, B., Tang, W., Mathews, A., Wang, C., Wei, C., & Tucker, J. D. (2017). Systematic review of innovation design contests for health: Spurring innovation and mass engagement. *BMJ Innovations*, 3(4), 227–237.

https://doi.org/10.1136/bmjinnov-2017-000203

*Pan-European Matchathon*. (n.d.). Retrieved October 8, 2020, from
https://www.euvsvirus.org/

Polk, M. (2015). Transdisciplinary co-production: Designing and testing a transdisciplinary research framework for societal problem solving. *Futures*, 65, 110–122.

https://doi.org/10.1016/j.futures.2014.11.001

Puranam, P., Alexy, O., & Reitzig, M. (2014). What’s “New” About New Forms of Organizing?
*Academy of Management Review*, 39(2), 162–180.

https://doi.org/10.5465/amr.2011.0436

Ramatowski, J. W., Lee, C. X., Mantzavino, A., Ribas, J., Guerra, W., Preston, N. D.,
Schernhammer, E., Madoff, L. C., & Lassmann, B. (2017). Planning an innovation
marathon at an infectious disease conference with results from the International

Meeting on Emerging Diseases and Surveillance 2016 Hackathon. *International Journal of Infectious Diseases*, 65, 93–97. https://doi.org/10.1016/j.ijid.2017.09.025

Ren, Y., Kraut, R., & Kiesler, S. (2007). Applying Common Identity and Bond Theory to
Design of Online Communities. *Organization Studies*, 28(3), 377–408.

https://doi.org/10.1177/0170840607076007

Riedl, C., & Woolley, A. W. (2016). Teams vs. Crowds: A Field Test of the Relative Contribution of Incentives, Member Ability, and Emergent Collaboration to Crowd-Based Problem Solving Performance. *Academy of Management Discoveries*, 3(4), 382–403. https://doi.org/10.5465/amd.2015.0097

Rowbotham, S., McKinnon, M., Leach, J., Lamberts, R., & Hawe, P. (2019). Does citizen science have the capacity to transform population health science? *Critical Public Health*, 29(1), 118–128. https://doi.org/10.1080/09581596.2017.1395393

Scharnhorst, A., Börner, K., & van den Besselaar, P. (Eds.). (2012). *Models of Science Dynamics: Encounters Between Complexity Theory and Information Sciences*. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-23068-4

Stokols, D. (2006). Toward a Science of Transdisciplinary Action Research. *American Journal of Community Psychology*, 38(1–2), 79–93.
https://doi.org/10.1007/s10464-006-9060-5

Uzzi, B., & Spiro, J. (2005). Collaboration and Creativity: The Small World Problem.
*American Journal of Sociology*, 111(2), 447–504. https://doi.org/10.1086/432782

Uzzi, B., Wuchty, S., Spiro, J., & Jones, B. F. (2012). Scientific teams and networks change the face of knowledge creation. In B. Vedres & M. Scotti (Eds.), *Networks in Social Policy Problems* (pp. 47–59). Cambridge University Press.
https://doi.org/10.1017/CBO9780511842481.003

Vayena, E., Brownsword, R., Edwards, S. J., Greshake, B., Kahn, J. P., Ladher, N., Montgomery, J., O'Connor, D., O'Neill, O., Richards, M. P., Rid, A., Sheehan, M., Wicks, P., & Tasioulas, J. (2016). Research led by participants: A new social contract for a new kind of research. *Journal of Medical Ethics*, 42(4), 216–219.
https://doi.org/10.1136/medethics-2015-102663
#WirVsVirus—#WirVSVirus. (n.d.). Retrieved October 8, 2020, from https://wirvsvirus.org/

Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a Collective Intelligence Factor in the Performance of Human Groups. *Science*, 330(6004), 686–688. https://doi.org/10.1126/science.1193147

World Health Organization. (2021). *Voice, agency, empowerment – handbook on social participation for universal health coverage*. World Health Organization.

https://apps.who.int/iris/handle/10665/342704

Wu, L., Wang, D., & Evans, J. A. (2019). Large teams develop and small teams disrupt science and technology. *Nature*, 566(7744), 378–382.

https://doi.org/10.1038/s41586-019-0941-9
Supplementaries

<table><caption>Table 3. Data structure</caption><thead><tr><th>First-order categories</th><th>Second-order themes</th><th>Aggregated dimensions</th></tr></thead><tbody><tr><td><ul><li>Possibility to join any time: “a program to collectively develop open-source and low-cost tools and methodologies that are safe and easy to use to fight the COVID-19 pandemic. Anyone can play a role”; “We can all contribute”</li><li>“Community members can join a team and contribute to building solutions and coordinate the initiatives as a member of any of the committees, such as governance, operations, communications, onboarding”</li><li>Onboarding page with a summary of projects, needs, members; “Get started” and “Learn more” latest announcements; Overview of active challenges</li><li>Members’ page with a possibility to “Follow” any of the community members with an overview of their skills, resources, interest in particular Sustainable Development Goals (SDGs) and their involvement in projects</li><li>Praising other members with “Clap to any message”</li><li>Triage system to help participants navigate through the program</li><li>Possibility to share specific needs on the platform; indicate skills that participants can donate and resources (i.e., “I need data collection skills to answer a few questions on this project. If you think you can help in this area!”)</li><li>Global Community Calls each Wednesday to update the community and share direction (thus far, 21 calls were organized and recorded)</li><li>Shared agenda for call: Google Doc where participants can add items they want to discuss during the community calls</li><li>Access recordings of all community calls (even for non-members)</li><li>Tracking task completions</li><li>Monitoring Slack channels</li><li>An onboarding Google Doc with the outline of the goal, processes (continuously updated by everyone)—living memory of the project</li><li>Structured FAQs on the platform website</li><li>Guidelines with the introduction to the platform and various tools and resources, tips on how to use different Slack channels, and what their purpose is</li><li>A regular snapshot of all activities by the coordination team</li><li>Possibility to see how expressed needs match your skills</li><li>Share the need or click on “I’ll help”</li><li>A member profile with an overview of activities and projects involved</li><li>Respect the bioethical and scientific guidelines of good conduct</li><li>Coordination team monitored the evolution and the program’s overall progress</li><li>Shared Google Docs for things that need to be improved</li><li>Trello to monitor task completion for the coordination team</li><li>Participants to become different committees’ members if interested</li></ul></td><td><ul><li>Platform to facilitate onboarding and orientation of voluntary participants—participative architecture</li><li>Continuously embedding and improving predictability for participants</li><li>Collective building of common understanding</li><li>Gradual self-accountability</li><li>Continuous experimentation</li></ul></td><td><ul><li>Emergent coordination</li></ul></td></tr></tbody></table>
<table><caption>Table 3. (Continued)</caption><thead><tr><th>First-order categories</th><th>Second-order themes</th><th>Aggregated dimensions</th></tr></thead><tbody><tr><td><ul><li>Seeking a variety of different skills: Particip978-pants are supported to use their differences to articulate complex problems using needs for the projects, resources required, and skills they can donate</li><li>Global community with participants from many different countries</li><li>Possibility to share specific needs on the platform</li><li>Exchange skills, needs, resources rather than final products</li><li>Encourage to explore ideas posted by others (commenting on another person's idea; joining the teams)</li><li>Create synergies instead of promoting separate ideas</li><li>An open lab notebook to capitalize on the knowledge produced and shared— living memory of the project</li><li>Each project is documented on the page indicating the problem and the background; the solution summary in simple and technical terms; the state of advancement of the project, the project timeline, the project implementation, a list of objectives, methodology, expected results; safety, quality assurance, and regulation; impact, issues and risks; a list of participating labs and their team members; funding</li><li>"Documents" tab with additional resources</li><li>Launching JOGL live initiative—1 hour discussion around one project using Zoom, and recorded</li><li>"Wikipedia of R&D"</li><li>Involving potential users of the solution at the start, i.e., the team created a partnership with the Paris Hospital Network (APHP) representing 39 hospitals in the Paris Region to evaluate and validate different solutions</li><li>Micro-grants available for each project</li><li>Possibility to participate in other Covid-19 challenges like XPRIZE</li></ul></td><td><ul><li>Multidisciplinary knowledge integration</li><li>Engaging in exchanging efforts and not committing directly to the entire project</li><li>Extreme transparency in knowledge produced and knowledge capitalization</li></ul></td><td><ul><li>Open knowledge exchange</li></ul></td></tr></tbody></table>

Other annexes can be found in this folder: https://drive.google.com/drive/folders/1Agu8XjpK_4vLidq9q-nRkjX2uZcEkw0W?usp=sharing