# The benefits to climate science of including early-career scientists as reviewers

Mathieu Casado, Gwenaëlle Gremion, Paul Rosenbaum, Jilda Alicia Caccavo, Kelsey Aho, Nicolas Champollion, Sarah L Connors, Adrian Dahood, Alfonso Fernandez, Martine Lizotte, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mathieu Casado, Gwenaëlle Gremion, Paul Rosenbaum, Jilda Alicia Caccavo, Kelsey Aho, et al. The benefits to climate science of including early-career scientists as reviewers. Geoscience Communication, 2020, 3, pp.89 - 97. 10.5194/gc-3-89-2020. hal-04954012

HAL Id: hal-04954012

https://hal.science/hal-04954012v1

Submitted on 18 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

# The benefits to climate science of including early-career scientists as reviewers

Mathieu Casado<sup>1,2</sup>, Gwenaëlle Gremion<sup>2,3</sup>, Paul Rosenbaum<sup>2,4</sup>, Jilda Alicia Caccavo<sup>2,5,6,7</sup>, Kelsey Aho<sup>2,8</sup>, Nicolas Champollion<sup>2,9</sup>, Sarah L. Connors<sup>10</sup>, Adrian Dahood<sup>2,11,12,†</sup>, Alfonso Fernandez<sup>2,13</sup>, Martine Lizotte<sup>2,14</sup>, Katja Mintenbeck<sup>5</sup>, Elvira Poloczanska<sup>5</sup>, and Gerlis Fugmann<sup>1,2</sup>

**Correspondence:** Mathieu Casado (mathieu.casado@gmail.com)

Received: 1 October 2019 – Discussion started: 10 October 2019

Accepted: 11 March 2020 – Published: 21 April 2020

**Abstract.** Early-career scientists (ECSs) are a large part of the workforce in science. While they produce new scientific knowledge that they share in publications, they are rarely invited to participate in the peer-review process. Barriers to the participation of ECSs as peer reviewers include, among other things, their lack of visibility to editors, inexperience in the review process and lack of confidence in their scientific knowledge. Participation of ECSs in group reviews, e.g. for regional or global assessment reports, provides an opportunity for ECSs to advance their skill set and to contribute to policy-relevant products. Here, we present the outcomes of a group peer review of the First Order Draft of the Intergovernmental Panel on Climate Change (IPCC) Special Report on

the Ocean and Cryosphere in a Changing Climate (SROCC). Overall, PhD students spent more time on the review than those further advanced in their careers and provided a similar proportion of substantive comments. After the review, participants reported feeling more confident in their skills, and 86% were interested in reviewing individually. By soliciting and including ECSs in the peer-review process, the scientific community would not only reduce the burden carried by more established scientists but also permit their successors to develop important professional skills relevant to advancing climate science and influencing policy.

<footnote n="1"><sup>1</sup> Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research, Telegrafenberg A45-2N, 14473 Potsdam, Germany</footnote>

<footnote n="2"><sup>2</sup> Association of Polar Early Career Scientists (APECS), Telegrafenberg A5, 14473 Potsdam, Germany</footnote>

<footnote n="3"><sup>3</sup> Institut des Sciences de la Mer, Université du Québec à Rimouski, Rimouski, Quebec, G5L3A1, Canada</footnote>

<footnote n="4"><sup>4</sup> Department of Business Studies, Uppsala University, Uppsala, 75313, Sweden</footnote>

<footnote n="5"><sup>5</sup> Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research, Am Handelshafen 12, 27568 Bremerhaven, Germany</footnote>

<footnote n="6"><sup>6</sup> Berlin Center for Genomics in Biodiversity Research, Königin-Luise-Str. 6–8, 14195 Berlin, Germany</footnote>

<footnote n="7"><sup>7</sup> Department of Evolutionary Genetics, Leibniz Institute for Zoo and Wildlife Research, Alfred-Kowalke-Straße 17, 10315 Berlin, Germany</footnote>

<footnote n="8"><sup>8</sup> International Arctic Research Center, University of Alaska Fairbanks, P.O. Box 757340 Fairbanks, AK 99775, USA</footnote>

<footnote n="9"><sup>9</sup> Climate Lab, University of Bremen, 28359 Bremen, Germany</footnote>

<footnote n="10"><sup>10</sup> IPCC Working Group I Technical Support Unit, Université Paris-Saclay, 91190 Saint-Aubin, France</footnote>

<footnote n="11"><sup>11</sup> Institute of Marine Sciences, University of California, Santa Cruz, Santa Cruz, CA 95064, USA</footnote>

<footnote n="12"><sup>12</sup> Antarctic Ecosystem Research Division, Southwest Fisheries Science Center, National Marine Fisheries Service, National Oceanic and Atmospheric Administration, 8901 La Jolla Shores Drive, La Jolla, CA 92037-1023, USA</footnote>

<footnote n="13"><sup>13</sup> Department of Geography, Universidad de Concepción, 4030000, Concepción, Chile</footnote>

<footnote n="14"><sup>14</sup> Department of Biology, Université Laval, Quebec City, Quebec, G1V0A6, Canada
†deceased</footnote>
# 1 Introduction

Acting as peer reviewers is an important opportunity for scientists to recognise the components of strong scientific papers, and it can help to improve the quality of their own work (Silver, 2016; Lerback and Hanson, 2017). While several training opportunities exist for early-career scientists (ECSs; also referred to as early-career researchers – ECRs) to bolster their voice and develop writing skills and leadership (Geffers et al., 2017), only 6% of journal articles are reviewed by ECSs (Taylor and Francis, 2016).

Global assessment reports rely on reviewers from multiple regions and diverse scientific disciplines to ensure that they are scientifically accurate and are widely understandable. Here, we focus on a report produced by the Intergovernmental Panel on Climate Change (IPCC) that provides an assessment of the scientific, technical and socio-economic literature on the current state of knowledge on climate change (IPCC, 2013a). Ensuring effective climate change adaptation and mitigation requires policymakers to be informed by the scientific community through robust and evidence-based reports reflecting the scientific consensus (Bolin, 2007; Tollefson, 2010; Ding et al., 2011; Lewandowsky et al., 2013). To best achieve this goal, the scientific community requires the inclusion of scientists from heterogeneous backgrounds and experiences (Maibach et al., 2014; Hallegatte et al., 2016), including ECSs, who can provide diverse perspectives. Each IPCC report undergoes a multistage review process by expert and government representatives (IPCC, 2013b). For example, the Working Group I *Fifth Assessment Report (WGI AR5)* attracted comments from 1089 expert reviewers from 55 countries. While no individual possesses the required expertise to review an entire IPCC report, as a group, ECSs have also proven to be efficient and motivated reviewers, providing added value to this type of manuscript as shown by van der Veer et al. (2014), who pioneered the crowdsourcing and organisation of groups of ECSs to participate in the IPCC report review process. By serving as reviewers, ECSs support the IPCC assessment process and learn more about interdisciplinary endeavours while developing new skills for synthesising their own research into the limited scope of a publishable paper.

Through the Association of Polar Early Career Scientists (APECS), ECSs were recruited to participate in a group review of the First Order Draft (FOD) of the IPCC Special Report on the Ocean and Cryosphere in a Changing Climate (SROCC). APECS is an international and interdisciplinary network for undergraduate and graduate students, postdoctoral researchers (postdocs), early-career professionals, educators, and others interested in polar and alpine regions and the wider cryosphere (Allen et al., 2014). While most ECSs in APECS are PhD students or postdocs, membership also includes master students, undergraduate students, early-career academics and other educators. APECS thus organised a group review of the SROCC which included 75 ECSs from

22 of the countries within the APECS network (Fig. 1a). The pool of reviewers was composed of 38 women and 37 men. Earth sciences were most represented among the reviewers, while other disciplines included biology, geography, social sciences and civil engineering (Fig. 1b).

Recognising that some of the reviewers had neither published a paper nor participated in peer review before, APECS designed a comprehensive training programme, providing ECSs with a rigorous reviewing frame, ensuring that the reviews produced would all be of equal quality. Here, we report the first results of this programme and highlight that the competence of a reviewer is not influenced by career level. We first describe the methods, how the group review was organised and the statistics applied for this study. We then make use of the statistics and of the response to a post-review survey to explore the number of comments by career level. Finally, we conclude with a list of propositions for editors to encourage the inclusion of ECSs in review processes.

# 2 Methods

## 2.1 IPCC review process

The drafts of the IPCC reports undergo two rounds of review which have an open call to the scientific community to participate as expert reviewers. Here, for the SROCC, we focus on the first of these, the expert review of the FOD. The expert review process is open, transparent and objective, with wide participation involving independent experts from a range of scientific disciplines, regions and countries. Expert reviewers self-nominate through a registration process, and, after being approved and signing a confidentiality agreement, expert reviewers receive an individually watermarked draft of chapters and a pre-formatted spreadsheet for entering review comments. The individual expert reviewer spreadsheets are submitted to the Technical Support Unit (TSU), which combines and sends all the review comments to report authors for consideration and response. All comments are assigned to an expert reviewer, and the report authors provide a written response to each review comment. For the SROCC, a process was also put in place for group reviews, where multiple reviewers work as a team, with a single spreadsheet being delivered by the group and attributed to the group name (e.g. APECS) and list of all expert reviewers participating in the group review submitted to the Working Group II TSU. The names of expert reviewers, their comments and IPCC author responses to the comments are freely available once the report is released after its approval by member governments (for the SROCC on 25 September 2019).

For the SROCC, comments could be entered under two categories “editorial” and “substantive”. Editorial comments are those related to language, sentence structure and formatting. Substantive comments are those focused on content and structure of the report and chapters, concepts, literature assessed, confidence language, conclusions, gaps, overlaps,
and the use of and content within figures and tables. Expert reviewers are encouraged to focus on the latter, as the final report undergoes professional copy editing before publication. Expert reviewers can choose to review the whole report, chapter(s) within the report or section(s) of chapters most relevant to their scientific expertise.

## 2.2 Organisation and recruitment of the APECS group review

The project was led by 10 council members of APECS from various academic levels: master (1), PhD (4), postdoc (3) and early-career academics (2), representing six different countries (Canada, Chile, France, Germany, Italy and the USA). Each chapter was chaired by two to five of these project leaders, whose roles were (a) selecting the participants, (b) assigning chapter sections to the participants, (c) reviewing and sorting the comments (e.g. finding duplicates), and (d) finalising the global review sent by APECS.

The call for participants, published on the APECS website, requested information about applicants' credentials and motivations for participating in the group review and received 153 applications. Among the applicants, 72% had already reviewed a scientific document (such as a paper, a proposal or a scientific report). Not all chapters received the same number of applicants (the outline of the report can be found at https://www.ipcc.ch/site/assets/uploads/2018/11/Decision_Outline_SR_Oceans.pdf, last access: 9 April 2020). For instance, the chapter on "Polar Regions" (estimated at 50 pages) was requested by 110 applicants, while the chapter on "Changing Ocean, Marine Ecosystems, and Dependent Communities" (estimated at 65 pages) was only requested by 33 applicants.

As the number of applications from less experienced scientists (PhD students and below) was roughly equal to the number of postdocs and early-career academics, it was decided to review the applications separately, opening the same number of positions for both categories. We used the applications to evaluate (1) motivation, (2) experience and relevance of the application, (3) country of residence, and (4) equal distribution amongst chapters, meaning that applicants who applied to more than one chapter were often assigned to a secondary choice.

The resulting selection of participants in the review process strongly reflects the established network of APECS members and mailing list recipients (Table 1). While gender was not a criterion for selection, 51% of the participants were women. Only 5 of the 22 countries represented are developing countries or economies in transition. The selected participants included a majority of PhD students, a similar number of postdocs and early-career academics, and only two undergraduate and master students (Table 2).

The horizontal structure of this group review (all participants are ECSs self-organising to realise this review) strongly differs from the previous group review attempts of an IPCC

<table><caption>**Table 1.** Number of participants and their countries of origin (including the project leaders); see also Fig. 1a.</caption><thead><tr><th>Countries</th><th>Number</th><th>Countries</th><th>Number</th></tr></thead><tbody><tr><td>Australia</td><td>5</td><td>Italy</td><td>2</td></tr><tr><td>Belgium</td><td>1</td><td>Mexico</td><td>1</td></tr><tr><td>Brazil</td><td>1</td><td>New Zealand</td><td>3</td></tr><tr><td>Canada</td><td>8</td><td>Norway</td><td>3</td></tr><tr><td>Chile</td><td>3</td><td>Poland</td><td>1</td></tr><tr><td>China</td><td>1</td><td>Spain</td><td>1</td></tr><tr><td>Denmark</td><td>1</td><td>Sweden</td><td>2</td></tr><tr><td>France</td><td>4</td><td>Switzerland</td><td>1</td></tr><tr><td>Germany</td><td>10</td><td>The Netherlands</td><td>2</td></tr><tr><td>Iceland</td><td>1</td><td>UK</td><td>10</td></tr><tr><td>India</td><td>3</td><td>USA</td><td>11</td></tr></tbody><tfoot><tr><td>Total</td><td></td><td></td><td>75</td></tr></tfoot></table>

<table><caption>**Table 2.** List of academic levels of the participants (including the project leaders).</caption><thead><tr><th>Level</th><th>Number</th></tr></thead><tbody><tr><td>Undergraduate and master</td><td>2</td></tr><tr><td>PhD</td><td>31</td></tr><tr><td>Postdoc</td><td>23</td></tr><tr><td>Early-career academics</td><td>19</td></tr></tbody><tfoot><tr><td>Total</td><td>75</td></tr></tfoot></table>

review project, for which a government was organising the review and providing incentives to participants to participate in such review (van der Veer et al., 2014).

## 2.3 Training of the ECS reviewers

In order to introduce the review process to participants, a guide was created by the project leaders from APECS, validated by the APECS executive director and reviewed by members of the IPCC TSUs from the two working groups (WGs), providing scientific leadership to the SROCC (WGI: The Physical Science Basis; WGII: Impacts, Adaptation and Vulnerability). This guide explained the process including the objectives, timeline, leadership and rules. The guide is available in the Supplement.

Two series of online workshops were organised to train the participants in the review process. To facilitate participation from numerous time zones, two sessions were organised for each workshop series at different times, and recordings were made available for later viewing (session 1: https://vimeo.com/292679338, last access: 9 April 2020, session 2.1: https://vimeo.com/292679417, last access: 9 April 2020; session 2.2: https://vimeo.com/292679451, last access: 9 April 2020). Each online workshop lasted between 1.5 and 2 h.
The first series of workshops was conducted by mem-
bers of the IPCC TSUs (WGI: Sarah Connors; WGII:
Katja Mintenbeck and Elvira Polozczanska), who introduced
the IPCC and its processes, discussed the purpose and
government-approved structure of the SROCC, and ad-
vised participants on what entails a constructive review.
The second workshop involved vice-chairs of WGI and
WGII and was held after the distribution of the respec-
tive chapters and sections to the participants. Greg Flato
(Canada, WGI vice-chair, workshop session 1) and An-
dreas Fischlin (Switzerland, WGII vice-chair, workshop
session 2) discussed the use of the IPCC uncertainty
language (https://www.ipcc.ch/site/assets/uploads/2018/05/
uncertainty-guidance-note.pdf, last access: 9 April 2020),
which is a key component of IPCC assessments. Other advice
for reviewers was provided by Carlos Mendez (Venezuela,
WGII vice-chair) and Jan Fuglestvedt (Norway, WGI vice-
chair) for the first and second session, respectively. All doc-
uments shared during these workshops (i.e. PowerPoint pre-
sentation or minutes) were made available to the participants
for later viewing. Furthermore, the guide was amended dur-
ing the training process to include updated information.

## 2.4 Organisation of the APECS group review

The review of the FOD of the SROCC was done as a group
review, meaning that each participant of the group was only
in charge of reviewing a small section of the report and that
all the comments were then combined and submitted together
under the flag of APECS. Each chapter was distributed by the
project leaders to the participants. Depending on the number
of participants for each chapter, 10 to 20 pages were assigned
to each participant to review. In addition, the entire chapter
was made available to participants so that they could provide
additional comments outside of the assigned pages and be
able to put their assigned section into context. We attempted
to assign whole sections as much as possible. We also at-
tempted to balance the workload and in some instances, re-
viewers were assigned non-contiguous sections to even out
the number of pages they were responsible for.

Each selection of pages was given to at least two categories of participants: university students (bachelor, master and PhD students) and postdocs or early-career academics. The initial purpose was to promote interactions amongst participants during the review process, through online exchanges and, in particular, to encourage more experienced ECSs to provide advice. Based on the participants' feedback, it seems that such an exchange between the paired participants did not ultimately occur, and solutions to enhance this mentoring will be applied in the next round of group reviews of the SROCC organised by APECS.

Though participants themselves chose the chapter which
they would have to review, a significant number of concerns
were raised from participants that felt that the section they
were assigned did not correspond to their particular exper-

tise. While this was a valid concern, it was not possible to
screen which section would be best suited for each partic-
ipant and still cover the entire report, as the content of the
report was unknown beforehand. As the primary purpose of
this report is to inform policymakers, who will not neces-
sarily have scientific expertise concerning the content of the
report, having peripheral expertise would not preclude par-
ticipants' ability to review a given section, and instead we
deemed this an asset. In the forthcoming round of the group
reviews, applicants will request the specific sections that they
want to review rather than chapters, in order to have a more
specific application process.

The participants had a month to review their assigned sec-
tion. After the deadline, project leaders spent another month
compiling the 2155 received comments into one document,
removing potential duplicate comments and filtering those
interpreted as out of the scope (see Fig. 1c) to ensure the
relevance of the comments transferred to the IPCC. The
total workload of the participants (obtained through a sur-
vey after the review) was less than the project leaders, who
spent an estimated 40 h preparing the project, participating
in the webinars, reading the different chapters in which they
were involved and sorting all the comments. Following dis-
cussions with authors of IPCC reports and with members
of the IPCC TSU, the 2155 comments were sorted by the
project leaders as “editorial”, “substantive” and “unfit for
submission”. Editorial comments corresponded to sugges-
tions such as copy editing and reference corrections; substan-
tive comments were those indicating errors, the need for new
and more relevant references, and new content. Comments
deemed unfit for submission included mostly duplicates from
several reviewers and, in some rare cases, inappropriate com-
ments. We acknowledge that the comment sorting includes
some level of subjectivity, as a single chair was evaluating
each comment’s categorisation (in addition to self-evaluation
from the participants), and as a result, we only report the total
number of comments for each category and do not evaluate
the quality of the review in this paper. This procedure re-
sulted in a list of 2012 comments submitted to the IPCC. This
sorting and organisation was carried out during the 3 weeks
that followed the receipt of comments from the participants,
leaving 5 d to combine the selected comments and to upload
them to the IPCC website.

## 3 Results and discussion

### 3.1 Analyses of the number of APECS review comments

As described above, the comments were sorted by the
APECS project leaders and not by the IPCC. Nonetheless,
as the same criteria were endorsed and applied by all project
leaders; no obvious difference in the distribution of com-
ments from group review participants was evident.
<figure><img src="image_6.png" /><figcaption>**Figure 1.** Summary of the results from the APECS group review: (a) geographical distribution of ECSs who participated in the process and the number of participants per country, (b) disciplines of the participants involved in the group review, and (c) comments and time spent per participant by academic stage (the average training time was 3.5 h per participant, regardless of career level).</figcaption></figure>

Statistical analyses of the different number of comments were conducted to evaluate the significance of the obtained figures. The average number of comments per person was $31.8 \pm 4.6$ (errors on the average; sample size $n = 61$). The average and standard deviation of the numbers of comments for PhD, postdocs and early-career academics are $39.9 \pm 6.9$, $28.9 \pm 7.7$ and $34.9 \pm 9.5$ respectively (sample sizes $n = 26$, 21 and 14 respectively). These averages are relatively similar to the range calculated from the overlap of the standard

deviations, suggesting that these three figures are not significantly different. However, the comments of only one participant from the undergraduate and master academic level were obtained (the second one only acted as a project leader and did not provide comments); thus these were not included in the analyses.

For the proportion of comments deemed substantive, editorial and unfit for submission, no significant differences between either the number or the proportion were found
for each of the three academic levels detailed here. As the distribution of number of comments for each category and academic level were not normal, we performed a Kruskall–Wallis test and obtained $\chi^2 = 0.080$ and a $p$ value = 0.957, confirming that there were no significant differences in the proportion of comment types for each academic level.

In contrast, the average time spent by reviewers from each academic levels was, at least slightly, significantly different ($\chi^2 = 7.16$; $p$ value = 0.067), with groups varying by roughly 1 h, and the average error for each academic level below ±0.6 h.

## 3.2 Discussion

The participants produced a total of 2155 comments, with a mean of 31.8 comments per participant. Of the 2155 comments collected, 693 were considered substantive; 1319 were considered editorial; and 143 were considered unfit for submission. To explore potential links between the type of comments made by the participants and their academic level, the average numbers of comments (divided by category: substantive, editorial or unfit for submission) were plotted by academic level (PhD, postdoc or early-career academics; see Fig. 1c). PhD students provided the largest number of comments per participant (almost 40), of which 30% were substantive comments. Postdocs and early-career academics provided a slightly higher percentage of substantive comments with 34% and 36% respectively. The number of comments deemed unfit for submission was highest for early-career academics (10%), followed by PhD students (6%) and postdocs (4%), noting that this category included duplicate, thus redundant, comments. Amongst the different academic levels, no significant difference in the total number and category of comments was noticeable between the different career stages (see statistical analyses in Methods).

Compared to the 388 other expert reviewers of the SROCC FOD, the APECS group review (accounting for a single reviewer) provided 2012 comments to the SROCC authors, which was 17% of the 12 002 total number of comments (https://www.ipcc.ch/srocc/download/, last access: 18 April 2020). The 388 other experts produced on average 26 comments per person, slightly less than the ECS participants (31.8, the difference being above 1 standard deviation), who had a specific set of 10 to 20 pages each to review, rather than an entire chapter. Van der Veer et al. (2014) proposed that a large number of comments does not necessarily guarantee better quality, but as we only self-evaluated the quality of the comments, we are not able to validate this point with our results.

The PhD students provided as many substantive comments as the more experienced participants of the group review (i.e. postdocs and early-career academics); thus the length of the academic career was ruled out as a factor in the ability to effectively produce reviews. A more comprehensive analysis would benefit from the use of several metrics to determine the

quality of the review and the link to the length of academic careers. The comparison with other expert reviewers also supports that ECSs are as productive and efficient reviewers as their more senior peers (Schiermeier, 2016). As 20% of researchers take on 70% to 90% of the peer-review burden (Kovanis et al., 2016; Taylor and Francis, 2016), the peer-review process would be more equitably distributed across the scientific community by the solicitation and inclusion of PhD students and other ECSs as peer reviewers. Widening the reviewer pool could also reduce potential conflicts of interest in the review process and increase its quality. Considering the need for increased transparency in the peer-review process of IPCC reports following false accusations raised in reaction to the publication of the *Second Assessment Report* (SAR/AR2) (Edwards and Schneider, 2001), this latter aspect is particularly relevant for the climate change community.

Following the completion of the group review, participants were surveyed on their experiences. On average, the participants spent 7 h preparing prior to the review. Half of this time was spent reviewing the guide. The other half was spent attending the two training webinars on the review process and answering pending questions. Nearly all (98%) of the participants stated that the amount of training provided was adequate.

APECS participants spent an average of 7.3 h reviewing their assigned content (10 to 20 pages; Fig. 1c). However, 6% of the participants spent less than 3 h on the review, and 35% spent 10 h or more. Overall, participants spent less time than they had anticipated (35 h) on the review. The relatively reduced time commitment might make participating in future IPCC reviews more appealing to ECSs, not only as a group, but also individually. Indeed, based on the survey responses, the time spent to review a chapter of an IPCC report (between 80 and 120 pages) can be estimated at less than 1 week for reviewers of all career stages.

Participants were asked about their willingness to take part in another review as part of a group or individually (Fig. 2a). The majority (92%) of participants stated that they would take part in another group review, and 86% reported that they would participate in an individual review, adding more than 60 potential reviewers to the climate change community. Compared to the total number of experts who reviewed the FOD of the SROCC (389, including participants of the APECS group review), this means that including more ECSs in the peer-review process is a significant addition to the pool of expert reviewers. We expect similar results for other scientific fields and hope that other professional societies will consider collaborating in such group reviews in the future.

The most common motivations that ECSs stated for participating in this review process were learning more about the IPCC (59%) and experience building (53%; Fig. 2b). These survey responses, combined with the time commitment that reviewers within our group were ready to invest in this project (7 h of preparation and up to 35 h on the review), indicate that ECSs are highly committed to opport
<figure><img src="image_7.png" /><figcaption>**Figure 2.** (a) Participants' answers to the questions: "Would you be willing to participate in another group (left pie chart) and/or individual (right pie chart) review?" and (b) motivations for participating in such a review.</figcaption></figure>

tunities that involve international-policy-relevant processes. ECSs also value the experience of learning how to review and improve manuscripts. For example, 82% declared having achieved personal goals by participating in this review.

Prior to the review, one major concern of many participants was a lack of confidence in the value of their feedback to the IPCC, either because they were unsure about the quality of their comments or the sufficiency of their expertise, as the report covers a large ensemble of disciplines that might not fit their breadth of experience. Because the primary audience of IPCC reports are policymakers and stakeholders who may not have scientific backgrounds, IPCC reports should be written unambiguously. Even ECSs who do not have expertise in a particular topic can still provide constructive comments that strengthen the clarity of a report and the discussion of key concepts. Thus, in our group review, participants could choose the chapter they wished to review, but we randomly assigned within the sections this chapter they actually reviewed, regardless of their expertise or desire. The influence of the assignment process could not be assessed during this project.

## 4 Conclusions

This review included a large and diverse group of ECSs, both in terms of disciplines and countries represented. Participating in such a project furthers ECS career development (Moore et al., 2018), particularly in building skills towards connecting science and policy (Petes and Meyer, 2018). It was also an opportunity for ECSs to gather ideas for their future research projects, as knowledge gaps are clearly identified in the reports. The climate change scientific community needs to train more scientists to tackle future challenges (Goswami et al., 2015), and this type of initiative represents a valuable mechanism to prepare future IPCC authors. Influencing the policy-making process requires persistence

(Weible et al., 2012); thus, by including ECSs in the review of IPCC reports, the probability that they have an impact on climate-related policy during their careers increases (Evans and Cvitanovic, 2018).

Participating as reviewers has many benefits for ECSs, i.e. developing skills such as time management, responsible authorship, review and publication practices, and getting recognition for critical review skills within the scientific community. For the scientific community, it also increases the reviewer pool, which could alleviate the workload of senior scientists, and hopefully enhances the scientific rigour of journal articles and reports that support policy-making processes.

APECS is a volunteer-driven, professional society that seeks to provide career development opportunities for ECSs. APECS recognises the valuable contributions that ECSs make to furthering international scientific efforts and communicating the results of these studies to policymakers and the public. APECS organised and trained ECSs to review a large policy-relevant scientific report because there are few peer-review training opportunities for ECSs. APECS hopes that this initiative, which demonstrated that ECSs can serve as adequate peer reviewers, will inspire journal editors and report managers to include more ECSs as reviewers.

There are several scientific societies, which support climate change research and ECSs, which could organise their own group review of future IPCC draft reports. While we use the review of the IPCC report as a case study for the climate change community, we expect that similar results would be found for ECSs across scientific disciplines and for journal articles (Ilgen et al., 2016). We note that advice for ECSs to produce robust reviews can easily be found (Schiermeier, 2016; Silver, 2016), so instead, we offer recommendations to editors of journals and managers of assessment reports to include more ECSs in their review processes:

- Proactively seek to include at least one ECS when inviting reviewers for a manuscript. This could be accomplished by requesting that senior reviewers suggest a suitable ECS to review the manuscript.

- Include ECSs as guest editors. This would diversify perspectives of editorial boards.

- Encourage reviewers to review the sections of a manuscript they feel most comfortable with. This may facilitate participation both for new reviewers and senior scientists who may be too busy to review the entire manuscript.

- Keep reviewers blind to other reviewers' comments during the review phase. As recommended in Besançon et al. (2019), this will facilitate independent feedback, especially for ECSs, who may hesitate to disagree with more senior reviewers.

- Consider methods to encourage and train ECSs to act as reviewers. This could be in group reviews or mentor-
ing programmes (De Vries et al., 2009; Walker, 2018), potentially using ECS-based networks (e.g. APECS) as a focal point for such training, in particular for thematic special issues.

**Data availability.** All the data needed to reproduce the results shown in this paper are compiled in the Supplement.

**Supplement.** The supplement related to this article is available online at: https://doi.org/10.5194/gc-3-89-2020-supplement.

**Author contributions.** MC and GG designed the study; all authors contributed to the results and wrote the paper.

**Competing interests.** The views of the authors of this paper do not reflect the official view of the IPCC. Three of the authors are Technical Support Unit staff members of the IPCC.

**Acknowledgements.** This paper is dedicated to the memory of our friend and colleague Adrian Dahood, who sadly passed away in August 2019. The research leading to these results has received funding from the Humboldt Foundation project DEAPICE. Sarah Connors works at the IPCC Working Group I Technical Support Unit, and Elvira Poloczanska and Katja Mintenbeck work at the IPCC Working Group II Technical Support Unit, which are based at the University Paris Saclay, France, and the Alfred Wegener Institute for Polar and Marine Research, Germany, respectively. The views in this paper do not reflect any official opinion of the IPCC. We acknowledge the help and support of APECS executive committee, particularly of Juan Höfer, Gabriela Roldan and Andrea Piñones. We acknowledge the fruitful comments of Lonni Besancon, Hans Visser, Gary McDowell and one anonymous reviewer.

**Financial support.** The article processing charges for this open-access publication were covered by a Research Centre of the Helmholtz Association.

**Review statement.** This paper was edited by Jon Tennant and reviewed by Hans Visser, Gary McDowell, Lonni Besancon, and one anonymous referee.

## References

* Allen, P., Gerlis, F., and Frigga, K.: Association of Polar Early Career Scientists Promotes Professional Skills, Eos, Transactions American Geophysical Union, 95, 204–204, https://doi.org/10.1002/2014EO240004, 2014.

* Besançon, L., Rönnberg, N., Löwgren, J., Tennant, J. P., and Cooper, M.: Open Up: A Survey

on Open and Non-anonymized Peer Reviewing, https://doi.org/10.20944/preprints201905.0098.v2, 2019.

* Bolin, B.: A history of the science and politics of climate change: the role of the Intergovernmental Panel on Climate Change, ISBN 0521880823, 2007.

* De Vries, D. R., Marschall, E. A., and Stein, R. A.: Exploring the peer review process: what is it, does it work, and can it be improved?, Fisheries, 34, 270–279, 2009.

* Ding, D., Maibach, E. W., Zhao, X., Roser-Renouf, C., and Leiserowitz, A.: Support for climate policy and societal action are linked to perceptions about scientific agreement, Nat. Clim. Change, 1, 462–466, 2011.

* Edwards, P. N. and Schneider, S. H.: Self-governance and peer review in science-for-policy: The case of the IPCC Second Assessment Report, in: Changing the atmosphere: Expert knowledge and environmental governance, 219–246, 2001.

* Evans, M. C. and Cvitanovic, C.: An introduction to achieving policy impact for early career researchers, Palgrave Communications, 4, 88, https://doi.org/10.1057/s41599-018-0144-2, 2018.

* Geffers, J., Beaudry, C., Yang, H.-C., Huang, F., Phanraksa, O., Dominik, M., Lin, Y.-C., Huang, M.-C., Komai, S., Lorimer, K., Piyawattanametha, W., Saengchantr, P., Saleh, H., Tagg, B., and Veerakumarasivam, A.: Global State of Young Scientists (GloSYS) in ASEAN – Creativity and Innovation of Young Scientists in ASEAN, Global Young Academy, 100 pp., 2017.

* Goswami, S., Xu, X., and Hayes, D.: Four key areas for training the next-generation of global change researchers, Curr. Sci., 108, 1195–1196, 2015.

* Hallegatte, S., Rogelj, J., Allen, M., Clarke, L., Edenhofer, O., Field, C. B., Friedlingstein, P., Van Kesteren, L., Knutti, R., and Mach, K. J.: Mapping the climate change challenge, Nat. Clim. Change, 6, 663–668, 2016.

* Ilgen, J. S., Jr, A. R. A., Simpson, D., Yarris, L. M., Chretien, K. C., and Sullivan, G. M.: Group Peer Review: The Breakfast of Champions, Journal of Graduate Medical Education, 8, 646–649, https://doi.org/10.4300/jgme-d-16-00531.1, 2016.

* IPCC: Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA, 1535 pp., 2013a.

* IPCC: Principles Governing Ipcc Work: Procedures for the Preparation, Review, Acceptance, Adoption, Approval and Publication of IPCC Reports: https://www.ipcc.ch/site/assets/uploads/2018/09/ipcc-principles-app appendix-a-final.pdf (last access: 9 April 2020), 2013b.

* Kovanis, M., Porcher, R., Ravaud, P., and Trinquart, L.: The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise, PLoS ONE, 11, e0166387, https://doi.org/10.1371/journal.pone.0166387, 2016.

* Lerback, J. and Hanson, B.: Journals invite too few women to referee, Nature News, 541, 455–457, 2017.

* Lewandowsky, S., Gignac, G. E., and Vaughan, S.: The pivotal role of perceived scientific consensus in acceptance of science, Nat. Clim. Change, 3, 399–404, 2013.

* Maibach, E., Myers, T., and Leiserowitz, A.: Climate scientists need to set the record straight: There is a scientific consensus that human-caused climate change is happening, Earth’s Future, 2, 295–298, 2014.
Moore, M., Martinson, M. L., Nurius, P. S., and Kemp, S. P.: Trans-disciplinarity in research: Perspectives of early career faculty, Res. Social Work Prac., 28, 254–264, 2018.

Petes, L. E. and Meyer, M. D.: An ecologist’s guide to careers in science policy advising, Front. Ecol. Environ., 16, 53–54, https://doi.org/10.1002/fee.1761, 2018.

Schiermeier, Q.: Peer review: close inspection, Nature, 533, 279–281, 2016.

Silver, P.: Advice for early-career peer reviewers and authors re-sponding to peer reviews, Freshw. Sci., 35, 1073–1075, 2016.

Taylor & Francis: Peer review in 2015: A global view. ParkDrive, England: Taylor & Francis, retrieved from: https://authorservices.taylorandfrancis.com/peer-review/ peer-review-global-view/ (last access: 20 April 2020) 2016.

Tollefson, J.: Climate science: An erosion of trust?, Nature News, 466, 24–26, 2010.

van der Veer, L., Visser, H., Petersen, A., and Janssen, P.: Innovating the IPCC review process – the potential of young talent, Climatic Change, 125, 137–148, 2014.

Walker, T.: Help graduate students to become good peer reviewers, Nature, 561, p. 177, 2018.

Weible, C. M., Heikkila, T., DeLeon, P., and Sabatier, P. A.: Understanding and influencing the policy process, Policy Sci., 45, 1–21, 2012.