# Update on ACM SIGCOMM CCR reviewing process: towards a more open review process (Editorial article)

Ralph Holz, Marco Mellia, Olivier Bonaventure, Hamed Haddadi, Matthew Caesar, Sergey Gorinsky, Gianni Antichi, Joseph Camp, Kc Klaffy, Bhaskaran Raman, et al.

► To cite this version:

Ralph Holz, Marco Mellia, Olivier Bonaventure, Hamed Haddadi, Matthew Caesar, et al. Update on ACM SIGCOMM CCR reviewing process: towards a more open review process (Editorial article). Computer Communication Review, 2020, 50 (3), pp.55-56. 10.1145/3411740.3411748. hal-04766791

HAL Id: hal-04766791

https://inria.hal.science/hal-04766791v1

Submitted on 5 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Update on ACM SIGCOMM CCR reviewing process:
towards a more open review process

**Ralph Holz**

University of Twente, The
Netherlands
r.holz@utwente.nl

**Marco Mellia**

Polytechnico di Torino, Italy
mellia@polito.it

**Olivier Bonaventure**

Université catholique de Louvain,
Belgium
Olivier.Bonaventure@uclouvain.be

**Hamed Haddadi**

Imperial College, UK
h.haddadi@imperial.ac.uk

**Matthew Caesar**

UIUC, USA
caesar@illinois.edu

**Sergey Gorinsky**

IMDEA, Spain
sergey.gorinsky@imdea.org

**Gianni Antichi**

Queen Mary University of London,
UK
g.antichi@qmul.ac.uk

**Joseph Camp**

SMU, USA
camp@smu.edu

**kc Klaffy**

CAIDA, USA
kc@caida.org

**Bhaskaran Raman**

IIT Bombay, India
br@cse.iitb.ac.in

**Anna Sperotto**

University of Twente, The
Netherlands
a.sperotto@utwente.nl

**Aline Viana**

INRIA, France
aline.viana@inria.fr

**Steve Uhlig**

Queen Mary University of London,
UK
steve.uhlig@qmul.ac.uk

This article is an editorial note submitted to CCR. It has NOT been peer-reviewed.

The authors take full responsibility for this article's technical content. Comments can be posted through CCR Online.

## ABSTRACT

This editorial note aims to first inform the SIGCOMM community on the reviewing process in place currently at CCR, and second, share our plans to make CCR a more open and welcoming venue by making changes to the review process, adding more value to the SIGCOMM community.

## CCS CONCEPTS

* General and reference → Validation.

## KEYWORDS

Editorial

## 1 OVERVIEW

We present a number of changes we make to the way CCR operates and handles and evaluates submissions. Section 2 describes the role of the editorial board, the type of papers CCR accepts, and the way decisions are made. We also discuss how authors currently interact with reviewers and our plans to improve on this in the future and allow a more productive and interactive form of communication. We also discuss the review process. Section 3 introduces a central aspect where we want encourage change: reviewer anonymity.

Section 4 addresses a second necessary change aiming at widening the scope of the contributions considered, by adding a new metric to the review process, called "value to the community".

## 2 EDITORIAL BOARD AND TURN-AROUND

This section describes the role of the editorial board and the current process used to handle papers. Then, we explain the changes we will make to the review process to improve turn-around time as well as to have more open communication between authors and reviewers.

### 2.1 Role of the editorial board

CCR publishes four issues each year: January, April, July, and October. At the moment, once a paper is submitted on the CCR hotcrp review system, the CCR editor sends the list of papers submitted to the next issue to area chairs for them to bid which papers they would like to handle. After the bidding process, the selected area chairs invite reviewers for their papers. Typically, there are three reviewers for each paper, and a decision is made within about ten weeks from the reviewer assignment. The final decision can be either made solely by the area chair when there is a clear outcome
based on the reviews, or it may involve reviewers and the CCR editor (e.g., often for “revise and resubmit to the next issue” decisions).

Three types of decisions are currently employed: “accept”, “reject”, and “revise and resubmit to the next issue”. These three options enable fast decisions to a quarterly publication. In the case of acceptance or rejection, the area chair posts a summary letter on the hotcrp system as a comment to the submission. The case “revise and resubmit to the next issue” requires the authors to submit their revisions within a limited time (typically for the next issue, around two months after the decision), to keep the turn-around time limited while allowing for improvements.

## 2.2 Empowering area chairs

One of the changes proposed is to give area chairs the power to do “fast-track rejects” to speed up the turn-around time for returning decisions. Note that this is already done by the editor for submissions that are clearly out-of-scope. Also, from now on, it is the sole responsibility of the area chairs to decide on a paper, although they may draw on the assistance of the editor when required.

The goal of these changes is to empower area chairs, who, based on the advice of the reviewers, will make decisions that they believe are best for CCR, and the SIGCOMM community.

## 2.3 More open communication with reviewers

Currently, the main communication channel used between reviewers and authors is through the reviews and the decision letter (comment) from the area chair. The comments reflecting the discussion among reviewers and area chair that led to the decision are not visible to authors. The authors also have limited means to discuss issues with the reviewers, e.g., through a rebuttal or a revision plan, which are only offered to authors in specific cases. Therefore, the review process is mostly opaque to authors, sometimes leading to frustration if the reviews were perceived as unfair, based on a misunderstanding, or suffering from some form of bias, e.g., too much emphasis on novelty.

We plan to improve the transparency of our review process. Our plan is to selectively make comments visible to authors and inviting them to respond before a final decision on a submission is made. Using comments more extensively between authors and reviewers as a communication channel may help clarify misunderstandings regarding a paper’s contribution or allow to discuss technical details that might be not clear enough.

# 3 REVIEW TRANSPARENCY

Currently, the review form already includes a tick box allowing reviewers to make their name visible to authors. However, it is not yet a widespread practice for reviewers to tick it. We envision a reviewing model where it becomes standard practice for reviewers to make themselves known.

Such models do exist. For instance, BMJ Open runs an open peer-review process. BMJ Open<fnref n="1" /> requires reviewers to (1) sign reviews with their name, position and institution, (2) declare any competing interests, and (3) reviews are published online alongside the authors’ original versions and replies to the reviewers’ comments if the article is published.

We understand that encouraging reviewers to make their identity known may feel risky, e.g., due to possible retaliation. Therefore, we do not plan to require reviewers to drop their anonymity unless they prefer to do so. However, following the example of BMJ Open, we will ask the area chairs to share their public reviews together with the original reviews submitted by the reviewers, which may include the reviewer name if they waived their anonymity.

We hope that by doing this, the reviewing process will be more transparent for the entire community as well as create a virtuous circle among reviewers that encourages high-quality reviews, in particular together with the improved communication channels described above.

# 4 PAPER EVALUATION: BEYOND NOVELTY

Currently, the CCR review form focuses on novelty to decide on technical papers. The reason for this is historical, grounded in the original purpose of CCR as a platform for fast dissemination of new ideas and early but promising results in our community. With the widening of the type of contributions that are considered at CCR (namely technical papers, editorial notes, reproducible papers, and educational contributions), the tight focus on novelty is not appropriate any more.

With the educational track introduced in 2020, as well as the reproducibility track that encourages sharing artefacts, CCR needs to account for this broader scope. To this end, the review form now includes a numerical score for “value to the research community”, which will be taken into account in the review process.

Emphasising technical novelty in paper acceptance indeed makes SIGCOMM publications selective and prestigious. However, such emphasis may also lead to ignoring valuable contributions to the community. CCR, therefore, already made a huge step forward by welcoming different types of contributions. We still need to make improvements to explicitly allow reviewers to identify and positively score the value of different kinds of submissions. The new “value to the research community” will hopefully serve this purpose.

# 5 CONCLUSION

We have outlined a number of changes that we are introducing in CCR to reduce the turn-around time, improve communication between authors and reviewers, make the review process more transparent, and allow reviewer to assess papers more holistically. We will carefully evaluate how these changes are perceived by authors, reviewers, and the community as a whole, and we hope to report on this soon. Please send feedback, comments, and questions to the editor at ccr-editor@sigcomm.org, or post your comments online on https://ccronline.sigcomm.org/.

<footnote n="1">¹See https://bmjopen.bmj.com/pages/reviewerguidelines/.</footnote>