How to be a good meta-reviewer?
Meta-reviewer tutorial for ICML 2022
ICML 2022 Program Chairs
Based on material from CVPR 2022

Acknowledgement: We are grateful for the CVPR 2022 chairs who shared their slides on the same topic. These slides are largely based on theirslides, with some modifications to better fit the needs of ICML.

1TL;DR• You really should look at the whole slide deck, even if you are aseasoned meta-reviewer (MR), since there are various new policies
and guidelines.
•

Some main changes this year that will affect your role:•••

The review process has 2 phases
Reviewers will not provide scores, only a verbal evaluation and weighing ofstrengths and of weaknesses
Meta-reviewers are responsible for ensuring that the papers they handle
receive professional reviews and are in general treated fairly2

What you will learn from this tutorial• Understand the nature of the review process• Understand the role of meta-reviewers in the review process• Understand the expectations for meta-reviewers• Understand new policies & guidelines for ICML 2022
• Understand what should be done & avoided when reaching paper decisions• Understand how to deal with policy issues
3

The decision process: Phase 1
MR reads reviews. For papers marked
by both reviewers for rejection, check
reviews carefully and conservatively
recommendation rejections, if
sufficient grounds.

Abstract
submissions

Paper
submissions

MetaReviewers

SMRs review proposals,
recommend for PCs

Senior MetaReviewers
Decisions to authors

Programchairs

Reviewers

Create paper
assignments,
including two
reviewers + one MR

At least 2 reviewers, atleast one experienced,
both provide full reviews.
Paper can be flagged forrejection

ProgramchairsPCs finalize
decisions then sendout notification forPhase 1 rejects

Phase 1
rejectdecisions

The decision process: Phase 2
Papersnot rejected in Phase 1

MRs read reviews, ensure they
are polite & professional, read
The extra reviewer(s)rebuttal, consolidate all, direct
per paper provide a full Authors get all
discussion, prepare
review, not seeing the reviews, write
recommendation, write
previous reviews
rebuttal if needed
Decisions to authors
meta-review“Surviving”
Papers

Reviewers

Rebuttals

MetaReviewers

Paper
Decisions

MetaReviewers

Programchairs

Reviewers

Senior MetaReviewers

Programchairs

Add an extra
reviewer per paper

Make sure all papers
have at least 3 reviewers

Reviewers read andconfirm reading
rebuttal, update
reviews if needed,
sort out differences

Control/oversee
decision process,provide input, checkquality, discuss andfinalize decisions

Discuss decisions,finalize them,oversee process

The nature & principles of the review process• The review process will never be perfect.
• Some authors will inevitably be unhappy about the decisions.

• But, we want it to be a fair and transparent process for every paper.
• The authors are explicitly informed on WHY their paper is accepted/rejected.

• Authors may appeal only if they identify clearly flawed reasoning in theprocess.
• But it is not valid for them to simply appeal the specific decision.

• We strive for the most informed & fair decision for each paper.
• Best possible decisions given the paper, reviews, rebuttals, and discussions.
• We strive for a consistent process.
• Clear and uniform policies that are well understood.
6

Your tasks as an MR• Your community service is of utmost importance• You use your skill and experience to advise the PCs on paper decisions• You ensure that reviews and meta reviews are professional, respectful• You ensure that all authors understand the basis of the recommendations• You communicate with reviewers and SMRs on open questions andrecommendations• The quality of the program crucially depends on you!

• Treat all involved fairly, compassionately, and uniformly• Choose referees likely to understand the paper• Understand all issues raised in reviews and rebuttal• Do not make up policy on-the-fly• Behave ethically, and expect ethical behavior from others• Raise and deal with improprieties in papers• Avoid and report conflicts of interest
7

The decision process in detail (1/3)
1. Authors, PC & SPC update their meta-information
2. Senior meta-reviewers (SMR) bid on meta-reviewers (MRs). MRs are
assigned to SMRs. Each SMR works with 5-6 MRs
3. Reviewers and meta-reviewers (MRs) bid on papers
4. Phase I: Papers are assigned to reviewers and meta-reviewers. 2 reviewers
per paper, at least one “expert”, and one meta-reviewer per paper. Averageload for meta-reviewers: 18 papers
5. Reviewers submit phase I reviews, typically handling 2 papers each.
MRs check the quality of reviews, chase late reviewers, make reviewers fix
poor reviews. If a paper is flagged for Phase I reject by both reviews, carefully
evaluate the reasons and recommend accordingly. Recommend rejection
conservatively. If you do, write a respectful meta review explaining thereasons, with substantiation.
8

The decision process in detail (2/3)
6. SMRs check Phase I reject recommendations, and check quality, ensure
professionalism of reviews/meta review
7. Phase II: one more reviewer is assigned to each remaining paper. MRs canoverride system recommendation
8. MRs check the quality of reviews, chase late reviewers, make reviewers fix
poor reviews, and add emergency reviewers if need be. Each paper needs 3
reviews. Reviewers will not give scores, so ensure the reviews are sufficiently
informative
9. Authors receive reviews and write rebuttal
10. Discussion ensues among reviewers and MR, based on all reviews, rebuttal,
and paper: Goal is to resolve conflicts between reviews, or between reviews
and the rebuttal/paper. MRs stimulate and moderate the discussion.
Reviewers update their reviews and justifications
9

The decision process in detail (3/3)
6. MRs make decisions and write meta-reviews. Borderline decisions orquestions are discussed with the SMR. MR provide a ranked list of
nominations into categories – probably similar to long, short-talk, poster, but
details will be announced – and of borderline papers. For borderline papers,
provide clear information why this paper should be accepted, and what the
weaknesses are
7. SMR checks quality of meta reviews and generates a merged rough ranking intheir entire batch
8. PCs make final determination of paper categories for accepted papers, basedon the recommendations of the MRs, but taking into account time and spaceconstraints as well as topic diversity
10Other important instructions

11

Other important points
professionalism, be respectful, once more
- keep in mind accessibility, D&I: these are currently
recommendations, not yet enforced, but should be respected in thecamera ready version. If authors follow them, good, if not, remindthem to follow these for the camera ready (if accepted)- be available to your reviewers, MRs, SMRs, and reply quickly!
- carefully read the author guidelines, the ethics rules- reviewer resources:--

tutorial
review form
slidesnominations for reviewer awards: we are planning to have these;
more details to follow.
12

Who knows whose identity?
reviews are double-blind: reviewers do not know author identities,
authors do not know reviewer / MR / SMR identities• MR does not know author identity• PCs know author identity• SMR knows reviewer and MR identity, and MRs know SMR identity••

You can add emergency reviewers to a paper. COIs: since MRs do notknow author identity, add the reviewer as an extra reviewer withquota 1 (you will be able to do this), this will reveal conflicts. You
may also talk to your SMR.
13

What should be included in the review?
• A concise summary of the paper• What problem is addressed in the paper?
• Is it a new problem? If so, why does it matter? If not, why does it still matter?
• What is the key to the solution? What is the main contribution?
• Do the experiments sufficiently support the claims?

• A clear statement of strengths and weaknesses• What are the key contributions and why do they matter?
• What aspects of the paper most need improvement?

• A comprehensive check of potential fundamental flaws in the paper• Are the assumptions and theories (mathematically) sound?
• Are the experiments scientifically sound and valid?
• Is the problem addressed trivial?
• Did the paper miss important prior work? Has it been done before? If yes, where?

What should be avoided in the review?
• Common mistakes in a sloppy review• Arrogance, ignorance and inaccuracy• Be responsible!

• Pure opinions• Be grounded!

• Novelty fallacy• Be knowledge-driven!

• Blank assertions• Be substantial!

• Policy entrepreneurism• Be sensible!

• Intellectual laziness• Be active!

[Developed from: Reviewer Slides for CVPR’21]

Guideline: Discussion of Limitations• Summary: Authors are encouraged to explicitly and honestly discuss
limitations.
• Motivation: Discussing limitations used to be commonplace in ourcommunity, but seems to be increasingly lost. Yet, discussing
limitations is important and useful.
• Our procedure:• We do not formally require the inclusion of an explicit limitations section.
• Do NOT penalize papers for honestly discussing limitations.
• Rather weigh the inclusion of an honest discussion POSITIVELY.

16

Guideline: Attribution of Data Assets• Summary: Authors are advised that they need to cite data assets used
(e.g. datasets or code) much like papers.
• Motivation: We want to raise awareness that data assets need to becited as carefully as papers.
• Our procedure:• Treat used data assets just like previous work in the form of papers.

17

Your Work as MR

18

Phases of MR duties
1. Suggest reviewers for each paper••

In some ways, this is actually the most important step and requires time. If done well, therest of the process is a piece of cake. If not, it is a disaster!
Do not rely solely on TPMS and topic matching. You are smarter than a weighted
histogram matcher, no? The automated tools provide initial rankings, but ultimately you
should rely on your judgment. You can check reviewers’ publication lists, experience etc.

2. Guide the review/discussion process•••

Check that submitted reviews are complete and respectful. Catching bad reviews early cansave you (and the authors) a lot of pain later.
Post-review, start and moderate discussions where appropriate to clarify any open issues
and/or conflicting opinions.
Get reviewers to update their final ratings – especially to reconsider their positions once
they have more context (rebuttal, other reviews) and time to reflect.

3. Decide in communication with your SMR••

Discuss and decide each paper with the help of your SMR. For that, write an initial meta-review
to summarize and explain. Then, discuss all papers with your SMR, in particular borderline
papers. Use the summary and your suggestion as a basis for discussion.
Update the meta-review and recommendation after discussing with your SMR.
19

How to choose reviewers
Spend a few minutes reading the abstract / intro / figures / citations to understandthe key idea, main technical topics, and community involved.
2.
While reading the paper, jot down names of reviewers that come to mind. Lookthrough the list of references for more ideas. Finally, look at top-ranked reviewers
according to TPMS & topic scores. Check these reviewers in more detail.
3.
Suggest at least 5 potential reviewers in Phase I who have expertise in each maintechnical topic, are likely to enjoy reading the paper, and can give feedback on the
big ideas as well as technical details. Suggest at least 3 such reviewers in Phase II,
who would give useful complementary information to existing reviews.
Pro tips:
1.

•
•

•

Do not trust TPMS blindly; many of its suggestions can be bad. Verify the suggestions by checking thereviewers’ Google Scholar profiles.
Senior reviewers sometimes receive surprisingly low TPMS scores even though they are experts on the
topic (but also many others, hence the TPMS score). That said, famous names do not guarantee good
reviews. Often lesser-known experts for the specific area are just what you need.
Some reviewers may not show up in the TPMS rankings because they did not create a TPMS profile. If
20
you think of somebody who may be good to review a paper, search for their name directly.

Phase I recommendations•

After Phase I reviewing, specifically check the papers that 2
reviewers marked for Phase I reject. These are candidates for Phase I
reject, all other papers continue into Phase II.
•
•

•

•

Carefully read the arguments in the reviews, make sure you understand andcheck if you agree (e.g., have a quick look at the paper too).
Recommend reject carefully. (You can override the reject recommendations
and you will obtain another reviewer in Phase II.) If you agree, write an
informative meta review and ensure all reviews are informative,
professional, respectful.
After Phase I, the only available recommendations are “reject” and“continue to Phase II”. Only papers marked by both reviewers for reject canbe recommended for reject. There is no author rebuttal at this stage yet.
Check all reviews to determine what kind of third reviewer would be mosthelpful.

21

How to guide the discussion
•
•
•
•
•
•
•
•
•

The goal of the discussion is to clarify questions/uncertainties, and possibly
conflicting opinions.
Make sure that reviewers are discussing the paper following the rebuttal.
Every reviewer should give a response to the rebuttal even if all reviewers
lean in the same direction.
Most often, reviewers will only discuss the paper or specific issues that wereraised if prompted by you. Make sure you stimulate and guide the discussion.
Pay attention especially to cases with diverging reviews. Encourage, but do
not force consensus.
For every paper, at least send a quick note of the type:
3x accept: "Given your clear accept recommendations, the paper is headed for
acceptance unless anyone objects."
3x (weak )reject: "Given your unanimous reject recommendations and no rebuttal fromthe authors, the paper seems to be headed for rejection. Any objections?" / "The paper
had 3 reject recommendations. Did the rebuttal change your assessment in any way?"
Mixed scores: "Is there someone who champions this paper? What do you make of eachother's reviews? Did the rebuttal change your assessment?"
Etc.

22

Phase II recommendations•

After Phase II reviewing, the reviewers discuss (under yourguidance), and then you discuss your papers, in particularborderline papers, with your SMR.
•••

Ensure all reviews are appropriate, informative, professional
Engage in discussion with reviewers and SMR

Then you make your recommendation (Award level, categories ofpresentation (details to be announced), Reject) and write ameta-review substantiating the recommendation, and summarizing
the discussion (anonymously).

23

Level of decisions (details to be determined)• Award: major advances that will heavily impact the field; will be used by many
people, create new capabilities, etc.
• Oral: potential to be very significant; particularly innovative; worthwhile for thewhole community to hear about• Poster: incremental steps that expand the sum of the community’s knowledge or
add bricks to the cathedral of knowledge; papers introducing useful tools; papersof interest to a subcommunity• Also, creative ideas that are hard to judge but could be promising — no one
knows the future, so we should give the benefit of the doubt to plausible ideas• Reject: not correct, significantly incomplete, known, or unlikely to add relevant
knowledge to the community (note that “relevance” can be very broad, so treat
this carefully)• Your job as an MR is mainly to decide acceptance/rejection and to provide SMRs
and PCs with recommendations for levels of presentation
24

Why not accept everything?
Papers can have a negative impact:
Wrong or fraudulent results mislead the field and damage thereputation of the conference.
● Misleading evaluation makes it hard to compare with, kills follow-up.
● Creates bad precedent (weak paper X got in, so this one should too).
● Fatigue/overload of too many papers, wastes everyone’s time.
● Publishing results that were known can harm the original authors of
the result (undue credit)●

Each weak or mediocre paper we accept hurts the conference a little(though not as much as rejecting a good paper).
25

The role of experimental evaluation“Extraordinary claims require extraordinary evidence.”— Pierre Simon-Laplace, paraphrased
Different papers require different levels of evaluation, for example:
Potentially transformative idea: basic proof-of-concept● Established problem, plausible idea: benchmark results● Weird, overly complex, implausible, and/or seemingly incremental:
extraordinary results (which need to be scrutinized carefully)● Position piece or theory paper: no experiments●

Having a good idea is not enough to have a paper. Ideas are cheap.

26

How to decide• Judge each paper based on its own merits, not some ranking with respect toother submissions to the conference. There is no global quota on thenumber of papers we can accept.
• For unanimous accepts or rejects, mainly follow the reviewers (but do asanity check in case the reviews are completely off or missed something).
• For all other cases: Carefully consider reviews, see if you agree with claims.
• Which arguments did the reviewers make that you find compelling?
• Which claims need more substantiation?

• If you aren’t sure: get help from SMRs, or ask the SMR or PCs to put you intouch with another expert MR
27

Best practices for decisions• If all referees agree that a paper should be rejected, we expect the paper to• be accepted ONLY if there are unusual circumstances: examples are• a major and obvious referee error;• a compelling rebuttal that causes referees to change their mind.

• If all referees agree that a paper should be accepted, we expect the paper to• be rejected ONLY if there are unusual circumstances: examples are• a major technical error;• fraud or plagiarism not originally detected by referees.

• Even for clear cases, it is advisable to at least skim the paper.
• We do not want glaring false positives or false negatives.
• Well founded oral/poster recommendations require more careful reading.
28How to write meta-reviews• A meta-review is a well-founded argument on whether a paper
does/does not meet the criteria for acceptance. It is not simply asummary of the individual reviews. Especially in the case of difficult“reject” decisions, you must reconcile the reviews and give explicit
justification for the final recommendation.
• Key principle: would a reasonable author object to the meta-review
as the basis for the decision?
• Use impersonal language: MRs, authors – not “I”, “you”. Instead of“the authors do not…” say “this work does not…”• Take care when reviewers do not agree, especially if your decision
goes against the majority.
• Senior meta-reviewers must proofread every meta-review to make 29
sure it meets standards.

Meta-reviews for easy cases• When all reviewers agree, a couple of sentences will suffice.
• In cases of acceptance, do not mention level of presentation
recommendation in your meta-review since this status will not be
finalized until later by the PCs.
• In cases of rejection, briefly mention the main reasons for rejection
and acknowledge the rebuttal (if only to say that it was missing or did
not change anybody’s mind).
• Examples:• “Three reviewers agree that the paper contains technical errors, such as …
and is not clearly written. There is no rebuttal.”• “Three reviewers find that the paper is below ICML standards due to a lack ofnovelty [explain briefly] and insufficient experimental validation. The rebuttal
did not assuage these concerns.”
30

Meta-reviews for difficult cases

If referees disagree, or you disagree with them, or they makeborderline recommendations:
1.
2.
3.
4.
5.

Summarize the main arguments of the reviewers.
Indicate whether the rebuttal removed any initial concerns (if there was no
rebuttal, explicitly state “there was no rebuttal”).
Summarize any substantive points from the reviewer discussion (authors
will not see the discussion).
Reconcile all the above information and explain the main factors for yourdecision.
Discuss your decision with your senior MR. Include “This paper was
discussed by the senior program committee” or similar language.
31

Meta-reviews: Checklist••

Does the summary mention reviews or referees?
Did the referees agree?
•••

If referees disagree, or make borderline recommendations:
•
•
•
•
•

•

If all referees agree, is the summary consistent with that consensus?
(if not, it may be OK, but should be scrutinized very carefully, as if thereferees disagree; in this case, we expect the SMR to be involved, and thesummary to be a clear record as below)
Was there a rebuttal?
Does the meta-review mention the rebuttal?
Was there a discussion?
Does the meta-review mention the discussion?
Does the meta-review give the main points used to reach the decision?

Is the meta-review going against the majority of reviews?
• Is the senior meta-reviewer on-board?
• Is the decision clearly explained?

32

Meta-reviews: What NOT to do• “This sort of thing has been done for years and needs to stop.”• “Paper describes a method that has been known for a while.”• “Majority of reviewers vote X.”• “Two of three reviewers vote reject and there is a rebuttal and I agreewith the majority.”• “Three borderline reviews, discussion is mixed, there is a rebuttal, but
I don’t like the paper.”• “This paper has no value.” – do not use this language at all.

33

Recommendations for Handlingof Policy Issues

34

Policy Issue: Plagiarism• Stealing text from another paper, written by other authors.
• Our procedure:• For any charge of plagiarism, refer to PCs and proceed as if charge was FALSE;• prepare a case for review by SMR and PC; notify SMR and PCs• PCs will discuss the case and decide• Outcomes:• Decided that plagiarism: note to MR, authors, desk reject• Decided that not: note to MR, SMR, authors.

35

Policy Issue: Conflicts of Interests• Hard conflicts:• Resolved with help of CMT• Fully enforced.

• Soft conflicts:• There may be quite a lot of soft(er) conflicts, e.g.
• Paper might be by a friend;• you might have started a collaboration;• you owe author a favor;• you owe referee a favor; etc.

• Self-report any uncomfortable situations to PCs,
• even if you feel this will not affect your decision;• we will figure out how to proceed.
36Policy Issue: Anonymity and Format
Violations• Glaring anonymity violations are a desk reject, but small stuff is better
just noted in the summary.
• We do not want big organizations or famous people bullying referees, but it’srough to reject for minor issues;• reviewers occasionally postulate anonymity violations that do not hold up;• use your best judgement and check with SMR.

• Obvious format violations are a desk reject, but small stuff is better
just noted in the summary• We do not want authors to unfairly work around the page limit;• but an occasional negative \vspace is not grounds for rejection.

37

Policy Issue: arXiv and concurrent work
• Papers that are only on arXiv are not peer-reviewed. Be careful whenusing such papers against a paper, you then need to double check theother paper very carefully.
• Papers that appeared within one month of the ICML deadline should
be considered concurrent, and cannot be used against a paper.
Reviewers may still point out this concurrent work, to be added as areference later, but it can’t be used as “this has been done”.
• Papers can still cite arXiv papers.
• Use your best judgment about those.
38

Policy Issue: Suggested Changes• Script: Referee/MR reads paper, sees ways in which it could be better,recommends changes which authors refuse to adopt.
• Suggested solution: Authors’ problem. Also, if it is not acceptablewithout changes, reject it.
• Rationale: You cannot stop fools from being fools, and it is not worth
trying.
• We make the best decisions we can based on the information wehave, but if you make a suggestion that makes their paper better, andthey want to leave it out of final paper, they really have not read the
memo.

39

Policy Issue: extra experiments and results• In general, the very short rebuttal time leaves very limited time for extra
experiments, and reviewers must keep that in mind
• The main point of the rebuttal is to clarify open questions• Script: Referee wants extra experiments in rebuttal; authors supply (or donot).
• Suggested solution: Use your best judgement. You cannot reject because
people refused to supply extra experiments. You may find that theexperiments supplied pre-rebuttal sufficient. You cannot reject because
they supplied extra experiments, either.
• What will go in the camera ready? In the end, we have no control whether
authors will make significant additions/changes in the final version.
Essentially, reviewers and MRs need to hence judge the paper as is, at mostwith minor feasible changes taken into account.
40

Policy Issue: Data Contributions (1/2)• Script: The release of a dataset is a major contribution of a paper, yet
the authors do not commit to releasing the dataset to the public.
• Solution: Mark this in the meta-review form and advise authors thatthey need to provide a URL with the camera-ready, which will be
included on the conference website. We cannot/will not check if the
URL actually contains the dataset.
• Rationale: We want to reduce the cases of promising a release andunderdelivering, but you cannot stop fools from being fools.
• If the mandatory URL does not actually provide the dataset, it is theauthors' problem. This will be apparent on the conference website.
41

Policy Issue: Data Contributions (2/2)• Script: Paper describes an advanced methodology for capturing adataset, but does not actually contribute a dataset itself. The authors
do not commit to releasing data to the public.
• Solution: It is not mandatory to release a dataset. A dataset release is
expected only if the dataset itself a key contribution. Judge such
papers on the basis of technical merit.
• Rationale: There are cases when releasing a dataset is not possible.
This is similar to the case of proposing an algorithm and not releasing
code, which is acceptable.

42

Policy Issue: Secret Datasets• Script: Referee/MR rejects paper as “unscientific” because it isevaluated on a dataset that cannot be/will not be/has not been
published, so cannot be replicated.
• Solution: You really cannot do this. There is no such policy. Judgesituation on its merits.
• If an issue comes up that looks like a matter of policy, raise it withyour SMR and, if still unclear, the PC.

43Policy Issue: Attribution of Data Assets• Script: A paper is using a data asset (dataset or code), which it is not
citing.
• Solution: There is NO POLICY leading to an automatic rejection. Henceuse your best judgement and treat this like a missing paper reference.
Point out in your meta review that the camera-ready should provide
the missing reference to the data asset.
• Rationale: We want to raise awareness that data assets need to becited as carefully as papers, but we are not ‘policing’ this.

44

Policy Issue: Use of Personal Data /Human Subjects (1/2)• Script: A paper is using a dataset that has been withdrawn by its
creators, e.g. DukeMTMC-ReID or MS-Celeb-1M.
• Solution:• Not SOLELY grounds for rejection, but closely scrutinize.
• Report to SMR and PCs.
• Rationale: Such datasets should really not be used (anymore).
• It is a violation of policy for a reviewer or meta-reviewer to requirecomparison on a dataset that has been withdrawn without a detailed
consultation with PCs or DEI chairs or ombuds.

45

Policy Issue: Use of Personal Data /Human Subjects (2/2)• Script: A paper is using a broadly used public dataset for which the
IRB situation is unclear, but which has not been withdrawn.
• Solution:• Nothing special to do. Proceed as usual.
• Rationale: While datasets should have been created with IRB
approval, awareness in the community needs to be built over time.
Our policy focuses on newly created datasets. However, deprecateddatasets should not be used (see previous slide).

46

Policy Issue: Negative Societal Impact• Script: Paper has glaring negative societal impact (e.g., paper
proposes a technology for a weapon system).
• Suggested solution:• Inform the SMR and PCs, who will advise how to proceed.
• Until then, treat the paper as if there was no issue.
• Rationale: Grave ethical issues are obviously problematic, yet ourcommunity does not (yet) have formalized processes for dealing withthis. Need to consider on an individual basis.

47

Policy Issue: Limitations• Script: Paper has clear limitations, but the text does not discuss thisexplicitly.
• Suggested solution: There is no formal policy requiring a discussion oflimitations. Use your best judgement and treat limitations just likeyou have done in the past. Accept paper only if it is acceptable withthe apparent limitations. Advise authors to discuss limitations in thecamera ready.
• Rationale: You cannot make up a policy. A formal policy may beintroduced in the future. For now we are taking one step at a time.

48

need to add/include
This year, we are following a 2-phase process. You will need to write meta reviews for phase I
rejections and phase iI decisions.
Phase I rejections: check all reviews for being reasonable and polite. If paper marked by both
reviewers for rejection, carefully evaluate reason and conservatively recommend rejection. It is very
important that rejections happen for well substantiated reasons only, especially in this phase.
Summarize these reasons in a meta review for the authors
Phase I+II: very important to ensure reviews and meta reviews are informative, constructive,
professional and respectful. They should not use personal language. This is very important for thefield, as it can discourage especially junior authors, lead to loss of useful ideas, resubmission of
essentially the same paper to the next conference (waste of everyone’s time) and overall lowering ofreview standards, since frustrated authors may act similarly next time they review, or students aretaught the wrong standards.
This year, reviewers do not give scores. This means the MR must draw this information from the textof the review, weigh and evaluate. Hence, it is even more important that reviews are informative.
Please communicate with the reviewers if the necessary information / weighing of pros/cons is notincluded. You may add an additional reviewer if need be.
49

Work in the MR triplet• Decisions will be reached in MR triplets:• They are conflict free.
• They have a mix of new and more experienced MRs.
• Triplets discuss borderline cases.
• Triplets discuss accept cases toward oral/poster recommendations.
• Each paper has one secondary meta-reviewer from the triplet, whodouble-checks and signs off decision and meta review written by the primaryMR.

• If you need expertise not present in your triplet:• Contact the PCs who will put you in touch with a conflict-free expert inanother triplet.
50Policy Issue: Self-Plagiarism• Reusing text from the authors' own paper.
• Our procedure:• For any charge of self-plagiarism, refer to PCs and proceed as if charge was
FALSE;• one PC investigates, recommends to other PCs who vote accept/reject;• use (weak) IEEE Intellectual Property office standards.

• Outcomes:• note to MR, authors, pointing out policy of IEEE, no further action.

51

Policy Issue: arXiv• Script: Referee rejects paper because there are better results on arXiv.
• Solution: You cannot do this because according to PAMI-TC policies,
arXiv does not exist.
• Script: Referee rejects paper because it cites an arXiv paper.
• Solution: You cannot do this either; it is fine to cite arXiv papers.
• Script: Referee rejects paper because it does not cite a paper
published only on arXiv.
• Solution: Papers merely on arXiv are not peer-reviewed. Also, why
would you reject a paper merely for not citing a specific work? There
must be more substantial reasons.
52Policy Issue: Use of Personal Data /Human Subjects (1/4)• Script: A paper is using personal data or data from human subjects. Authors
did not obtain IRB / ethics approval but DO include a description of how
ethical principles were followed.
• Solution:• Check if the description seems to reasonably follow the ethics principles from the
Ethics Guidelines (see CVPR website).
• If YES, advise authors that they should be seeking IRB approval in the future.
Nothing else to do.
• If NOT, inform the PCs and continue to handle the paper as if there was no issue.
• PCs will reach out to ombud, who will review the case and advise how to proceed.

• Rationale: We want to raise awareness that the appropriate ethics review isnecessary, but we do not have a full fledged ethics review this year.
53

Policy Issue: Use of Personal Data /Human Subjects (2/4)• Script: A paper is using personal data or data from human subjects.
Authors did not obtain IRB / ethics approval and DO NOT include a
description of how ethical principles were followed.
• Solution:• Inform the PCs and continue to handle the paper as if there was no issue.
• PCs will reach out to ombud, who will review the case and advise how toproceed.

• Rationale: We want to raise awareness that the appropriate ethics
review is necessary. If the authors do not even to attempt to describethat data was used in an ethical fashion, the case needs to bereviewed.
54

Policy Issue: Extra Experiments• Script: Referee wants extra experiments in rebuttal; authors supply(or do not).
• Suggested solution: Use your best judgement. You cannot reject
because people refused to supply extra experiments. You may findthat the experiments supplied pre-rebuttal sufficient. You cannot
reject because they supplied extra experiments, either.
• Rationale: PAMI-TC policy states referees cannot require extra
experiments. Do not invent new policies.

55

Policy Issue: Will extra results fit?
• Script: Referee asks for extra in rebuttal; authors supply; now there ismore material, and we are not sure what will go in the paper.
• Suggested solution: Authors’ problem.
• Rationale: You cannot stop fools from being fools, and it is not worth
trying.
• We make the best decisions we can based on info, but if they haveinfo that makes their method look good, and they leave it out of the
final paper, they really have not read the memo.

56

Policy Issue: Negative Societal Impact (1/2)• Script: Paper does not discuss potential negative societal impact.
• Suggested solution: There is no formal policy requiring a discussion ofpotential negative societal impact. You should advise authors toinclude such a discussion in the camera ready. Nothing else to do.
• Rationale: You cannot make up a policy. A formal policy may beintroduced in the future. For now we are taking one step at a time.

57