Cycle 8 2021
ALMA Proposal Review Process
ALMA Proposal Handling TeamUpdated: 1 March 2021

ALMA Review Process (Cycle 8 2021 to ...)
1. Some context for the changes introduced in the ALMA Review Process
2. Overview of the ALMA Review Process (Cycle 8 2021 to …)
3. Panel review
4. Distributed peer review
5. About the merged, ranked list
6. A few updates for Large Programs and queue buildingSome context for the changes introduced inthe ALMA Review Process (Cycle 8 2021 to …)
A few challenges to address in Cycle 8 2021:• Large number of submitted proposals (~ 1800) places heavy burden on panelists. Workload may impact quality of the reviews.
• International Visiting Committee (IVC) and ASAC concerns about relatively few ambitious (in terms of time) projects proposed and accepted:• Number of submitted Large Programs continues to decline in each cycle (Figure a).
• Fewer proposals requesting > 20-30 hr (Figure b).
• Low (i.e., zero) acceptance rate (Grade A/B) for 40-50 hr proposals (Figure c).
• Potential biases in the review process to date.
30(a) Number of submitted Large Programs by cycle**See Proposers’ Guide

Number2520151050(b) # of proposals by 12m time (Cycle 7)

Cycle 4Cycle 5Cycle 6Cycle 7(c) Success rate by 12m time (Cycle 5-7)

Three major changes introduced in theALMA Review Process for Cycle 8 2021

Distributed peer review
Encouraging larger programs
Dual anonymous

Overview of the ALMA Review Process inCycle 8 2021
Proposal submission

Proposal review process

12-m Array: < 25 hrs
7-m Array (if ACA stand-alone): < 150 hrs

JAO final checks

Review results sent to PIs

12-m Array: >= 25 hrs
All Large Programs

Stage 1 assessment

Stage 1 review

TriageDistributed peer review

Stage 2 review

Panel review process
ARP Meetings (Stage 2 assessment)

APRC MeetingPanel review in ALMA Cycle 8 2021
Proposal submission

Proposal review process

In the panel review process, proposals are reviewed by panels of experts on thescientific subject of the proposals.

How it works:• Proposals assigned to panel based on their scientific category.
• Conflicts of interest checked• All panel members review all (non-conflicted) proposals assigned to their panel.

JAO final checks

12-m Array: >= 25 hrs
All Large Programs

Stage 1
• Conflict identification.
Stage •1 Science
review assessors score all their assigned proposals and write a comment.
• Triage performed as needed after Stage 1, minimizing workload in Stage 2.Panel review process
Stage 2: ALMA Review Panel (ARP)
Meetings peer review
Distributed• Face to face meeting —> Virtual in 2021
• Proposals are re-scored based on discussion; consensus reports are written.
ALMAStage
2 Proposal
review Review Committee (APRC) Meeting• Recommend which Large Programs should be scheduled• The panel review process results in a ranked list of all submitted proposals.
Panels in Cycle 8 2021 will be similar to previous cycles, but with reduced numberof proposals and total science assessors, although more panelists per panel
compared with previous cycles.

Review results sent to PIs

Stage 1 assessment

TriageARP Meetings (Stage 2 assessment)

APRC Meeting

Distributed peer review in ALMA Cycle 8 2021
Proposal submission

Proposal review process

JAO final checks

Distributed peer review is a process in which one member of the proposal team(either the PI or one of the CoIs) reviews ten other proposals.

12-m Array: < 25 hrs
7-m Array (if ACA stand-alone): < 150 hrs

Stage 1 review
Distributed peer review
Stage 2 review

Review results sent to PIs

Why is ALMA using distributed peer review? After Cycle 7 Supplemental Call,
distributed peer review proved to be a viable review system for the ALMA
community (see ALMA Memo 616). Additionally it provides a series of advantages
over panel review, such as more involvement of theStage
community,and a significant
1 assessment
reduction of the workload for each reviewer.

How it works:
reviewer.
• Each proposal team designates one member as the actingTriage• Proposals are assigned to reviewers based on reviewers' expertise.
review process
Stage Panel
1 (Mandatory)• Conflict identification
write a commentthe PI.
• Reviewers rank the assigned ten proposals
ARP andMeetings(Stage 2toassessment)
Stage 2 (Optional)• It is possible to re-rank proposals and edit the comments to the PIs based on
the comments made by the other (9) reviewers.
review isMeetingused by ALMA in a• Cycle 8 2021 will be the first time that distributed peerAPRC•main Call for Proposals.
In order to make assignments, ALMA users should update their expertise in
their science portal profile.

The outcome of the processes is a merged,
ranked list.
Proposal submission

Proposal review process

JAO final checks

12-m Array: < 25 hrs
7-m Array (if ACA stand-alone): < 150 hrs

Review results sent to PIs

12-m Array: >= 25 hrs
All Large Programs

Stage 1 assessment

Stage 1 review

TriageDistributed peer review

Panel review process
ARP Meetings (Stage 2 assessment)

Stage 2 review
Ranked list with all submitted proposals

APRC Meeting

Creating a ranked list from the panels
Category 1 rank 1

Category 2 rank 1

Category 3-5 rank 1

Category 1 rank 2

Category 2 rank 2

Category 3-5 rank 2

Category 1 rank 3...
...

Category 2 rank 3...
...

Category 3-5 rank 3...
...

• Panelists score proposals in their panel• Average score determines proposal rankings within a panel.

Creating a ranked list from the panels
Category 1 rank 1
Category 2 rank 1
Category 3-5 rank 1
Category 1 rank 2
Category 2 rank 2
Category 3-5 rank 2
Category 1 rank 3• Procedure to create a single ranked
list across all panels remains thesame as used in Cycles 0-7• Rankings are normalized by thetotal number of proposals withineach panel, and sorted by thenormalized ranks among all panels.

Category 2 rank 3
Category 3-5 rank 3...
...

• Interleave the rankings from thedifferent panels

Creating a ranked list from distributedreview• Each reviewer ranks theirassigned proposals from 1(best) to 10 (weakest),
essentially a series ofchoices.
• Choices from the individual
reviewers are used todetermine the consensus
ranked list of all proposals.

DPR rank 1
DPR rank 2
DPR rank 3
DPR rank 4
DPR rank 5
DPR rank 6
DPR rank 7
DPR rank 8...
...

Creating two ranked lists
Category 1 rank 1

DPR rank 1

Category 2 rank 1

DPR rank 2

Category 3-5 rank 1

DPR rank 3

Category 1 rank 2DPR rank 4

Category 2 rank 2

DPR rank 5

Category 3-5 rank 2

DPR rank 6

Category 1 rank 3

DPR rank 7

Category 2 rank 3

DPR rank 8

Category 3-5 rank 3

...
...

...
...

• Ranked list will be
generated for eachprocess (panel anddistributed review)Merging two ranked lists
Category 1 rank 1
DPR rank 1
DPR rank 2
DPR rank 3
Category 2 rank 1
DPR rank 4
DPR rank 5
DPR rank 6
Category 3-5 rank 1...
...

• Rankings from the panels and distributed peerreview are combined by normalizing the ranked
lists by the number of proposals in each review
process, and then sorting by the normalizedrank.
• Interleaving assures there are no systematicdifferences in the overall science rankings of thepanels vs. distributed peer review.
• Large programs recommended by the APRC will
have top priority in the final ranked list.

A few updates for Large Programs andqueue building
ALMA encourages PIs to submit larger, more ambitious proposals• No cap on the total time that can be allocated to Large Programs. Note, Large Programs still cannot fill
more than 50% of the time in a configuration/LST.

• Large Programs, and proposals that require more than 25 hours on the 12-m Array, will have first priority
to fill at least 10% of the observing queue.

Towards dual-anonymous review
The Joint ALMA Observatory (JAO) is committed to reducing biases in thereview process.

1. Modifications in the review process in recent cycles
2. Cycle 7 context
3. Updates for Cycle 8 2021: dual-anonymous guidelines

Modifications in the review process in recentcycles
Cycle 7:
• removed investigator names from reviewer tools• randomized investigator list on cover sheet to not identify PI• first names listed with first initial to not identify gender
Cycle 5-6: Institutions, emails, executive
removed from proposal cover sheet

Impact: some systematics changed,
but not others

Impact: no significant change

Cycle 4: Panel Chairs andReviewers informed ofsystematics during ALMA Review
Panel meeting orientation
Impact: no significant change

C

lecy0

lc
Cye1
Clcye

2
C

lecy3

lecy4

CLonsdale et al. (2016)

Clec
y5eClcy6elcy8elc
1Cy 0227C

Carpenter (2020)

Modifications in the review process in recentcycles: Cycle 7 context
SystematicRegional

Impact in Cycle 7
Chile

Relative rankings improved. While Chilean proposals had poorer ranks than Europe andNorth America, for the first time it was not statistically significant.

East Asia

No significant change.

First-time PIs
Experience

Second-time PIs
Most experienced PIs

GenderNo significant change. Continue to have the poorest overall rankings.
Rankings were indistinguishable from the most experienced PIs, whereas before they weresignificantly poorer.
For the first time, these PIs did not have the best overall proposal rankings.

Consistent with previous cycles within errors.

ImpactOverall: Some (but not all) systematics reduced after modifications in Cycle
7, suggesting some biases in the review process have been present.

Addressing systematics
Regional affiliation: ChileFigures: Carpenter (2020), updated for Cycle 7

ImpactRelative rankings improved in Cycle 7. While Chilean proposals had poorer ranks
than Europe and North America, for the first time it was not statistically significant.

Addressing systematics
Regional affiliation: East Asia

Figures: Carpenter (2020), updated for Cycle 7

ImpactNo significant change for relative rankingsof East Asian proposals in Cycle 7.

Addressing systematics
Experience of PI (“prestige” bias)

Figures: Carpenter (2020), updated for Cycle 7

Impact• First-time PIs (red curve) continue to have the poorest rankings.
• For the first time, the most experienced PIs (black) did not havethe best overall proposal rankings.

Addressing systematics
Second-time PIs

Figures: Carpenter (2020),
updated for Cycle 7
ImpactRankings for second-time PIs were indistinguishable from themost experienced PIs, whereas before Cycle 7 they weresignificantly poorer.

Addressing systematics
GenderFigures: Carpenter (2020),
updated for Cycle 7

Impact

For the first time, women did better than expected based on demographics,
although consistent with previous cycles within uncertainties.

Addressing systematics
Dual-anonymous in Cycle 8 2021
Proposal submission

Proposal review processsse

JAO final checks

corpeleohthsuwwooellhmoTyfno.
nsaelnialueddiug

12-m Array: < 25 hrs
7-m Array (if ACA stand-alone): < 150 hrs

Stage 1 review

Review results sent to PIs

12-m Array: >= 25
All Large Programs

Distributed peer review

Stage 2 review

lliw

Stage 1 assessment

TriagePanel review process

ARP Meetings (Stage 2 assessment)

Ranked list with all submitted proposals

APRC MeetingAddressing systematics
Dual-anonymous in Cycle 8 2021
Dual-anonymous review is when the proposal team does
not know the identity of the reviewers, and the reviewers
do not know the identity of the proposal team.
Why is ALMA adopting this system? To ensure that theproposal review process is as fair and unbiased as
possible for all ALMA users.
Systematics from Cycle 0-6, before dual-anonymous, arereported in Carpenter (2020). JAO committed to reducing
biases in the review process.

How it works:• Proposals are written in such a way to not identify theteam behind it.
• All identifying information related to the authors of theproposals is hidden from the reviewers and scienceassessors throughout the scientific review process.

Will some guidance be provided?
• Of course! Link will be shared.
• PIs encouraged to contact ALMA Helpdesk if in doubt.
• ARC staﬀ should contact PHT for clarification if theycannot point the PI to the proper documentation.

Documentation and resources
https://almascience.org/proposing/alma-proposal-review

Cycle 8 2021 Call for Proposals opens March 17, 2021
Helpdesk: help.almascience.org