ALMA Proposal Review Process
EA community meeting

Pei-Ying Hsieh on behalf of Proposal Handling Team
March 22, 2024

What’s new for Large Programs Review• Proposals for Large Programs are required to submit a management plan as part of
the Scientific Justification (which must follow dual anonymous guidelines)• Large Programs also require a one-page “Team Expertise” document, separate fromthe Scientific Justification. The document is allowed to include names andinstitutions, and should be uploaded to the OT at the time of submission.
• The ALMA Proposal Review Committee (APRC) will read the “Team Expertise”
document only after completing the scientific ranking of the proposals.

Goal of presentation
ALMA distributed peer review (DPR)

Large Programs review

Dual anonymous review

Basics of DPR
One member of each proposal team* commits to participate in the review process

Each reviewer reviews 10 proposals (Proposal Set) for each submitted proposal

The process• Stage 1• Reviewers identify conflicts of interest• Reviewers rank the proposals from 1 to 10 (best to weakest) and provide acomment• Stage 2• Access to anonymized reviews made by other reviewers.
• Have chances to modify ranks and comments.
* Excluding Large Programs

Reviewer timeline for Cycle 11April 25

1) Proposal PI designates the reviewer in Observing Tool (OT)

Proposal deadline

April 30
Expertise & conflicts

1) Reviewer specify scientific expertise in Preferences
2) Reviewer provide list of conflicts of interest in Preferences
3) Deadline to provide alternative reviewer, if necessaryStage 1

1) Plenary sessions May 9-14 (optional, and highly recommended)
2) Declare any conflicts of interest in assigned proposals by May 153) Complete reviews by June 5 @ 15 UT (MANDATORY!)

June 6 - June 20

1) Read reviews from other reviewers2) Modify your ranks and comments as needed

May 8 - June 5Stage 2

PI must designate the reviewer
April 25 UTC 15:00

1) Proposal PI designates the reviewer in Observing Tool (OT)

Proposal deadline

Student PIs can be reviewers, but need to specify a mentor who will assist in the review.

PI designates the reviewer
April 25 UTC 15:00

1) Proposal PI designates the reviewer in Observing Tool (OT)

Proposal deadline

A single reviewer can be assigned a maximum of FIVE Proposal Sets for DPR

How are proposals assigned to reviewers?
• Category/keywords of proposal• Categories/keywords of reviewer’sexpertise• List of Conflicts of interests

PI proposals

Proposal topics
Machinelearning
Computesimilarities
Reviewer’s proposal history

Reviewer Expertise
Machinelearning

Optimizeassignments

Proposal sets

Providing Reviewer expertise
April 30 UTC 15:00
Expertise & conflicts

1) Reviewer specify scientific expertise in Preferences
2) Reviewer provide list of conflicts of interest in Preferences
3) Deadline to provide alternative reviewer, if necessary

1) Log in to the ALMA Science Portal
2) Edit your “Preferences”
3) Go to the Expertise tab4) Select keywords that match your scientific expertise5) Go to the Confirm tab to saveReviewers can specify their conflicts of interest
April 30 UTC 15:00
Expertise & conflicts

1) Reviewer specify scientific expertise in Preferences
2) Reviewer provide list of conflicts of interest in Preferences
3) Deadline to provide alternative reviewer, if necessary

1) Log in to the ALMA Science Portal
2) Edit your “Preferences”
3) Go to the Conflicts of Interest tab4) Identify ALMA users for which you have a conflict5) Go to the Confirm tab to save

What is considered a conflict of interest?
• In general, a reviewer has a major conflict of interest when their personal or work
interests would benefit if the proposal under review is accepted or rejected.

• Close collaborators, which are defined as a substantial collaboration on three or more
papers within the past three years or an active, substantial collaboration on a current
project. Co-membership in a large team on its own does not constitute a conflict ofinterest.
• Students and postdocs under supervision of the reviewer within the past three years• A reviewer’s supervisor (for student and postdoc reviewers)• Close personal ties (e.g., family member, partner) that are ALMA users• Any other reason in which a reviewer believes a major conflict of interest exists
If a reviewer does not provide their conflicts, the PHT will determine conflicts based on thereviewer’s proposal history for the past three cycles.

Alternative reviewers
April 30 UTC 15:00
Expertise & conflicts1) Reviewer specify scientific expertise in Preferences
2) Reviewer provide list of conflicts of interest in Preferences
3) Deadline to provide alternative reviewer, if necessary

A single reviewer can be assigned a maximum of FIVE Proposal Sets

• After the proposal deadline, the PHT will contact those reviewers who haveselected to receive more than five Proposal Sets
• It is expected that for these cases, the reviewer identifies a new reviewer among
the proposal co-Is• If no alternative has been identified by the deadline (April 30 UTC 15:00), the PHT
will cancel the reviewer’s highest code(s) proposals• Email address --> the PHT will communicate with the reviewers using the email
address registered in the science portal, so it is important that it is up to date andvalid as soon as possible

Stage 1: Review assigned proposals
May 8 - June 5Stage 1

1) Plenary sessions May 9-14 (optional, and highly recommended)
2) Declare any conflicts of interest in assigned proposals by May 153) Complete reviews by June 5 @ 15 UT (MANDATORY!)• The PHT will host three Webinars (links can be found in the ALMAScience Portal (Proposing -> ALMA Proposal Review)
Session 1: Thursday May 9, 17:00 UTCSession 2: Friday May 10, 13:00 UTCSession 3: Tuesday May 14, 2:00 UTC• During this sessions, the PHT will explain the different aspects of
distributed peer review, and will be available to answer questions• The presentation and slides will be posted in the ALMA Science
Portal before the Webinars• Attending one of the sessions is not mandatory, but it is highly
recommended

Stage 1: Review assigned proposals
May 8 - June 5Stage 11) Plenary sessions May 9-14 (optional, but highly recommended)
2) Declare any conflicts of interest in assigned proposals by May 153) Complete reviews by June 5 @ 15 UT (MANDATORY!)

Declare any additional conflicts in your assigned proposals by May 15 (in the reviewer tool)• for example: observing the same object(s) with the same goals

If you identify a conflict after you submitted your conflicts in reviewer tool, create a Helpdesk
ticket to be assigned another proposal.

Stage 1: Review assigned proposals
May 8 - June 5Stage 11) Plenary sessions May 9-14 (optional, but highly recommended)
2) Declare any conflicts of interest in assigned proposals by May 153) Complete reviews by June 5 @ 15 UT (MANDATORY!)• Rank the proposals within a proposal set from 1 (strongest) to 10 (weakest) based on
scientific merit.
• Write comments that summarize the strengths and weaknesses of the proposal• Comments will be sent to the PI verbatim.
• Reviewer’s proposal will be canceled if the reviews are not submitted on time!
• Extensions will not be granted since Stage 2 starts on June 6.

The reviewer can be changed after the proposal deadline in exceptional circumstances by having
the proposal PI submit a Helpdesk ticket. The Stage 1 deadline remains the same.

Stage 2: Finalize the ranks and reviews
June 6 - June 20
Stage 21) Read reviews from other reviewers2) Modify your ranks and comments as needed

The reviewers can read comments of the other 9 reviews per proposal within a proposal
set. This is to see if you overlook any critical strengths or weaknesses.

You have chance to update your ranks and comments as needed.

Stage 2 is encouraged! However, if a reviewer does not complete Stage 2, the Stage 1
ranks/comments are considered final.

Cycle 11 Large Programs Review• The APRC will attend the virtual meeting(June 24-28) and prepare the list ofrecommendation (16-18 people).

• They attend only stage 1 (May 5June 8) : read 1-3 proposals andwrite the reports (<4000 words) tothe APRC (~90 people).

Review Criteria Large Programs

Dual anonymous: overview
All proposals must be written following the dual anonymous review guidelines• Basic principle is that the proposal should not reveal the proposal team• Reviewers should focus on the proposed science, and not the proposal team

Guidelines provided on the ALMA Science Portal (Proposing => ALMA Proposal Review).

Do not reveal the teams!
• Do not list the PI, co-PIs, or coIs anywhere in the proposal• Includes abstract, Scientific Justification, and Technical Justification

Use third person phrasing• Reference your own work in the third person

In Smith et al. (2018), we demonstrated …
Our study (Smith et al. 2018) showed that …

As demonstrated in Smith et al. (2018), …
Smith et al. (2018) showed that …

Referencing data and software anonymously• Do not refer to software or data from ALMA or other observatories in a self-identifying fashion• If software or datasets are available in a public repository (e.g., GitHub) or in a public paper, they can bereferenced per normal practices• If software or datasets are not public reference them as "obtained via private communication" or similar language

The proposed ALMA observations will be combined with our HST data (code, PI) ...
We use our group's line identification package STAR ...

The proposed ALMA observations will be combined with available HST data (private communication) ...
We use the line identification package STAR (obtained via private communication) ...

Do not list PIs of other proposals• Do not name the PI when listing a project code, even if it is not your own project

Figure 1 shows the image from the Cycle 7 program (2019.1.02045.S, PI Pérez).
Figure 1 shows the image from the Cycle 7 program (2019.1.02045.S)
Figure 1 shows the image from the Cycle 7 program (private communication)

Referencing papers in preparation• Papers in preparation need to be referenced as private communication without anassociated name.

Figure 1 shows the CO image from Gómez et al. (in preparation)

Figure 1 shows the CO image (private communication)

Referencing submitted papers• References to submitted papers are not permitted (use “private communication”)• If a submitted paper has been posted on the archive (e.g, arXiv), the archive paper
can be referenced per usual practices
Our sample was obtained from a recent survey (Chang et al. submitted).
Our sample was obtained from a recent survey (Atro-ph 123).
Our sample was obtained from a recent survey (private communication).
The sample was obtained from a recent survey (Astro-ph 123).

Resubmissions• Proposers may note if they are resubmitting an ongoing proposal. This is usually done in the“duplication” box on the cover sheet.
• Do not list the proposal code, ranking, priority grade, or the PI of the previous proposal in theresubmission statement.
• If data from the previous proposal are presented in the Scientific Justification, it must be presented ina dual anonymous fashion.

This is a resubmission of our ongoing program 2021.1.02045.S (PI: Smith). Half of the targets
have been observed and we are resubmitting the proposal to observe the remaining half.

This is a resubmission of our ongoing program. Half of the targets have been observed and
we are resubmitting the proposal to observe the remaining half.

More information
https://almascience.org/proposing/alma-proposal-review• Dual-anonymous guidelines• Description of the distributed peer review• Detailed guidelines for the reviewers• FAQFor proposal and review help
https://help.almascience.org• “Proposal Handling” department• “Proposal Review Support” department

Thank you!
Questions?